How to Find All Existing and Archived URLs on a web site

How to Find All Existing and Archived URLs on a web site

Blog Article

There are lots of good reasons you would possibly want to discover all the URLs on an internet site, but your exact intention will decide Everything you’re searching for. For illustration, you may want to:

Establish each and every indexed URL to research difficulties like cannibalization or index bloat
Gather recent and historic URLs Google has found, specifically for site migrations
Come across all 404 URLs to Get well from put up-migration mistakes
In Each individual circumstance, an individual Resource won’t give you all the things you may need. Unfortunately, Google Lookup Console isn’t exhaustive, as well as a “web page:illustration.com” research is restricted and tough to extract info from.

With this publish, I’ll wander you thru some applications to create your URL record and just before deduplicating the info employing a spreadsheet or Jupyter Notebook, dependant upon your website’s measurement.

Outdated sitemaps and crawl exports
For those who’re trying to find URLs that disappeared through the Stay web-site not too long ago, there’s a chance another person in your team can have saved a sitemap file or even a crawl export ahead of the adjustments have been designed. In the event you haven’t currently, look for these information; they will normally offer what you require. But, for those who’re looking through this, you probably didn't get so Fortunate.

Archive.org
Archive.org
Archive.org is a useful Device for Search engine optimisation jobs, funded by donations. If you try to find a domain and select the “URLs” selection, you could obtain as many as 10,000 detailed URLs.

However, There are some limitations:

URL Restrict: You can only retrieve up to web designer kuala lumpur 10,000 URLs, and that is inadequate for larger web-sites.
Quality: Quite a few URLs might be malformed or reference source data files (e.g., images or scripts).
No export alternative: There isn’t a designed-in strategy to export the checklist.
To bypass the lack of an export button, use a browser scraping plugin like Dataminer.io. On the other hand, these constraints imply Archive.org might not deliver a complete Option for bigger web pages. Also, Archive.org doesn’t show no matter if Google indexed a URL—however, if Archive.org found it, there’s a great possibility Google did, as well.

Moz Professional
Although you could generally make use of a url index to search out exterior web sites linking for you, these applications also explore URLs on your web site in the method.


Tips on how to utilize it:
Export your inbound inbound links in Moz Pro to secure a swift and easy list of target URLs from the web page. When you’re handling a massive Web-site, consider using the Moz API to export facts past what’s manageable in Excel or Google Sheets.

It’s important to Observe that Moz Professional doesn’t verify if URLs are indexed or learned by Google. Nonetheless, considering the fact that most internet sites implement the identical robots.txt policies to Moz’s bots because they do to Google’s, this technique commonly is effective perfectly to be a proxy for Googlebot’s discoverability.

Google Look for Console
Google Look for Console presents various precious sources for creating your list of URLs.

Links stories:


Comparable to Moz Pro, the Hyperlinks portion delivers exportable lists of goal URLs. Unfortunately, these exports are capped at 1,000 URLs Each individual. You'll be able to utilize filters for particular web pages, but since filters don’t use to your export, you might must rely on browser scraping resources—limited to 500 filtered URLs at a time. Not perfect.

Functionality → Search engine results:


This export will give you an index of web pages getting lookup impressions. Whilst the export is limited, You can utilize Google Search Console API for larger datasets. Additionally, there are free Google Sheets plugins that simplify pulling extra intensive information.

Indexing → Internet pages report:


This portion delivers exports filtered by problem form, although these are definitely also restricted in scope.

Google Analytics
Google Analytics
The Engagement → Web pages and Screens default report in GA4 is a superb source for gathering URLs, by using a generous limit of 100,000 URLs.


A lot better, you may use filters to produce distinct URL lists, successfully surpassing the 100k Restrict. For instance, if you wish to export only blog site URLs, follow these measures:

Action 1: Incorporate a phase into the report

Move two: Simply click “Produce a new section.”


Phase 3: Determine the phase that has a narrower URL pattern, for example URLs containing /blog/


Observe: URLs found in Google Analytics might not be discoverable by Googlebot or indexed by Google, but they supply useful insights.

Server log files
Server or CDN log information are perhaps the ultimate Instrument at your disposal. These logs capture an exhaustive checklist of every URL path queried by buyers, Googlebot, or other bots throughout the recorded period.

Considerations:

Details dimensions: Log documents could be huge, so many web-sites only keep the final two weeks of data.
Complexity: Analyzing log information is often challenging, but different equipment can be obtained to simplify the procedure.
Incorporate, and superior luck
When you’ve collected URLs from these sources, it’s time to mix them. If your site is sufficiently small, use Excel or, for greater datasets, tools like Google Sheets or Jupyter Notebook. Ensure all URLs are continuously formatted, then deduplicate the checklist.

And voilà—you now have an extensive listing of present, outdated, and archived URLs. Good luck!