How to find all 301 redirect for URL xyz.com/products (internal and external)?

[tpt.com]

This is what we are thinking:

• Get all URL of the xyz.com/products using XENU software.
• Search  those URL on google (site;xyz.com url ) to find out if they are crawled by google, do the same on bing (as currently google shows 4k URL and bing 11k )
• Use opensiteexplorer (301 redirect ) and using (internal external) to get the desired result.

Is this the right approach? If not, what is the best way to find the correct result?

All suggestions are welcome.

[RyanKent]

I used to use Xenu until shortly after Dr Pete shared this blog (http://moz.com/blog/crawler-faceoff-xenu-vs-screaming-frog) which introduced me to Screaming Frog. Both will work, but you will find XENU is more like using DOS whereas Screaming Frog has a very nice interface.

For internal 301s, you can clearly crawl the site and export a list of all 301 redirects to the target page.

For external URLs, there is not a simple method. I suggest two tactics:

1. Examine your analytics for referring URLs

2. Examine your backlink reports for links to the page

You can then crawl the list of URLs and determine which pages are being redirected. With the above understood, the primary concern should be your internal URLs.

[tpt.com]

Hi Ryan,

Thanks for your quick response.

Yes we heard about the screaming frog but never used it, will give it a try.

We do not have access to google/bing webmaster tool or analytics. We are more like a third party company working on this project.

Any other ideas?

[RyanKent]

Your original question expressed a desire to locate "all" redirects for a given URL. It is highly unlikely to locate all such links without access to the link data in Google WMT and Bing WMT, along with the referrer data in GA.

The best you can do to find external URLs is to build a comprehensive backlink report using data from multiple providers (OSE, AHREFs, Majestic, etc). You should know from the start you will not cover all the links unless you are working with pages which have a low number of links.

[CleverPhD]

I agree - screaming frog is an awesome tool for finding the internal 301s.  You can export a spreadsheet of all the pages that contain the 301s etc and so it makes creating a task list to work through (or pass on) pretty straight forward.

[tpt.com]

Thanks Ryan and everyone else, amazing answers So here is my understanding:

For Internal 301:

1. Use Xenu or Screaming to scan url and create the list. I hope we can get a clean report from screaming.

For External:

1. Use Ose to find all backlinks and save them on excel

2. Use AHREFs to find all backlinks and save them

3. Use Majestic to find all backlinks and save them

4. Combine all url and remove duplicate(I guess manually we gotta do that)

Questions

a) How do we find out which one is 301 redirected beside checking each of them?

b) for backlinks we should check what google and bing crawled?

[RyanKent]

You can check the redirects by uploading the LIST of URLs to Screaming Frog, which will then crawl the list and inform you of any header responses (301, 302, etc)

I am unclear on your second question. You previously stated the site involved is not a client and you do not have access to Webmaster Tools. What exactly are you asking or suggesting?

[tpt.com]

Thanks @Ryan

I was referring https://www.google.com/#q=site:domain.com using site:domain.com , I know it doesnt give me all urls but isnt what we care about, at least matching numbers I mean?

[RyanKent]

You are welcome to work with the URLs in search results. I am unsure what numbers you are attempting to match.