Mozbot Can Not Crawl Entire Domain

[LabeliumUSA]

I'm trying to crawl Redken.com in Moz Analytics and the Search Diagnostics is only crawling 4 pages. The domain uses a "select your country" the first time you visit, and it seems as though the bot is not getting beyond that (aka, not clicking on "USA") and is therefore not crawling the rest of the domain. There is no country specific URL other than redken.com.

I've tried entering both "redken.com" and "www.redken.com" as the URL, but no luck.

Any tips?

[DmitriiK]

Hi there.

Which pages are those 4 pages? Is your robots.txt blocking it for some reason maybe?

[LabeliumUSA]

Thanks for the response!

These are the pages it crawled.

<colgroup><col width="420"></colgroup>
| http://redken.com |
| http://www.redken.com/ |
| http://www.redken.com/international/ |
| http://www.redken.com/USA |
| http://www.redken.com/?country=USA |

Robots.txt looks clean, nothing that should have stopped it from crawling more.

[DmitriiK]

Well, I just noticed that website is in flash! I believe non of crawl bots are able to crawl flash websites.

It seems that if I try to access redken.com it redirects me to flash version (/international).

Actually, now I can't recreate that. Super weird. Is there something "special" going on with automatic redirects? Look into that.

[Guest]

This post is deleted!

[DirkC]

It's caused by the way you have build your site. If you click on redken.com - you get the choice of language. If you select "USA" you're redirected with 302 to redken.com/USA - then with 302 to redken.com/?country=USA then with 302 to redken.com I guess for browsers you store this somewhere (cookie?) - however for a simple bot (like Moz - but I have the same with Screaming Frog) - you just go back where you started = redken.com which again will start the same loop.

So - only 4 url's can be crawled. The other countries are on different url's so will not be included in the crawl.

Google bot is smarter and acts more like a real browser so will crawl the site - but Mozbot can't do that.

rgds

Dirk

Update - I actually forgot one redirect - redken.com first is redirected with 302 to redken.com/international

PS The site is horribly slow as well - and the redirect chain is certainly not helping.