Problem crawling a website with age verification page.

[catalinmoraru]

Hy every1,

Need your help very urgent. I need to crawl a website that first has a page where you need to put your age for verification and after that you are redirected to the website. My problem is that SEOmoz, crawls only that first page, not the whole website. How can I crawl the whole website?, do you need me to upload a link to the website?

Thank you very much

Catalin

[Martijn_Scheijbeler]

Hi Catalin,

The best way do to this is of course to include a link to the rest of the Web site (you could remove the link of course when Roger came by). But what you also could is redirect the user based on the user agent when linking wouldn't be an option.

Hope this helps!

[catalinmoraru]

OK, thank you for your short answer, but the thing is I didn't understand anything from what you wrote :).

I want to add that I do not own the website. I dont have acces to back-end, cms, etc. The client just wants me to crawl the whole website to see if something is wrong. I can see with my own eyes that the website has duplicate content, but seomoz doesnt crawls the website, because of that first page with verification.

[Martijn_Scheijbeler]

Well that's a small side note to your problem ;-), are you able to just set up a crawl for a sub folder? Or do you have to pass the verification at all times?

[catalinmoraru]

the problem is that the pages are not in a subfolder. I have to pass the verification page every time :(. SEOMoz is crawling only the first page.

[Abe_Schmidt]

Hello Catalin,

Our crawler will not be able to get past an age verification page. You will need to find or unlock a subfolder or subdomain  to bypass this if you would like our crawlers to be able to get through. Luckily, Google's crawlers are a bit more thorough a will be able to index your site properly. We are hoping to add this ability soon and I hope you can find a way for us to get through in the meantime.