SEOMOZ crawl all my pages

[hcetinsoy]

SEOMOZ crawl all my pages including ".do" (all web pages after sign up )

. Coz of this it finishes all my 10.000 crawl page quota  and be exposed to dublicate pages.

Google is not crawling pages that user reach after sign up. Because these are private pages for customers I guess

The main question is how we can limit SEOMOZ crawl bot. If the bot can stay out of ".do" java extensions it'll perfect to starting SEO analysis.

Do you know think about it?

Cheers

Example;

.do java extension (after sign up page) (Google can't crawl)

http://magaza.turkcell.com.tr/showProductDetail.do?psi=1001694&shopCategoryId=1000021&model=Apple-iPhone-3GS-8GB

Normal Page (Google can crawl)

http://magaza.turkcell.com.tr/telefon/Apple-iPhone-3GS-8GB/1001694/.html

[ChiarynMiranda]

Hi There,

Thanks for writing in and sorry for the confusion.

It actually isn't possible for the SEOmoz crawler to access pages that require a user login. I went to those URLs and I was able to access the pages with out be logged in as a user, so they don't require user sign up to access them. Since these pages are linked to by other pages on your site and our crawler is not being blocked from these pages and the pages don't actually require a user to be signed in to access them, we will crawl them. I can't say why Google wouldn't be crawling those pages, but there is definitely nothing in place that would stop our crawler from accessing them.

If you would like to stop our crawler from accessing those pages in the future, you may consider adding a disallow directive in your robots.txt file using the user-agent rogerbot.

I hope this helps. Let me know if you have any other questions.

Chiaryn