Why the number of crawled pages is so low¿?

[levalencia1]

Hi, my website is www.theprinterdepo.com and I have been in seomoz pro for 2 months.

When it started it crawled 10000 pages, then I modified robots.txt to disallow some specific parameters in the pages to be crawled.

We have about 3500 products, so thhe number of crawled pages should be close to that number

In the last crawl, it shows only 1700, What should I do?

[RobertFisher]

levalencia1

What did you disallow?

Are there specific categories or products you know are missing?

Is there a specific sub directory(s) that is missing?

What is it you wanted to block with robots?

[levalencia1]

this is my robots.txt

User-agent: *
Disallow: */product_compare/*
Disallow: *dir=*
Disallow: *order=*

[levalencia1]

ANY IDEA?

[RobertFisher]

Sorry, This one got lost. I will look at it in the a.m. and give you the feedback. Have you run anything like Xenu on the site? Do you know what is not showing up that would be outside of the robots.txt?

[RobertFisher]

Sorry, This one got lost. I will look at it in the a.m. and give you the feedback. Have you run anything like Xenu on the site? Do you know what is not showing up that would be outside of the robots.txt?

[RobertFisher]

levalencia1

Still don't know what you wanted to accomplish with Robots re: I modified robots.txt to disallow some specific parameters in the pages to be crawled.

Go to GWMT: http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449&from=35237&rd=1

This will allow you to determine what your robots.txt accomplished or not:

The Test robots.txt tool will show you if your robots.txt file is accidentally blocking Googlebot from a file or directory on your site, or if it's permitting Googlebot to crawl files that should not appear on the web. When you enter the text of a proposed robots.txt file, the tool reads it in the same way Googlebot does, and lists the effects of the file and any problems found.

Hope it helps you out,

[Cyrus-Shepard]

Hi levelencia1,

This could have been caused by many factors. Was the robots.txt the only change you made? Other things that could have caused it could have been meta "noindex" tags, nofollow links, or broken navigation structures.

In rare instances, sometimes rogerbot has a hiccup.

Let us know if things return to normal on your next crawl. If you have any difficulties feel free to contact the help team (help@seomoz.org) and they should be able to get things straightened out.

Best of luck with your SEO!