Why do we have so many pages scanned by bots (over 250,000) and our biggest competitors have about 70,000? Seems like something is very wrong.

[achituv]

We are trying to figure out why last year we had a huge (80%) and sudden (within two days) drop in our google searches.  The only "outlier" in our site that we can find is a huge number of pages reported in MOZ as scanned by search engines.  Is this a problem?  How did we get so many pages reported?  What can we do to bring the number of searched pages back to a "normal" level?

BT

[Stewart_SEO]

Have you looked at your competitors robots.txt file? they are probably blocking the very same searches you are talking about. if there is a particular bot like a Chinese crawler for example baidu that you don't want to come to your site you can block them via the command: User-agent: Baiduspider

[achituv]

Stewart_SEO

Thanks for your quick response.  We did review the robots.txt of the competitors.  Not line by line - they took surprisingly different approaches to the robots.txt.  But there were the usual exclusions for wish lists, etc.  We've gone back and tightened up our robots.txt and haven't yet seen any changes.  Several months ago we were at about 600,000 pages and it is dropping.  Very mysterious.

[danatanseo]

Hi. A mystery indeed! Have you recently upgraded or changed Web platforms or changed or upgraded what you are using for your site navigation?