The "webmaster" disallowed all ROBOTS to fight spam! Help!!

[JoshuaLindley]

One of the companies I do work for has a magento site.  I am simply the SEO guy and they work the website through some developers who hold access to their systems VERY tightly.  Using Google Webmaster Tools I saw that the robots.txt file was blocking ALL robots.

I immediately e-mailed out and received a long reply about foreign robots and scrappers slowing down the website.  They told me I would have to provide a list of only the good robots to allow in robots.txt.

Please correct me if I'm wrong.. but isn't Robots.txt optional?? Won't a bad scrapper or bot still bog down the site?  Shouldn't that be handled in httaccess or something different?

I'm not new to SEO but I'm sure some of you who have been around longer have run into something like this and could provide some suggestions or resources I could use to plead my case!

If I'm wrong.. please help me understand how we can meet both needs of allowing bots to visit the site but prevent the 'bad' ones.  Their claim is the site is bombarded by tons and tons of bots that have slowed down performance.

Thanks in advance for your help!

[Travis_Bailey]

Yes, there are a ton of bad bots one may want to block. Can you show us the robots.txt file? If they aren't blocking legit search engine bots, you're probably okayish. If they are actually blocking all bots, you have cause for concern.

Can you give us a screenshot from GWT?

I use a program called Screaming Frog daily. It's not malicious, off the shelf. I just want to crawl and gather meta data. I can tell it to disregard robots.txt. It will crawl a site until it hit's something password protected. There's not much any robots.txt can do about it, as it can also spoof user agents.

[Martijn_Scheijbeler]

What you could do is just add Allow statements for the different Googlebots and the bots of other search engines. This will probably make the developers happy so they can keep other bots out of the door (although I doubt this would work and definitely don't think that this should be the option to keep spammers away, but that says more about the quality of development ;-)).

[justin-brock]

You can get the list of good robots from the list at Robotstxt.org: http://www.robotstxt.org/db.html.

I'd recommend creating an edited version of the robots.txt file yourself, specifically Allowing googlebot and others. Then send that with a link to the robotstxt.org site.

You may need to get the business owners involved. IT exists to enable the business, not strap it down so it can't move.

[JoshuaLindley]

Thanks for the suggestions!! I'll keep you updated.