How to know bad bots or spiders scraping my website?

[Trigun]

This post is deleted!

[stefanok]

Sorry to hear about this, this can really cause a big headache. I came across these three articles which im sure you would find helful.

1. http://perishablepress.com/press/2010/09/24/content-scrapers-suck-ass/

Basically using HTACESS to block these domains. here is an excerpt:

"These are the tools I use when dealing with content scrapers. For bigger sites like DigWP.com, I agree with Chris that no action is really required. As long as you are actively including plenty of internal links in your posts, scraped content equals links back to your pages. For example, getting a link in a Smashing Magazine article instantly provides hundreds of linkbacks thanks to all of thieves and leeches stealing Smashing Mag’s content. Sprinkling a few internal links throughout your posts benefits you in some fantastic ways:

• Provides links back to your site from stolen/scraped content
• Helps your readers find new and related pages/content on your site
• Makes it easy for search engines to crawl deeply into your site"

2.http://www.famousbloggers.net/content-scrapers-thieves.html

basically identify them. Contact site owners, hosting services etc.

"What Can You Do When You Catch Someone Stealing Your Content? In addition to several basic steps that you can immediately take, there are also a few extra tricks you can use to protect your content: Contact the blog or website’s owner and politely ask them to remove the stolen content. 95% of the time, this has been the only step I’ve needed to take. You can use the Whois Lookup from Domain Tools to help you find the blog or website’s owner contact information. On the rare occasions when this isn’t successful, move on to the next steps.Contact Google and file a Digital Millennium Copyright Act (DMCA) complaint. In addition to Google giving your site credit for the original content, filing a DMCA complaint may result in Google completely removing a blog or website that is full of stolen content from their index. You can also file a Spam Report with Google to help fight back against content thieves.Contact the blog or website’s hosting company and file a Digital Millennium Copyright Act (DMCA) complaint. Hosting companies are required by law to shut down the blog or website until the stolen content is removed. Most reputable hosting companies already have procedures in place for lodging your DMCA complaints with their security or abuse departments. The key to successfully using this technique is that you will need to prove to the hosting company that you were the first one to publish the content. A simple and effective way to do this is by using the free Wayback Machine from Archive.org. This technique has worked for me on several occasions when a blog or website owner refused to remove the stolen content on their own."

3. Great 5 minute solution: http://blog.effortlessebookwriting.com/my-blog-content-was-stolen-here-is-an-effective-5-minute-solution-to-that/

Hope that this helps you. let me know, regards Stef

[Distil]

I agree with Stefano, this can be a huge headache.  Since Google implemented the latest iteration of their search algorithms (Panda) there have been serios implications for duplicate content on the web.   You now are penalized for the duplicated content and will lose pangerank.

We always recommend all our clients take immediate action against offending sites.  The first step is contacting the site.  If that bears no results then you should absolutely file DMCA complaints with Google, their hosting provider, and any advertisers on their site.

You should also take steps to prevent the continued scraping of your site.  Another good article for reference:
http://www.blueglass.com/blog/content-scraping-prevention-benefits/

HTACESS is a great start but scraping is a business, and the scrapers have gotten very good at masquerading their theft. That is why there are services, such as ours, that can help protect you against site scraping.

Rami Founder, CEO
www.distil.it

[Trigun]

Hi Stefano,

First of all, I would like to thank you for the help.  I really appreciate it.

I have tried to implement the (http://perishablepress.com/press/2009/03/29/4g-ultimate-user-agent-blacklist/) and it seems that it minimized the scrapers visiting my site.  I paired it with blocking specific domains and IPs.

Once this are applied,  an error occurs every time I add a picture into my posts.   What do I need to remove in the list of blocked bots to allow the adding of pictures?

by the way, I am using wordpress as my CMS.

Thanks in advance...