Possible Cause/Fix for 500 404 Errors?

[Leadhub]

This post is deleted!

[CommT]

Aaron,

I'm not 100% sure what to do about the 404 problem, but you should be able to block the robot from this site with your robots.txt page. You'll have to find out what the robot is called though and they may have more than one so it could be a needle in the haystack kind of situation.

Otherwise, you could 301 redirect all these 404 pages to the correct page, but I'm not sure this is the best thing to do if it keeps on happening - though if you block it from accessing your site then it would stop, I think.

Best of luck finding a solution, I'd be interested in what others have to say on the subject.

Amelia

[JaneCopland]

Hi Aaron,

Definitely try get that bot blocked. Can you see its activity in raw log files if it's scraping your site?

[Leadhub]

I haven't been able to find the bot to get it blocked just yet, but that is something I need to look in to.

The site we designed is really popular within their industry and I know there have been a lot of franchises contacting us about duplicating the code base or theme (which we won't do). So I can only assume someone was trying to use this tool to recreate our site.

My quick fix was to traceroute their website's ip address and blacklist it on my server. If I visit the old url's where my code was previously being output (with links to my pages) they are now broken on the Japanese website. I'm happy with that for now.

[JaneCopland]

Hi Aaron,

Glad you sorted out a fix for now - I couldn't find a bot name either (and people who are simply thieving aren't likely to obey robots.txt either).

Let us know if we can help more.

Cheers,

Jane