Disallow: /404/ - Best Practice?

[niconico101]

Hello Moz Community,

My developer has added this to my robots.txt file: Disallow: /404/

Is this considered good practice in the world of SEO? Would you do it with your clients?

I feel he has great development knowledge but isn't too well versed in SEO.

Thank you in advanced,

Nico.

[LesleyPaone]

This comes down to personal preference in my opinion. I think honestly it is neither here nor there. The instances that your 404 page could come up in the SERP's is more than likely pretty low and at the same time it really does not offer any useful content. So disallowing it would not really be any loss. One reason why it might be disallowed is that you have an e-commerce site that rotates products. When a product is deleted, the developer has a 301 to the 404 page, then with the robots.txt saying not to index the 404 page, the other page will drop out of search engines. If this is the case I would rethink that strategy. If you notice a lot of sites like amazon and other big sites leave the page in the index even if the product is no longer for sale. The thought is traffic is traffic, the hardest part in the whole equation is getting someone to your site, if the page is ranking, why delete it.

The only time I can think that I would specifically allow it and optimize it is if you have a cool 404 page. Some companies actually spend a bit of time on their pages and it gets them a little pop of viral traffic from social sharing sites like reddit. If you do have one that is funny or unique I would allow it and actually optimize it for a term like "funny 404 page" or something like that.

[niconico101]

Thank you Lesley.

This really helps a lot. I appreciate it very much. This is my site by the way: http://devilswink.com/

Thanks.

Nico.