Do you add 404 page into robot file or just add no index tag?

[Rubix]

Hi,

got different opinion on this so i wanted to double check with your comment is.

We've got /404.html page and I was wondering if you would add this page to robot text so it wouldn't be indexed or would you just add no index tag? What would be the best approach?

Thanks!

[Martijn_Scheijbeler]

Hi Sida,

I would add a noindex to the page and as you also will return the 404 status code this is enough data for Google to tell not to index the page itself.

Hope this answers your question.

[Rubix]

Thanks Martijn,

I actually want to know what would you do for the 404 page itself. It is something like:

www.mainurl.com/404.html and for some reason this started to create some other links such as

www.mainrul.com/404.html?aspxerrorpath=/contact-office.aspx

Do you think I should add 404 page and subpages to Robot.txt ?

Thanks!

[Saijo.George]

404 are OK on your site just make sure you send the proper 404 header response for the 404 page ... Google does NOT index 404 pages ( as long as it sends the 404 header response ) , so you don't need to block them via robots.txt or meta robots.

Infact GWT warns you about these if they are able to crawl the so called 404 pages that doesn't send a 404 header response , so I think its a good idea NOT to noindex them you will get the warning if something is wrong.

Google will only index your 404 if you don't do that..they call it soft 404 : https://support.google.com/webmasters/answer/181708?hl=en

worth reading : http://moz.com/learn/seo/http-status-codes

[Everett]

Hello Rubix,

Saijo gave you some great advice, but I'm concerned about the fact that you have that page in the first place, and that it produces those URL parameters. It suggests to me that instead of showing a 404 error on the contact-office.aspx page (assuming that pages doesn't exist on that URL) you are redirecting the user who tries to access that URL to the /404.html page (e.g. /404.html?aspxerrorpath=/contact-office.aspx).

Typically you want the 404 http status code to show on the URL the user is trying to unsuccessfully access. In this case instead of redirecting them to your "404 page URL" you would want to show your customized 404 message (and ensure it returns a 404 status code, use this tool) on www.yourdomain.com/contact-office.aspx.

I hope this makes sense to you. If not, feel free to ask for clarification.