Robots.txt versus sitemap

[anthematic]

Hi everyone,

Lets say we have a robots.txt that disallows specific folders on our website, but a sitemap submitted in Google Webmaster Tools that lists content in those folders.

Who wins? Will the sitemap content get indexed even if it's blocked by robots.txt? I know content that is blocked by robot.txt can still get indexed and display a URL if Google discovers it via a link so I'm wondering if that would happen in this scenario too.

Thanks!

[SarahGoliger]

For Google, that content will not get indexed.

Robots will win the fight of Robots vs Sitemap, as it says "Don't access or index this content, even if you find a way into it". Sitemap.xml is helping them find their way to content that they won't access or index.

Bing and other engines may be different on this. I'm not sure. I would guess that Bing at least will also respect Robots over sitemap (as it seems the proper behavior), but I have not tried this ever.

[RobMay]

I would also take the time to clean up your XML Sitemap file for crawling, just in case. It'll be better for you to keep track of any files/URL's you don't want indexed by the search bots.

Just good practice