Sitemap Warnings

[Aggie]

Due to an issue with our CMS, I had a bunch of URL aliases that were being indexed and causing duplicate content issues.

I disallowed indexing of the bad URLs (they all had a similar URL structure so that was easy).  I did this until I could clean up the bad URLs

I then recieved a bunch of sitemap warnings that the URLs that I blocked URLs with robots.txt that were in the sitemap.

Isn't this the point of robots.txt?  Why am I getting warnings and how can I get rid of them?

[irvingw]

you are not supposed to include blocked URLs in the sitemap.xml files, or Google considers it wasting their crawl time. Are these automated sitemap.xml files?

You're basically saying "come index these pages i've listed, but don't index them!"

Remove the URLs that are blocked content (or rerun/regenerate them) and resubmit the sitemaps and the warnings will go away.

[pattersonla]

The purpose of your sitemap is to tell Google to go out and index the pages you specify. The purpose of the robots.txt is to tell Google not to index the page. The warning is likely just a precaution to let you know that you may have by accident requested them to block something in robots.txt. If you remove the URL's from your submitted sitemap the warnings should disappear. If you leave them, you will have warnings but Google should not index the content since your blocked it in robots.txt.

[Aggie]

Irving,

Totally get that and we're working to ensure they are no longer included in the sitemap.

Thanks,

Lisa

[Aggie]

Ok, so we took the restriction out of robots.txt while IT tries to fix the issue of URLs showing up on the sitemap that shouldn't.

Warnings haven't fallen off and now our sitemap is a day behind now as it's stuck in pending for almost a full day.

Any thoughts on what might be causing? I'm assuming this is impacting what's indexed and hurting our site.

[Aggie]

Irving -

Ok, so we took the restriction out of robots.txt while IT tries to fix the issue of URLs showing up on the sitemap that shouldn't.

Warnings haven't fallen off and now our sitemap is a day behind now as it's stuck in pending for almost a full day.

Any thoughts on what might be causing? I'm assuming this is impacting what's indexed and hurting our site.