Google Search console says 'sitemap is blocked by robots?

[Extima-Christian]

Google Search console is telling me "Sitemap contains URLs which are blocked by robots.txt."

I don't understand why my sitemap is being blocked? My robots.txt look like this:

User-Agent: *
Disallow:

Sitemap: http://www.website.com/sitemap_index.xml

It's a WordPress site, with Yoast SEO installed. Is anyone else having this issue with Google Search console? Does anyone know how I can fix this issue?

[Alick300]

Hi,

Use this plugin

https://wordpress.org/plugins/wp-robots-txt/

it will remove previous robots.txt and set simple wordpress robots.txt and wait for a day

problem can be solved.

Also watch this video on the same @ https://www.youtube.com/watch?v=DZiyN07bbBM

Thanks

[BlueprintMarketing]

No, you do not need to change or plug-in what is happening is Webmaster tools is telling you that you have no index or no follow were robots xTag somewhere on your URLs inside your sitemap.

Run your site through Moz, screaming frog Seo spider or deepcrawl and look for no indexed URLs.

webmaster tools/search console is telling you that you have no index URLs inside of your XML sitemap not that you robots.txt is blocking it. This would be set in the Yoast plugin. one way to correct it is to look for noindex URLs &  filter them inside Yoast so they are not being presented to the crawlers.

If you would like you can turn off the sitemap on Yoast and turn it back on if that does not work I recommend completely removing the plug-in and reinstalling it

• https://kb.yoast.com/kb/how-can-i-uninstall-my-plugin/
• https://kinsta.com/blog/uninstall-wordpress-plugin/

Can you send a screenshot of what you're seeing?

When you see it in Google Webmaster tools are you talking about the XML sitemap itself mean no indexed because all XML sitemaps are no indexed.

Please add this to your robots.txt

`User-agent:*
Disallow:/wp-admin/
Allow:/wp-admin/admin-ajax.php` Sitemap: http://www.website.com/sitemap_index.xml

I hope this is of help,

Tom

[Extima-Christian]

Thank you for the response.

I just scanned the site using 'Screaming frog'. Under Internal>Directives there were zero 'no index' links. I also check for '404 errors', server 505 errors, or anything 'blocked by robots.txt'.

Google search console is still showing me that there are URL's being blocked by my sitemap. (I added a screenshot of this). When I click through, it tells me that the 'post sitemap' has over +300 warnings.

I have just deleted the YoastSEO plugin, and I am now re-installing it. hopefully, this fixes the issue.

B7KE6nc ShhJY55

[Extima-Christian]

Thanks for the response. Do you think this is a robots.txt issue? Or could this be caused by the YoastSEO plugin?

Do you know if this plug-in works with YoastSEO together? Or will it cause issues?

[BlueprintMarketing]

I think you need to make an https robots.txt file if you are running https if running https

https://moz.com/blog/xml-sitemaps

`User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php`

Sitemap: https://domain.com/index-sitemap.xml

(that is a https site map)

can you send the sitemap URL or run it though deepcrawl

Hope this helps?

Did you make a new robots.txt file?

ShhJY55.png

[davebuts]

Hi Christian,

The current robots.txt shouldn't be blocking those URLs.

Did you or someone else recently change the robots.txt file? If so, give Google a few days to re-crawl your site.

Also, can you check what happens when you do a fetch and render on one of the blocked posts in Search Console? Do you have issues there?

Cheers,

David

[Extima-Christian]

Hey guys, I ended up disabling the sitemap option from YoastSEO, then installed the 'Google (XML) sitemap' plug-in. I re-submitted the sitemap to Google last night, and it came back with no issues. I'm glad to finally have this sorted out.

Thanks for all the help!

[BlueprintMarketing]

I agree with David

Hey is your dev Greg Reindel? If so you can call me for help PM me here for my info.

Thomas Zickell

[BlueprintMarketing]

Nice happy to hear that do you work with Greg Reindel? He is a good friend I looked at your IP that is why I ask?

Tom