XML Sitemap Issue or not?

[Tay1986]

Hi Everyone,

I submitted a sitemap within the google webmaster tools and I had a warning message of 38 issues.

Issue: Url blocked by robots.txt.

Description: Sitemap contains urls which are blocked by robots.txt.

Example: the ones that were given were urls that we don't want them to be indexed: Sitemap: www.example.org/author.xml

Value: http://www.example.org/author/admin/

My issue here is that the number of URL indexed is pretty low and I know for a fact that Robot.txt aren't good especially if they block URL that needs to be indexed. Apparently the URLs that are blocked seem to be URLs that we don't to be indexed but it doesn't display all URLs that are blocked.

Do you think i m having a major problem or everything is fine?What should I do? How can I fix it?

FYI: Wordpress is what we use for our website

Thanks

[SEO5Team]

The first question i would ask is how big is the difference. If the difference is a large in the # of pages on your site and the ones indexed by Google, then you have an issue. The blocked pages might be the ones linking to the ones that have not been indexed and causing issues. Try removing the no follow on those pages and then resubmit your sitemap and see if that fixes the issue. Also double check your site map to make sure you have correctly added all the pages in it.

[Tay1986]

Hi

when generating sitemap there are 46 URLs detected by xml-sitemaps.com but when adding the sitemap to WMT only 12 get submitted and 5 are indexed which is really kind of worrying me. This might be because of the xml sitemap plugin that I installed. May be something is wrong with my settings(doc attached 1&2)

I am kind of lost especially that SEOmoz hasn't detected any URLs blocked by Robot.txt

It would be great if you could tell me what should I do next ?

Thanks

zYmVB.png

[SEO5Team]

Lisa, try checking manually which URL is not getting indexed in Google. Make sure you do not have any no follows on those pages. If all the pages are connected / linked together, then Google will crawl your whole site eventually, just a matter of time.

[evolvingSEO]

Lisa

I would honestly switch to the Yoast SEO plugin. It handles the SEO (and robots.txt) a lot better, as well as the XML sitemaps all within that one plugin.

I'd check out my guide for setting up WordPress for SEO on the moz blog.

Most WP robots.txt files will look like this;

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

And that's it.

You could always just try changing yours to the above setting first, 

before switching to Yoast SEO - I bet that would clear up

the sitemap issues.

Hope that helps!

-Dan
```

[Tay1986]

Hi Dan

Thanks for your answer. Would you really recommend using the plugin instead of just uploading the xml sitemap directly to the website's root directory? If yes why?

Thanks