How to stop robots.txt restricting access to sitemap?
-
I'm working on a site right now and having an issue with the robots.txt file restricting access to the sitemap - with no web dev to help, I'm wondering how I can fix the issue myself?
The robots.txt page shows
User-agent: * Disallow: / And then sitemap: with the correct sitemap link -
This post is deleted! -
Hi there
Right now, you're telling crawlers to not crawl your entire site, so the sitemap XML would be included in that. Are you wanting your site to be crawled completely? Simply change the robots.txt to this...
User-agent: *
Allow: /Here is another great resource from SEOBook to check out!
Hope this helps! Good luck!
Patrick