The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. How to stop robots.txt restricting access to sitemap?

    How to stop robots.txt restricting access to sitemap?

    Technical SEO Issues
    3 3 181
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Ad-Rank
      Ad-Rank last edited by

      I'm working on a site right now and having an issue with the robots.txt file restricting access to the sitemap - with no web dev to help, I'm wondering how I can fix the issue myself?

      The robots.txt page shows

      User-agent: *
      Disallow: /
      And then sitemap: with the correct sitemap link 
      
      1 Reply Last reply Reply Quote 0
      • Guest
        Guest last edited by

        This post is deleted!
        1 Reply Last reply Reply Quote 0
        • PatrickDelehanty
          PatrickDelehanty last edited by

          Hi there

          Right now, you're telling crawlers to not crawl your entire site, so the sitemap XML would be included in that. Are you wanting your site to be crawled completely? Simply change the robots.txt to this...

          User-agent: *
          Allow: /

          Here is another great resource from SEOBook to check out!

          Hope this helps! Good luck!
          Patrick

          1 Reply Last reply Reply Quote 0
          • 1 / 1
          • First post
            Last post
          • 2 sitemaps on my robots.txt?
            MonkStein
            MonkStein
            0
            6
            9.2k

          • Why is robots.txt blocking URL's in sitemap?
            PurpleGriffon
            PurpleGriffon
            0
            3
            346

          • Have I constructed my robots.txt file correctly for sitemap autodiscovery?
            Bedsite
            Bedsite
            0
            4
            231

          • Will an XML sitemap override a robots.txt
            mememax
            mememax
            0
            5
            2.4k

          • Question about construction of our sitemap URL in robots.txt file
            danatanseo
            danatanseo
            0
            16
            1.6k

          • BEST Wordpress Robots.txt Sitemap Practice??
            evolvingSEO
            evolvingSEO
            0
            2
            1.7k

          • Webmaster tools - Restricted by robots.txt
            RyanKent
            RyanKent
            0
            5
            3.0k

          • Restricted by robots.txt and soft bounce issues (related).
            RyanKent
            RyanKent
            0
            4
            772

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy