The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. How to stop robots.txt restricting access to sitemap?

    How to stop robots.txt restricting access to sitemap?

    Technical SEO Issues
    3 3 181
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Ad-Rank
      Ad-Rank last edited by

      I'm working on a site right now and having an issue with the robots.txt file restricting access to the sitemap - with no web dev to help, I'm wondering how I can fix the issue myself?

      The robots.txt page shows

      User-agent: *
      Disallow: /
      And then sitemap: with the correct sitemap link 
      
      1 Reply Last reply Reply Quote 0
      • Guest
        Guest last edited by

        This post is deleted!
        1 Reply Last reply Reply Quote 0
        • PatrickDelehanty
          PatrickDelehanty last edited by

          Hi there

          Right now, you're telling crawlers to not crawl your entire site, so the sitemap XML would be included in that. Are you wanting your site to be crawled completely? Simply change the robots.txt to this...

          User-agent: *
          Allow: /

          Here is another great resource from SEOBook to check out!

          Hope this helps! Good luck!
          Patrick

          1 Reply Last reply Reply Quote 0
          • 1 / 1
          • First post
            Last post
          • Is sitemap required on my robots.txt?
            LoganRay
            LoganRay
            0
            4
            7.3k

          • Adding multi-language sitemaps to robots.txt
            Schwaab
            Schwaab
            0
            2
            5.1k

          • Have I constructed my robots.txt file correctly for sitemap autodiscovery?
            Bedsite
            Bedsite
            0
            4
            231

          • Will an XML sitemap override a robots.txt
            mememax
            mememax
            0
            5
            2.4k

          • Oh no googlebot can not access my robots.txt file
            BistosAmerica
            BistosAmerica
            0
            9
            6.3k

          • Robots.txt versus sitemap
            RobMay
            RobMay
            0
            3
            1.7k

          • Webmaster tools - Restricted by robots.txt
            RyanKent
            RyanKent
            0
            5
            3.0k

          • Robots.txt and robots meta
            TheEspresseo
            TheEspresseo
            0
            5
            1.1k

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy