The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. Why is robots.txt blocking URL's in sitemap?

    Why is robots.txt blocking URL's in sitemap?

    Technical SEO Issues
    3 2 346
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • PurpleGriffon
      PurpleGriffon last edited by

      Hi Folks,

      Any ideas why Google Webmaster Tools is indicating that my robots.txt is blocking URL's linked in my sitemap.xml, when in fact it isn't?

      I have checked the current robots.txt declarations and they are fine and I've also tested it in the 'robots.txt Tester' tool, which indicates for the URL's it's suggesting are blocked in the sitemap, in fact work fine.

      Is this a temporary issue that will be resolved over a few days or should I be concerned.

      I have recently removed the declaration from the robots.txt that would have been blocking them and then uploaded a new updated sitemap.xml. I'm assuming this issue is due to some sort of crossover.

      Thanks

      Gaz

      1 Reply Last reply Reply Quote 0
      • MattAntonino
        MattAntonino last edited by

        Yes, it should clear up by itself after a day or two.  Google will need to recrawl the robots.txt file next time they visit your site.  It should clear up then.  If you make immediate changes and try to resubmit a sitemap, you almost always get this issue.

        PurpleGriffon 1 Reply Last reply Reply Quote 1
        • PurpleGriffon
          PurpleGriffon @MattAntonino last edited by

          Brilliant, thanks for the clarification Matt. Much appreciated. 🙂

          1 Reply Last reply Reply Quote 0
          • 1 / 1
          • First post
            Last post
          • "Url blocked by robots.txt." on my Video Sitemap
            sergeystefoglo
            sergeystefoglo
            0
            3
            521

          • Strange URL's for client's site
            everestagency
            everestagency
            0
            3
            456

          • Sitemap url's not being indexed
            GreenStone
            GreenStone
            0
            7
            158

          • Sitemap issue? 404's & 500's are regenerating?
            jeff-rackaid.com
            jeff-rackaid.com
            0
            5
            263

          • We have designed a new site and are in dilemna whether or not to change the site's URL structure or maybe few odd looking urls. How exactly do we go about the URL thing in toto? thanks in advance, any suggestion(s) would be dearly welcome
            ShaunPhilips
            ShaunPhilips
            1
            7
            109

          • What's the best way to handle Overly Dynamic Url's?
            GKLA
            GKLA
            0
            2
            309

          • Blocked URL's by robots.txt
            meralucian37
            meralucian37
            0
            7
            398

          • Switching ecommerce CMS's - Best Way to write URL 301's and sub pages?
            tylerfraser
            tylerfraser
            0
            7
            1.2k

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy