The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. What are the negative implications of listing URLs in a sitemap that are then blocked in the robots.txt?

    What are the negative implications of listing URLs in a sitemap that are then blocked in the robots.txt?

    Technical SEO Issues
    3 3 153
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • richdan
      richdan last edited by

      In running a crawl of a client's site I can see several URLs listed in the sitemap that are then blocked in the robots.txt file.

      Other than perhaps using up crawl budget, are there any other negative implications?

      1 Reply Last reply Reply Quote 0
      • LesleyPaone
        LesleyPaone last edited by

        I personally do not think there is any penalty SEO wise in doing it. Although, I do think it will mess up the metric in GWT that shows how many pages have been submitted and how many have been indexed. I find that metric useful, so it would make it no longer useful if there are a lot of pages blocked by the robots.txt.

        1 Reply Last reply Reply Quote 1
        • Dezzign
          Dezzign last edited by

          I highly doubt it would effect rankings due to low quality issues but it will show that you have site map error warnings in your GWT console. That issue is technically classified as 'Warnings' and not 'Errors'. The right thing to do in that scenario is take the robots.txt block off and just use a 'noindex' tag on the pages. That way they can stay in the site map but they won't show up in the index. Otherwise you should remove them from the sitemap if you don't want the warnings in GWT.

          1 Reply Last reply Reply Quote 1
          • 1 / 1
          • First post
            Last post
          • Is sitemap required on my robots.txt?
            LoganRay
            LoganRay
            0
            4
            7.3k

          • Why is robots.txt blocking URL's in sitemap?
            PurpleGriffon
            PurpleGriffon
            0
            3
            346

          • Decrease in index URLs, increase in URLs blocked by robots and drop in traffic
            pugh
            pugh
            0
            6
            2.4k

          • "Extremely high number of URLs" warning for robots.txt blocked pages
            KristinaKledzik
            KristinaKledzik
            0
            8
            380

          • Same URL in "Duplicate Content" and "Blocked by robots.txt"?
            alsvik
            alsvik
            0
            3
            502

          • Robots.txt versus sitemap
            RobMay
            RobMay
            0
            3
            1.7k

          • Blocking robots.txt
            de4e
            de4e
            0
            4
            432

          • Blocking other engines in robots.txt
            RyanKent
            RyanKent
            0
            2
            581

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy