The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Moz Tools
    4. How to remove URLS from from crawl diagnostics blocked by robots.txt

    How to remove URLS from from crawl diagnostics blocked by robots.txt

    Moz Tools
    2 2 416
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • SimonBond
      SimonBond last edited by

      I suddenly have a huge jump in the number of errors in crawl diagnostics and it all seems to be down to a load of URLs that should be blocked by robots.txt. These have never appeared before, how do I remove them or stop them appearing again?

      1 Reply Last reply Reply Quote 0
      • GrouchyKids
        GrouchyKids last edited by

        Hi Simon,

        Noindex Follow meta tag sounds like the way to go.

        Best to read this first... http://www.seomoz.org/blog/duplicate-content-in-a-post-panda-world

        Hope this helps.

        Justin

        1 Reply Last reply Reply Quote 0
        • 1 / 1
        • First post
          Last post
        • Robots.txt blocking Moz
          ChristinaRadisic
          ChristinaRadisic
          0
          6
          291

        • Block Moz (or any other robot) from crawling pages with specific URLs
          Blacktie
          Blacktie
          0
          3
          844

        • Website blocked by Robots.txt in OSE
          edlondon
          edlondon
          0
          4
          190

        • Will moz crawl pages blocked by robots.txt and nofollow links?
          Ryan_Watson
          Ryan_Watson
          0
          2
          184

        • Meta-Robots noFollow and Blocked by Meta-Robots
          gfiedel
          gfiedel
          0
          4
          223

        • Does SeoMoz realize about duplicated url blocked in robot.txt?
          Abe_Schmidt
          Abe_Schmidt
          0
          3
          218

        • HTTPS site, server root Robots.txt and Crawl errors
          KeriMorgret
          KeriMorgret
          0
          3
          390

        • Why is Roger crawling pages that are disallowed in my robots.txt file?
          MeltButterySpread
          MeltButterySpread
          0
          5
          928

        Get started with Moz Pro!

        Unlock the power of advanced SEO tools and data-driven insights.

        Start my free trial
        Products
        • Moz Pro
        • Moz Local
        • Moz API
        • Moz Data
        • STAT
        • Product Updates
        Moz Solutions
        • SMB Solutions
        • Agency Solutions
        • Enterprise Solutions
        • Digital Marketers
        Free SEO Tools
        • Domain Authority Checker
        • Link Explorer
        • Keyword Explorer
        • Competitive Research
        • Brand Authority Checker
        • Local Citation Checker
        • MozBar Extension
        • MozCast
        Resources
        • Blog
        • SEO Learning Center
        • Help Hub
        • Beginner's Guide to SEO
        • How-to Guides
        • Moz Academy
        • API Docs
        About Moz
        • About
        • Team
        • Careers
        • Contact
        Why Moz
        • Case Studies
        • Testimonials
        Get Involved
        • Become an Affiliate
        • MozCon
        • Webinars
        • Practical Marketer Series
        • MozPod
        Connect with us

        Contact the Help team

        Join our newsletter
        Moz logo
        © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
        • Accessibility
        • Terms of Use
        • Privacy