The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. Impact of "restricted by robots" crawler error in WT

    Impact of "restricted by robots" crawler error in WT

    Technical SEO Issues
    3 3 314
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • phogan
      phogan last edited by

      I have been wondering about this for a while now with regards to several of my sites.  I am getting a list of pages that I have blocked in the robots.txt file.  If I restrict Google from crawling them, then how can they consider their existence an error?  In one case, I have even removed the urls from the index.

      And do you have any idea of the negative impact associated with these errors.

      And how do you suggest I remedy the situation.

      Thanks for the help

      1 Reply Last reply Reply Quote 0
      • ClickConsult
        ClickConsult last edited by

        Hi Patrick,

        That section is simply there to advice on any URLs that Google feels are wrongly excluded within the robots.txt

        If the URLs are not wrongly excluded, don't worry about it showing in WMT's - it's there just as an advisory.

        Good luck!

        KeriMorgret 1 Reply Last reply Reply Quote 2
        • KeriMorgret
          KeriMorgret @ClickConsult last edited by

          Google is just showing you a warning that hey, these are excluded, make sure that you want them excluded. They're not passing a judgement on whether or not they should be excluded. So, as long as they're excluded on purposes, no worries.

          1 Reply Last reply Reply Quote 1
          • 1 / 1
          • First post
            Last post
          • "Moz encountered an error on one or more pages on your site" Error
            Hurf
            Hurf
            0
            4
            61

          • Robots.txt - "File does not appear to be valid"
            PeaSoupDigital
            PeaSoupDigital
            0
            3
            319

          • Rel="publisher" validation error in html5
            Andy.Drinkwater
            Andy.Drinkwater
            0
            7
            1.4k

          • Best action to take for "error" URLs?
            ramangarg
            ramangarg
            0
            2
            172

          • Will I still get Duplicate Meta Data Errors with the correct use of the rel="next" and rel="prev" tags?
            irvingw
            irvingw
            0
            6
            911

          • 404 crawl errors from "tel:" link?
            loopyal
            loopyal
            0
            3
            4.0k

          • Should we use "and" or "&"?
            IPROdigital
            IPROdigital
            0
            9
            3.2k

          • Google Crawler Error / restricting crawling
            Cyrus-Shepard
            Cyrus-Shepard
            0
            2
            628

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy