The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Moz Tools
    4. Does SeoMoz realize about duplicated url blocked in robot.txt?

    Does SeoMoz realize about duplicated url blocked in robot.txt?

    Moz Tools
    3 3 218
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Viada
      Viada last edited by

      Hi there:

      Just a newby question...

      I found some duplicated url in the "SEOmoz Crawl diagnostic reports" that should not be there.

      They are intended to be blocked by the web robot.txt file.

      Here is an example url (joomla + virtuemart structure):

      _http://www.domain.com/component/users/?view=registration_

      and the here is the blocking content in the robots.txt file

      User-agent: *

      _ Disallow: /components/_

      Question is:

      Will this kind of duplicated url errors be removed from the error list automatically in the future?

      Should I remember what errors should not really be in the error list?

      What is the best way to handle this kind of errors?

      Thanks and best regards

      Franky

      1 Reply Last reply Reply Quote 0
      • Andropenis_Australia
        Andropenis_Australia last edited by

        Don't be too worried about SEOMOZ's errors. Just be aware of them, and if you have done what you need to for the robots file in regards to S.E robots, they should take notice and there shouldn't be any issues. Always be  sure to check GWT for errors, those are the ones you should fix asap.

        1 Reply Last reply Reply Quote 0
        • Abe_Schmidt
          Abe_Schmidt last edited by

          Hello Franky,

          Yes, our crawler obeys robots.txt files. If you recently made that change to your robots then it should reflect in your next crawl. If this error doesn't go away, feel free to let us know help@seomoz.org. Thanks for letting us know!

          -Abe

          1 Reply Last reply Reply Quote 0
          • 1 / 1
          • First post
            Last post
          • Robots.txt blocking Moz
            ChristinaRadisic
            ChristinaRadisic
            0
            6
            291

          • Duplicate URLs
            Discountvc
            Discountvc
            0
            3
            98

          • Blocked by Meta Robots.
            TammyWood
            TammyWood
            0
            4
            1.1k

          • Will moz crawl pages blocked by robots.txt and nofollow links?
            Ryan_Watson
            Ryan_Watson
            0
            2
            184

          • How to remove URLS from from crawl diagnostics blocked by robots.txt
            GrouchyKids
            GrouchyKids
            0
            2
            416

          • Difference in data between http://pro.seomoz.org/tools/keyword-difficulty and http://lsapi.seomoz.com/linkscape/url-metrics/
            claytons
            claytons
            0
            3
            507

          • Seomoz bar: No Follow and Robots.txt
            squareplug
            squareplug
            0
            7
            1.3k

          • Duplicate page error from SEOmoz
            scanlin
            scanlin
            0
            5
            721

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy