The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. Robots.txt

    Robots.txt

    Technical SEO Issues
    5 2 99
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Dan-Lawrence
      Dan-Lawrence last edited by

      I have a client who after designer added a robots.txt file has experience continual growth of urls blocked by robots,tx but now urls blocked (1700 aprox urls) has surpassed those indexed (1000).  Surely that would mean all current urls are blocked (plus some extra mysterious ones). However pages still listing in Google and traffic being generated from organic search so doesnt look like this is the case apart from the rather alarming webmaster tools report

      any ideas whats going on here ?

      cheers

      dan

      1 Reply Last reply Reply Quote 0
      • KevinBudzynski
        KevinBudzynski last edited by

        The url's may be blocked for indexing, but if they were indexed at one time there is a lag time before they are typically removed from the organics. Is this is what you were asking?

        1 Reply Last reply Reply Quote 1
        • Dan-Lawrence
          Dan-Lawrence last edited by

          Thanks for taking time to comment Kevin

          No - what i'm worried about is if more urls are blocked than indexed doesnt that mean that all site pages are blocked (when at least 50 of them dont want to be blocked since want to be crawled/indexed) ? in addition how can more urls be blocked than those that exist/indexed ?

          cheers
          Dan

          1 Reply Last reply Reply Quote 0
          • KevinBudzynski
            KevinBudzynski last edited by

            Do you use any type of dynamic parameters?

            1 Reply Last reply Reply Quote 0
            • Dan-Lawrence
              Dan-Lawrence last edited by

              Not as far as I'm aware

              1 Reply Last reply Reply Quote 0
              • 1 / 1
              • First post
                Last post
              • Robots.txt
                MarieHaynes
                MarieHaynes
                0
                8
                115

              • Robots.txt
                MichaelC-15022
                MichaelC-15022
                0
                7
                1.0k

              • Robots.txt
                WesleySmits
                WesleySmits
                0
                9
                168

              • Question about Robot.txt
                paumer80
                paumer80
                0
                5
                379

              • Do I need robots.txt and meta robots?
                Cyrus-Shepard
                Cyrus-Shepard
                0
                7
                1.1k

              • Robots txt
                LadyApollo
                LadyApollo
                0
                3
                427

              • Robots.txt not working?
                levalencia1
                levalencia1
                0
                4
                465

              • Robots.txt and robots meta
                TheEspresseo
                TheEspresseo
                0
                5
                1.1k

              Get started with Moz Pro!

              Unlock the power of advanced SEO tools and data-driven insights.

              Start my free trial
              Products
              • Moz Pro
              • Moz Local
              • Moz API
              • Moz Data
              • STAT
              • Product Updates
              Moz Solutions
              • SMB Solutions
              • Agency Solutions
              • Enterprise Solutions
              • Digital Marketers
              Free SEO Tools
              • Domain Authority Checker
              • Link Explorer
              • Keyword Explorer
              • Competitive Research
              • Brand Authority Checker
              • Local Citation Checker
              • MozBar Extension
              • MozCast
              Resources
              • Blog
              • SEO Learning Center
              • Help Hub
              • Beginner's Guide to SEO
              • How-to Guides
              • Moz Academy
              • API Docs
              About Moz
              • About
              • Team
              • Careers
              • Contact
              Why Moz
              • Case Studies
              • Testimonials
              Get Involved
              • Become an Affiliate
              • MozCon
              • Webinars
              • Practical Marketer Series
              • MozPod
              Connect with us

              Contact the Help team

              Join our newsletter
              Moz logo
              © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
              • Accessibility
              • Terms of Use
              • Privacy