The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. Blocked URL's by robots.txt

    Blocked URL's by robots.txt

    Technical SEO Issues
    7 3 398
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • meralucian37
      meralucian37 last edited by

      In Google Webmaster Tools shows me 10,936 Blocked URL's by robots.txt and it is very strange when you go to the "Index Status" section where shows that since April 2012 robots.txt blocked many URL's.

      You can see more precise on the image attached (chart WMT)

      I can not explain why I have blocked URL's ? because I have nothing in robots.txt.
      My robots.txt is like this:

      User-agent: *

      I thought I was penalized by Penguin in April 2012 because constantly i'am losing visitors now reaching over 40%. It may be a different penalty?

      Any help is welcome because i'm already so saturated.

      Mera

      robotstxt.jpg

      1 Reply Last reply Reply Quote 0
      • Highland
        Highland last edited by

        You can also exclude URLs by meta tag. Do you have any meta tags like those on this page?

        http://support.google.com/webmasters/bin/answer.py?hl=en&answer=93710

        meralucian37 1 Reply Last reply Reply Quote 0
        • Mark_Ginsberg
          Mark_Ginsberg last edited by

          Check in webmaster tools what Google is seeing for your robots.txt - this will confirm what you see is what Google is seeing

          Can you paste the full text here?

          meralucian37 1 Reply Last reply Reply Quote 0
          • meralucian37
            meralucian37 @Mark_Ginsberg last edited by

            I attached the image

            robotstxt1.jpg

            Mark_Ginsberg 1 Reply Last reply Reply Quote 0
            • meralucian37
              meralucian37 @Highland last edited by

              I don't have any meta tags to block URL's

              1 Reply Last reply Reply Quote 0
              • Mark_Ginsberg
                Mark_Ginsberg @meralucian37 last edited by

                I just checked your robots file, and I see this:

                User-agent: *

                Disallow: /cars/index

                Disallow: /cars/details

                meralucian37 1 Reply Last reply Reply Quote 0
                • meralucian37
                  meralucian37 @Mark_Ginsberg last edited by

                  I added the 2 lines several hours because i saw that Google had crawl some zend routes.

                  I changed the entire content of the website 2 weeks ago and during this process i notice the problem.

                  Many thanks Mark for your help.

                  1 Reply Last reply Reply Quote 0
                  • 1 / 1
                  • First post
                    Last post
                  • Clean URL vs. Parameter URL and Using Canonical URL...That's a Mouthfull!
                    Roman-Delcarmen
                    Roman-Delcarmen
                    0
                    4
                    170

                  • Can I Block https URLs using Host directive in robots.txt?
                    LoganRay
                    LoganRay
                    0
                    4
                    760

                  • Why is robots.txt blocking URL's in sitemap?
                    PurpleGriffon
                    PurpleGriffon
                    0
                    3
                    346

                  • GWT returning 200 for robots.txt, but it's actually returning a 404?
                    LesleyPaone
                    LesleyPaone
                    0
                    2
                    101

                  • We have designed a new site and are in dilemna whether or not to change the site's URL structure or maybe few odd looking urls. How exactly do we go about the URL thing in toto? thanks in advance, any suggestion(s) would be dearly welcome
                    ShaunPhilips
                    ShaunPhilips
                    1
                    7
                    109

                  • How to solve the meta : A description for this result is not available because this site's robots.txt. ?
                    Vale7
                    Vale7
                    0
                    5
                    1.2k

                  • Google insists robots.txt is blocking... but it isn't.
                    Christy-Correll
                    Christy-Correll
                    0
                    4
                    523

                  • Same URL in "Duplicate Content" and "Blocked by robots.txt"?
                    alsvik
                    alsvik
                    0
                    3
                    502

                  Get started with Moz Pro!

                  Unlock the power of advanced SEO tools and data-driven insights.

                  Start my free trial
                  Products
                  • Moz Pro
                  • Moz Local
                  • Moz API
                  • Moz Data
                  • STAT
                  • Product Updates
                  Moz Solutions
                  • SMB Solutions
                  • Agency Solutions
                  • Enterprise Solutions
                  • Digital Marketers
                  Free SEO Tools
                  • Domain Authority Checker
                  • Link Explorer
                  • Keyword Explorer
                  • Competitive Research
                  • Brand Authority Checker
                  • Local Citation Checker
                  • MozBar Extension
                  • MozCast
                  Resources
                  • Blog
                  • SEO Learning Center
                  • Help Hub
                  • Beginner's Guide to SEO
                  • How-to Guides
                  • Moz Academy
                  • API Docs
                  About Moz
                  • About
                  • Team
                  • Careers
                  • Contact
                  Why Moz
                  • Case Studies
                  • Testimonials
                  Get Involved
                  • Become an Affiliate
                  • MozCon
                  • Webinars
                  • Practical Marketer Series
                  • MozPod
                  Connect with us

                  Contact the Help team

                  Join our newsletter
                  Moz logo
                  © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                  • Accessibility
                  • Terms of Use
                  • Privacy