The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. Google (GWT) says my homepage and posts are blocked by Robots.txt

    Google (GWT) says my homepage and posts are blocked by Robots.txt

    Technical SEO Issues
    8 3 930
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • FrederikTrovatten22
      FrederikTrovatten22 last edited by

      I guys.. I have a very annoying issue..

      My Wordpress-blog over at www.Trovatten.com has some indexation-problems..

      Google Webmaster Tools data:
      GWT says the following: "Sitemap contains urls which are blocked by robots.txt." and shows me my homepage and my blogposts..

      This is my Robots.txt: http://www.trovatten.com/robots.txt
      "User-agent: *
      Disallow: /wp-admin/
      Disallow: /wp-includes/

      Do you have any idea why it says that the URL's are being blocked by robots.txt when that looks how it should?
      I've read a couple of places that it can be because of a Wordpress Plugin that is creating a virtuel robots.txt, but I can't validate it..

      1. I have set WP-Privacy to crawl my site
      2. I have deactivated all WP-plugins and I still get same GWT-Warnings.

      Looking forward to hear if you have an idea that might work!

      1 Reply Last reply Reply Quote 0
      • stubby
        stubby last edited by

        Is Yoast generating another sitemap for you?

        You have a sitemap from a different plugin, but Yoast can also generate sitemaps, so perhaps you have 2 - and one of the sitemaps lists the items that you are disallowing.

        FrederikTrovatten22 2 Replies Last reply Reply Quote 0
        • FrederikTrovatten22
          FrederikTrovatten22 @stubby last edited by

          Thanks for your comment Stubby and you are probably right.

          But the problem is the Disallowing and not the sitemaps.. And based on my Robots.txt should everything be crawable.

          What I'm worried about is that the virtuel Robots.txt that WP-generates is trouble.

          1 Reply Last reply Reply Quote 0
          • FrederikTrovatten22
            FrederikTrovatten22 @stubby last edited by

            I have tweak around with an XML SItemap-generater and I think it works. I'll give an update in a couple of hours!

            Thansk!

            1 Reply Last reply Reply Quote 0
            • Anthony_NorthSEO
              Anthony_NorthSEO last edited by

              Hey Frederick,

              Here's your current meta data for robots on your home page (in the section):

              name="robots" content="noodp, noydir" />

              Should be something like this:

              name="robots" content="INDEX,FOLLOW" />

              I don't think it's the robots.txt that's the issue, but rather the meta-robots in the head of the site.

              Hope this helps!

              Thanks,

              Anthony

              [moderator's note: this answer was actually not the correct answer for this question, please see responses below]

              stubby 1 Reply Last reply Reply Quote -1
              • stubby
                stubby @Anthony_NorthSEO last edited by

                I am going to disagree with the above.

                The command <meta < span="">name="robots" content="noodp, noydir" /> has nothing to do with denying any access to the robots.</meta <>

                It is used to prevent the engines from displaying meta descriptions from DMOZ and the Yahoo directory.  Without this line, the search engines might choose to use those descriptions, rather than the descriptions you have as meta descriptions.

                FrederikTrovatten22 stubby 2 Replies Last reply Reply Quote 2
                • FrederikTrovatten22
                  FrederikTrovatten22 @stubby last edited by

                  You are absolutely right.

                  The problem was that a plugin I installed messed with my robots.txt

                  1 Reply Last reply Reply Quote 0
                  • stubby
                    stubby @stubby last edited by

                    Do you know which plugin (or combination) was the trouble?

                    I use a lot of wordpress, and this is very interesting.

                    1 Reply Last reply Reply Quote 0
                    • 1 / 1
                    • First post
                      Last post
                    • Google Search console says 'sitemap is blocked by robots?
                      BlueprintMarketing
                      BlueprintMarketing
                      1
                      10
                      1.2k

                    • Robots.txt blocking Addon Domains
                      eglove
                      eglove
                      0
                      10
                      556

                    • Google Indexing Development Site Despite Robots.txt Block
                      DeanAndrews
                      DeanAndrews
                      0
                      6
                      990

                    • Google indexing despite robots.txt block
                      john4math
                      john4math
                      0
                      13
                      654

                    • Block Domain in robots.txt
                      donford
                      donford
                      0
                      6
                      2.2k

                    • I accidentally blocked Google with Robots.txt. What next?
                      SebastianCowie
                      SebastianCowie
                      0
                      7
                      2.1k

                    • How to block google robots from a subdomain
                      Alexey_mindvalley
                      Alexey_mindvalley
                      0
                      5
                      2.6k

                    • Blocking robots.txt
                      de4e
                      de4e
                      0
                      4
                      432

                    Get started with Moz Pro!

                    Unlock the power of advanced SEO tools and data-driven insights.

                    Start my free trial
                    Products
                    • Moz Pro
                    • Moz Local
                    • Moz API
                    • Moz Data
                    • STAT
                    • Product Updates
                    Moz Solutions
                    • SMB Solutions
                    • Agency Solutions
                    • Enterprise Solutions
                    • Digital Marketers
                    Free SEO Tools
                    • Domain Authority Checker
                    • Link Explorer
                    • Keyword Explorer
                    • Competitive Research
                    • Brand Authority Checker
                    • Local Citation Checker
                    • MozBar Extension
                    • MozCast
                    Resources
                    • Blog
                    • SEO Learning Center
                    • Help Hub
                    • Beginner's Guide to SEO
                    • How-to Guides
                    • Moz Academy
                    • API Docs
                    About Moz
                    • About
                    • Team
                    • Careers
                    • Contact
                    Why Moz
                    • Case Studies
                    • Testimonials
                    Get Involved
                    • Become an Affiliate
                    • MozCon
                    • Webinars
                    • Practical Marketer Series
                    • MozPod
                    Connect with us

                    Contact the Help team

                    Join our newsletter
                    Moz logo
                    © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                    • Accessibility
                    • Terms of Use
                    • Privacy