The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. What content should I block in wodpress with robots.txt?

    What content should I block in wodpress with robots.txt?

    Intermediate & Advanced SEO
    4 3 518
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • ENSO
      ENSO last edited by

      I need to know if anyone has tips on creating a good robots.txt. I have read a lot of info, but I am just not clear on what I should allow and not allow on wordpress.  For example there are pages and posts, then attachments, wp-admin, wp-content and so on. Does anyone have a good robots.txt guideline?

      1 Reply Last reply Reply Quote 0
      • KeriMorgret
        KeriMorgret last edited by

        Here's a sample robots.txt from the wordpress.org site itself:

        http://codex.wordpress.org/Search_Engine_Optimization_for_Wordpress#Robots.txt_Optimization

        1 Reply Last reply Reply Quote 1
        • JohannCR
          JohannCR last edited by

          The less you put in robots.txt the better...

          If there exists anywhere (on your site or on another) a link to some page that you put in robots.txt, it won't stop google from indexing it, and it will look bad (no title, no description since the bot can't access them). Worse, if you wan to noindex that page in the future it won't work !

          So block content if you know there's no link to it and there won't ever be, otherwise the right move is the noindex meta tag.

          ENSO 1 Reply Last reply Reply Quote 1
          • ENSO
            ENSO @JohannCR last edited by

            Here is what I have changed it to...found various articles including the one listed above and decided to go with this, not sure if it is good or bad.

            www.ensoplastics.com/robots.txt

            1 Reply Last reply Reply Quote 0
            • 1 / 1
            • First post
              Last post
            • Robots.txt blocked internal resources Wordpress
              Mat_C
              Mat_C
              1
              5
              985

            • Search Results Pages Blocked in Robots.txt?
              BeckyKey
              BeckyKey
              0
              3
              117

            • Robots.txt Blocking - Best Practices
              ReunionMarketing
              ReunionMarketing
              0
              7
              456

            • Block subdomain directory in robots.txt
              DirkC
              DirkC
              0
              5
              1.1k

            • Blocking out specific URLs with robots.txt
              Modi
              Modi
              0
              3
              133

            • Why are these results being showed as blocked by robots.txt?
              eyepaq
              eyepaq
              0
              9
              203

            • Files blocked in robot.txt and seo
              john4math
              john4math
              0
              4
              344

            • 10,000 New Pages of New Content - Should I Block in Robots.txt?
              EGOL
              EGOL
              0
              2
              709

            Get started with Moz Pro!

            Unlock the power of advanced SEO tools and data-driven insights.

            Start my free trial
            Products
            • Moz Pro
            • Moz Local
            • Moz API
            • Moz Data
            • STAT
            • Product Updates
            Moz Solutions
            • SMB Solutions
            • Agency Solutions
            • Enterprise Solutions
            • Digital Marketers
            Free SEO Tools
            • Domain Authority Checker
            • Link Explorer
            • Keyword Explorer
            • Competitive Research
            • Brand Authority Checker
            • Local Citation Checker
            • MozBar Extension
            • MozCast
            Resources
            • Blog
            • SEO Learning Center
            • Help Hub
            • Beginner's Guide to SEO
            • How-to Guides
            • Moz Academy
            • API Docs
            About Moz
            • About
            • Team
            • Careers
            • Contact
            Why Moz
            • Case Studies
            • Testimonials
            Get Involved
            • Become an Affiliate
            • MozCon
            • Webinars
            • Practical Marketer Series
            • MozPod
            Connect with us

            Contact the Help team

            Join our newsletter
            Moz logo
            © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
            • Accessibility
            • Terms of Use
            • Privacy