The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. Help needed with robots.txt regarding wordpress!

    Help needed with robots.txt regarding wordpress!

    Technical SEO Issues
    6 2 459
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • ENSO
      ENSO last edited by

      Here is my robots.txt from google webmaster tools. These are the pages that are being blocked and I am not sure which of these to get rid of in order to unblock blog posts from being searched.

      http://ensoplastics.com/theblog/?cat=743

      http://ensoplastics.com/theblog/?p=240

      These category pages and blog posts are blocked so do I delete the /? ...I am new to SEO and web development so I am not sure why the developer of this robots.txt file would block pages and posts in wordpress. It seems to me like that is the reason why someone has a blog so it can be searched and get more exposure for SEO purposes.

      IS there a reason I should block any pages contained in wodrpress?

      Sitemap: http://www.ensobottles.com/blog/sitemap.xml

      User-agent: Googlebot

      Disallow: /*/trackback

      Disallow: /*/feed

      Disallow: /*/comments

      Disallow: /?

      Disallow: /*?

      Disallow: /page/
      User-agent: *

      Disallow: /cgi-bin/

      Disallow: /wp-admin/

      Disallow: /wp-includes/

      Disallow: /wp-content/plugins/

      Disallow: /wp-content/themes/

      Disallow: /trackback

      Disallow: /commentsDisallow: /feed

      1 Reply Last reply Reply Quote 0
      • surfgimp
        surfgimp last edited by

        I'm a bit confused. You reference ensoplastics.com up the top and then show the robots text from ensobottles.com

        Are they using the same robots content? The sites use different url naming so ensobottles uses rewrite whereas the other site uses ?p=

        ENSO 3 Replies Last reply Reply Quote 0
        • ENSO
          ENSO @surfgimp last edited by

          Well that is a problem isn't it? Like I said I am new to a lot of this and I didn't develop either site, this robot.txt file is pointing to the wrong site map. So I am going to change that.

          However I am guessing I may need to change some of the rules to get it to where it is not blocking wordpress content.

          1 Reply Last reply Reply Quote 0
          • ENSO
            ENSO @surfgimp last edited by

            Well that is a problem isn't it? Like I said I am new to a lot of this and I didn't develop either site, this robot.txt file is pointing to the wrong site map. So I am going to change that.

            However I am guessing I may need to change some of the rules to get it to where it is not blocking wordpress content.

            1 Reply Last reply Reply Quote 0
            • ENSO
              ENSO @surfgimp last edited by

              So basically this site was duplicated and apparently the robots.txt file was duplicated. There is no sitemap for the blog created for the enso plastics site, so I am not sure how to proceed at this point. Should I just create a new robots.text file for enoplastics and replace this one? Or do I edit this one, and go create a sitemap for my blog?

              surfgimp 1 Reply Last reply Reply Quote 0
              • surfgimp
                surfgimp @ENSO last edited by

                I've just looked at the home pages of the two sites and they are pretty much the same apart from substituting plastics with bottles. I'm not an expert but I would have thought Google might think this is duplicate content.

                In my opinion I would concentrate on one of the sites say plastics and have the bottle specific stuff as a subsection. I'm not sure how the sites rank etc so that may be easier said than done.

                As for the site map / robot question, if you continue with two sites then I would recommend generating a new one for the copied site.

                1 Reply Last reply Reply Quote 1
                • 1 / 1
                • First post
                  Last post
                • Little confused regarding robots.txt
                  DenorL
                  DenorL
                  0
                  4
                  69

                • Easy Question: regarding no index meta tag vs robot.txt
                  Everett
                  Everett
                  0
                  4
                  213

                • Proxy Server & Wordpress - Need Help
                  RobertFisher
                  RobertFisher
                  0
                  2
                  374

                • RegEx help needed for robots.txt potential conflict
                  Marcus_Miller
                  Marcus_Miller
                  0
                  2
                  304

                • Do I need robots.txt and meta robots?
                  Cyrus-Shepard
                  Cyrus-Shepard
                  0
                  7
                  1.1k

                • Help needed regarding managing client expectations - tricky situation
                  BrianJGomez
                  BrianJGomez
                  0
                  7
                  559

                • Need some help with an old wordpress site we just merged with a new template
                  Jolora
                  Jolora
                  0
                  4
                  425

                • Need Help With Robots.txt on Magento eCommerce Site
                  Francisco_Meza
                  Francisco_Meza
                  0
                  4
                  5.3k

                Get started with Moz Pro!

                Unlock the power of advanced SEO tools and data-driven insights.

                Start my free trial
                Products
                • Moz Pro
                • Moz Local
                • Moz API
                • Moz Data
                • STAT
                • Product Updates
                Moz Solutions
                • SMB Solutions
                • Agency Solutions
                • Enterprise Solutions
                • Digital Marketers
                Free SEO Tools
                • Domain Authority Checker
                • Link Explorer
                • Keyword Explorer
                • Competitive Research
                • Brand Authority Checker
                • Local Citation Checker
                • MozBar Extension
                • MozCast
                Resources
                • Blog
                • SEO Learning Center
                • Help Hub
                • Beginner's Guide to SEO
                • How-to Guides
                • Moz Academy
                • API Docs
                About Moz
                • About
                • Team
                • Careers
                • Contact
                Why Moz
                • Case Studies
                • Testimonials
                Get Involved
                • Become an Affiliate
                • MozCon
                • Webinars
                • Practical Marketer Series
                • MozPod
                Connect with us

                Contact the Help team

                Join our newsletter
                Moz logo
                © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                • Accessibility
                • Terms of Use
                • Privacy