The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. Should I robots block this directory?

    Should I robots block this directory?

    Intermediate & Advanced SEO
    3 3 666
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • CareerBliss
      CareerBliss last edited by

      There's about 43k pages indexed in  this directory, and while helpful to end users, I don't see it being a great source of unique content for search engines.

      Would you robots block or meta noindex nofollow these pages in the /blissindex/ directory?

      ie.

      http://www.careerbliss.com/blissindex/petsmart-index-980481/

      http://www.careerbliss.com/blissindex/att-index-1043730/

      http://www.careerbliss.com/blissindex/facebook-index-996632/

      1 Reply Last reply Reply Quote 0
      • RyanKent
        RyanKent last edited by

        Should I robots block this directory?

        I wouldn't.

        Robots.txt in general should only be used when there is no other alternate means available to block content. An example is when your site is created by a CMS or e-commerce platform which does not offer the flexibility to noindex individual pages.

        By blocking your site's content, you are preventing search engines not only from indexing the pages, but from following any links on those pages. You are restricting the way a crawler can travel on your site, which is generally a bad idea.

        Additionally, I would suggest those pages offer value. "Petco salary comparison", "Target wages" and other search queries could generate results for those pages. Those pages contain helpful information which is otherwise not easily found on the internet. If that was my site, I would work to improve the optimization of those pages, not block them.

        1 Reply Last reply Reply Quote 2
        • irvingw
          irvingw last edited by

          Totally agree with Ryan Kent. You should write a paragraph of content that is unique to the company featured. The chart is not unique enough and you will get flagged as having a high ratio of duplicate content. You should also look at all the other SEO elements on this page, understand what keyphrases you are targeting and modify the title, meta and H1 tags.

          1 Reply Last reply Reply Quote 1
          • 1 / 1
          • First post
            Last post
          • Pages blocked by robots
            mihoreis
            mihoreis
            0
            3
            69

          • Block subdomain directory in robots.txt
            DirkC
            DirkC
            0
            5
            1.1k

          • Robots.txt Blocked Most Site URLs Because of Canonical
            0
            1
            117

          • What should I block with a robots.txt file?
            Travis-W
            Travis-W
            1
            3
            298

          • Files blocked in robot.txt and seo
            john4math
            john4math
            0
            4
            344

          • What content should I block in wodpress with robots.txt?
            ENSO
            ENSO
            0
            4
            518

          • Block all but one URL in a directory using robots.txt?
            Cyrus-Shepard
            Cyrus-Shepard
            0
            3
            3.5k

          • Block an entire subdomain with robots.txt?
            kylesuss
            kylesuss
            1
            16
            102.1k

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy