The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Search Engine Trends
    4. Need only tens of pages to be indexed out of hundreds: Robots.txt is Okay for Google to proceed with?

    Need only tens of pages to be indexed out of hundreds: Robots.txt is Okay for Google to proceed with?

    Search Engine Trends
    2 2 24
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • vtmoz
      vtmoz last edited by

      Hi all,

      We 2 sub domains with hundreds of pages where we need only 50 pages to get indexed which are important. Unfortunately the CMS of these sub domains is very old and not supporting "noindex" tag to be deployed on page level. So we are planning to block the entire sites from robots.txt and allow the 50 pages needed. But we are not sure if this is the right approach as Google been suggesting to depend mostly on "noindex" than robots.txt. Please suggest whether we can proceed with robots.txt file.

      Thanks

      1 Reply Last reply Reply Quote 0
      • GastonRiera
        GastonRiera last edited by

        Hi vtmoz,

        Given the limitations you are telling us, I'd give noindex in robots.txt a try.
        I've run some experiments and found that noindex rule in Robots.txt works. It definitely won´t remove from index that pages, but it will stop showing them for search results.

        I'd suggest you to try using that rule with care. 
        Also, run some experiments on your own. My first test would be only adding one or two pages, the one that causes more trouble being indexed (maybe due to undesired traffic or due to ranking on undesired search terms).

        Hope it helps.
        Best luck!
        GR

        1 Reply Last reply Reply Quote 0
        • 1 / 1
        • First post
          Last post
        • Google Search Console Not Indexing Pages
          0
          1
          73

        • Indexed, though blocked by robots.txt: Need to bother?
          GastonRiera
          GastonRiera
          1
          2
          121

        • Why google stop index my home page
          Andy.Drinkwater
          Andy.Drinkwater
          0
          2
          67

        • How long for google to de-index old pages on my site?
          rubennunez
          rubennunez
          0
          7
          8.1k

        • Trying to figure out why one of my popular pages was de-indexed from Google.
          NoahsDad
          NoahsDad
          0
          5
          627

        • Does google index non-public pages ie. members logged in page
          leschal
          leschal
          0
          3
          704

        • Should I block non-informative pages from Google's index?
          UnderRugSwept
          UnderRugSwept
          1
          10
          795

        • Why would my product pages no longer be indexed in Google?
          KeriMorgret
          KeriMorgret
          0
          9
          2.7k

        Get started with Moz Pro!

        Unlock the power of advanced SEO tools and data-driven insights.

        Start my free trial
        Products
        • Moz Pro
        • Moz Local
        • Moz API
        • Moz Data
        • STAT
        • Product Updates
        Moz Solutions
        • SMB Solutions
        • Agency Solutions
        • Enterprise Solutions
        • Digital Marketers
        Free SEO Tools
        • Domain Authority Checker
        • Link Explorer
        • Keyword Explorer
        • Competitive Research
        • Brand Authority Checker
        • Local Citation Checker
        • MozBar Extension
        • MozCast
        Resources
        • Blog
        • SEO Learning Center
        • Help Hub
        • Beginner's Guide to SEO
        • How-to Guides
        • Moz Academy
        • API Docs
        About Moz
        • About
        • Team
        • Careers
        • Contact
        Why Moz
        • Case Studies
        • Testimonials
        Get Involved
        • Become an Affiliate
        • MozCon
        • Webinars
        • Practical Marketer Series
        • MozPod
        Connect with us

        Contact the Help team

        Join our newsletter
        Moz logo
        © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
        • Accessibility
        • Terms of Use
        • Privacy