The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Search Engine Trends
    4. Need only tens of pages to be indexed out of hundreds: Robots.txt is Okay for Google to proceed with?

    Need only tens of pages to be indexed out of hundreds: Robots.txt is Okay for Google to proceed with?

    Search Engine Trends
    2 2 24
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • vtmoz
      vtmoz last edited by

      Hi all,

      We 2 sub domains with hundreds of pages where we need only 50 pages to get indexed which are important. Unfortunately the CMS of these sub domains is very old and not supporting "noindex" tag to be deployed on page level. So we are planning to block the entire sites from robots.txt and allow the 50 pages needed. But we are not sure if this is the right approach as Google been suggesting to depend mostly on "noindex" than robots.txt. Please suggest whether we can proceed with robots.txt file.

      Thanks

      1 Reply Last reply Reply Quote 0
      • GastonRiera
        GastonRiera last edited by

        Hi vtmoz,

        Given the limitations you are telling us, I'd give noindex in robots.txt a try.
        I've run some experiments and found that noindex rule in Robots.txt works. It definitely won´t remove from index that pages, but it will stop showing them for search results.

        I'd suggest you to try using that rule with care. 
        Also, run some experiments on your own. My first test would be only adding one or two pages, the one that causes more trouble being indexed (maybe due to undesired traffic or due to ranking on undesired search terms).

        Hope it helps.
        Best luck!
        GR

        1 Reply Last reply Reply Quote 0
        • 1 / 1
        • First post
          Last post
        • Have you ever seen or experienced a page indexed which is actually from a website which is blocked by robots.txt?
          GastonRiera
          GastonRiera
          0
          2
          54

        • Why google stop index my home page
          Andy.Drinkwater
          Andy.Drinkwater
          0
          2
          67

        • What is the appropriate Robot.txt to unblock if Google cannot get all the resources from my homepage?
          OlegKorneitchouk
          OlegKorneitchouk
          0
          2
          5.7k

        • Trying to figure out why one of my popular pages was de-indexed from Google.
          NoahsDad
          NoahsDad
          0
          5
          627

        • Does google index non-public pages ie. members logged in page
          leschal
          leschal
          0
          3
          704

        • Stop google indexing CDN pages
          loopyal
          loopyal
          0
          3
          2.7k

        • Does Google index Wordpress pages with frames
          BradBorst
          BradBorst
          0
          3
          922

        • Why would my product pages no longer be indexed in Google?
          KeriMorgret
          KeriMorgret
          0
          9
          2.7k

        Get started with Moz Pro!

        Unlock the power of advanced SEO tools and data-driven insights.

        Start my free trial
        Products
        • Moz Pro
        • Moz Local
        • Moz API
        • Moz Data
        • STAT
        • Product Updates
        Moz Solutions
        • SMB Solutions
        • Agency Solutions
        • Enterprise Solutions
        • Digital Marketers
        Free SEO Tools
        • Domain Authority Checker
        • Link Explorer
        • Keyword Explorer
        • Competitive Research
        • Brand Authority Checker
        • Local Citation Checker
        • MozBar Extension
        • MozCast
        Resources
        • Blog
        • SEO Learning Center
        • Help Hub
        • Beginner's Guide to SEO
        • How-to Guides
        • Moz Academy
        • API Docs
        About Moz
        • About
        • Team
        • Careers
        • Contact
        Why Moz
        • Case Studies
        • Testimonials
        Get Involved
        • Become an Affiliate
        • MozCon
        • Webinars
        • Practical Marketer Series
        • MozPod
        Connect with us

        Contact the Help team

        Join our newsletter
        Moz logo
        © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
        • Accessibility
        • Terms of Use
        • Privacy