The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. Robots.txt file

    Robots.txt file

    Technical SEO Issues
    3 3 261
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • gimes
      gimes last edited by

      How do i get Google to stop indexing my old pages and start indexing my new pages even months down the line?

      Do i need to install a Robots.txt file on each page?

      1 Reply Last reply Reply Quote 0
      • AlanBleiweiss
        AlanBleiweiss last edited by

        there should be only one robots.txt file for the entire site. Do you mean "Do I need to add a separate line item within the robots.txt file that disallows each page I want deindexed?" ? The answer is that would not be the best solution.

        Other questions need to be asked as well.

        1. Have you created a sitemap.xml file with only the new pages in it? And if so, have you submitted that to Google through Webmaster Tools?

        2. Have you obtained any off-site links pointing to the new pages?

        3. Have you performed a site:mydomain.com search in Google to see whether those new pages have been indexed?

        4. Have you checked Google webmaster tools to see whether there are serious errors on your site their system has come across that might be related?

        5. Does your site's robots.txt file currently block those new pages from being crawled?

        All of these need to be answered to help determine a course of action.

        1 Reply Last reply Reply Quote 0
        • Asher
          Asher last edited by

          What CMS are you using?  If it is wordpress, there is a plugin that allows you to configure when google is asked to crawl your site and you can set it up so that they only crawl your "archives" monthly and set up how often you have new content on the site.  it is called WP Robots Txt

          1 Reply Last reply Reply Quote 0
          • 1 / 1
          • First post
            Last post
          • Do I need a separate robots.txt file for my shop subdomain?
            sjbridle
            sjbridle
            0
            6
            593

          • Is there any value in having a blank robots.txt file?
            LiamVictor
            LiamVictor
            0
            6
            3.1k

          • Meta Robots Noindex and Robots.txt File
            Devanur-Rafi
            Devanur-Rafi
            0
            2
            125

          • Oh no googlebot can not access my robots.txt file
            BistosAmerica
            BistosAmerica
            0
            9
            6.3k

          • Is my robots.txt file working?
            Resultify
            Resultify
            0
            5
            1.1k

          • Does Bing ignore robots txt files?
            Nightwing
            Nightwing
            0
            3
            2.8k

          • Use of Robots.txt file on a job site
            jennita
            jennita
            0
            5
            850

          • Robots.txt file question? NEver seen this command before
            omnea
            omnea
            0
            7
            772

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy