The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. Robots.txt file

    Robots.txt file

    Technical SEO Issues
    3 3 261
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • gimes
      gimes last edited by

      How do i get Google to stop indexing my old pages and start indexing my new pages even months down the line?

      Do i need to install a Robots.txt file on each page?

      1 Reply Last reply Reply Quote 0
      • AlanBleiweiss
        AlanBleiweiss last edited by

        there should be only one robots.txt file for the entire site. Do you mean "Do I need to add a separate line item within the robots.txt file that disallows each page I want deindexed?" ? The answer is that would not be the best solution.

        Other questions need to be asked as well.

        1. Have you created a sitemap.xml file with only the new pages in it? And if so, have you submitted that to Google through Webmaster Tools?

        2. Have you obtained any off-site links pointing to the new pages?

        3. Have you performed a site:mydomain.com search in Google to see whether those new pages have been indexed?

        4. Have you checked Google webmaster tools to see whether there are serious errors on your site their system has come across that might be related?

        5. Does your site's robots.txt file currently block those new pages from being crawled?

        All of these need to be answered to help determine a course of action.

        1 Reply Last reply Reply Quote 0
        • Asher
          Asher last edited by

          What CMS are you using?  If it is wordpress, there is a plugin that allows you to configure when google is asked to crawl your site and you can set it up so that they only crawl your "archives" monthly and set up how often you have new content on the site.  it is called WP Robots Txt

          1 Reply Last reply Reply Quote 0
          • 1 / 1
          • First post
            Last post
          • Multiple robots.txt files on server
            peakdistrictseo
            peakdistrictseo
            0
            8
            2.4k

          • One robots.txt file for multiple sites?
            RyanPurkey
            RyanPurkey
            0
            4
            1.1k

          • Sub Domains and Robot.txt files...
            Vizergy
            Vizergy
            0
            3
            160

          • Do i have my robots.txt file set up properly
            ClaireH-184886
            ClaireH-184886
            1
            4
            319

          • Is my robots.txt file working?
            Resultify
            Resultify
            0
            5
            1.1k

          • Does Bing ignore robots txt files?
            Nightwing
            Nightwing
            0
            3
            2.8k

          • Use of Robots.txt file on a job site
            jennita
            jennita
            0
            5
            850

          • Robots.txt File Redirects to Home Page
            NickPateman81
            NickPateman81
            0
            6
            4.1k

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy