The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. Robots.txt issue - site resubmission needed?

    Robots.txt issue - site resubmission needed?

    Technical SEO Issues
    4 3 227
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • GBC
      GBC last edited by

      We recently had an issue when a load of new files were transferred from our dev server to the live site, which unfortunately included the dev site's robots.txt file which had a disallow:/ instruction. Bad!

      Luckily I spotted it quickly and the file has been replaced. The extent of the damage seems to be that some descriptions aren't displaying and we're getting a message about robots.txt in the SERPs for a few keywords. I've done a site: search and generally it seems to be OK for 99% of our pages.

      Our positions don't seem to be affected right now but obviously it's not great for the CTRs on those keywords affected.

      My question is whether there is anything I can do to bring the updated robots.txt file to Google's attention? Or should we just wait and sit it out?

      Thanks in advance for your answers!

      1 Reply Last reply Reply Quote 0
      • Webrevolve
        Webrevolve last edited by

        Hi,

        We had a vaguely similar thing happen where we took over the running over a site and the old dev added a robots.txt disallowing the site from being indexed, which wasn't pick up for a while as dev work was on-going. Basically the sites rankings panned out.

        We fixed the issue an re-submitted the site through webmaster tools. Site was reindexed within a week or so and rankings came back over the next 6 weeks.

        AndreVanKets 1 Reply Last reply Reply Quote 1
        • AndreVanKets
          AndreVanKets @Webrevolve last edited by

          I agree with Michael.

          I have also seen a wordpress site that had blocked the robots from the entire site for 1 week.

          After allowing the robots back in, we saw the rankings improve with in a few days.

          Don't stress, just resubmit the sitemap or create a new one with the effected URL's

          Greg

          GBC 1 Reply Last reply Reply Quote 1
          • GBC
            GBC @AndreVanKets last edited by

            Hi Greg - I've done this and resubmitted the sitemap, but I'm now getting severe health warnings saying that robots.txt is blocking important pages.

            I've run our robots.txt file through the tester in Webmaster tools and it is saying allowed, however I'm obviously concerned about the warnings.

            Have you experience of anything like this?

            Thanks
            Rory

            1 Reply Last reply Reply Quote 0
            • 1 / 1
            • First post
              Last post
            • Should you use robots.txt for pages within your site which do not have high quality content or are not contributing a great deal so when Google crawls your site the best performing content has a higher chance of being indexed?
              Jacksons_Fencing
              Jacksons_Fencing
              0
              5
              44

            • Do I need a separate robots.txt file for my shop subdomain?
              sjbridle
              sjbridle
              0
              6
              593

            • One robots.txt file for multiple sites?
              RyanPurkey
              RyanPurkey
              0
              4
              1.1k

            • Do I need to block my cart page in robots.txt?
              Hutch42
              Hutch42
              0
              3
              2.4k

            • Removing robots.txt on WordPress site problem
              nextlevelweb
              nextlevelweb
              0
              21
              38.1k

            • Mobile site: robots.txt best practices
              AlanMosley
              AlanMosley
              0
              2
              542

            • Robots.txt and robots meta
              TheEspresseo
              TheEspresseo
              0
              5
              1.1k

            Get started with Moz Pro!

            Unlock the power of advanced SEO tools and data-driven insights.

            Start my free trial
            Products
            • Moz Pro
            • Moz Local
            • Moz API
            • Moz Data
            • STAT
            • Product Updates
            Moz Solutions
            • SMB Solutions
            • Agency Solutions
            • Enterprise Solutions
            • Digital Marketers
            Free SEO Tools
            • Domain Authority Checker
            • Link Explorer
            • Keyword Explorer
            • Competitive Research
            • Brand Authority Checker
            • Local Citation Checker
            • MozBar Extension
            • MozCast
            Resources
            • Blog
            • SEO Learning Center
            • Help Hub
            • Beginner's Guide to SEO
            • How-to Guides
            • Moz Academy
            • API Docs
            About Moz
            • About
            • Team
            • Careers
            • Contact
            Why Moz
            • Case Studies
            • Testimonials
            Get Involved
            • Become an Affiliate
            • MozCon
            • Webinars
            • Practical Marketer Series
            • MozPod
            Connect with us

            Contact the Help team

            Join our newsletter
            Moz logo
            © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
            • Accessibility
            • Terms of Use
            • Privacy