The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. Scary bug in search console: All our pages reported as being blocked by robots.txt after https migration

    Scary bug in search console: All our pages reported as being blocked by robots.txt after https migration

    Intermediate & Advanced SEO
    6 4 94
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • lcourse
      lcourse last edited by

      We just migrated to https and created 2 days ago a new property in search console for the https domain.

      Webmaster Tools account for the https domain now shows for every page in our sitemap the warning: "Sitemap contains urls which are blocked by robots.txt."Also in the dashboard of the search console it shows a red triangle with warning that our root domain would be blocked by robots.txt. 1) When I test the URLs in search console robots.txt test tool all looks fine.2) When I fetch as google and render the page it renders and indexes without problem (would not if it was really blocked in robots.txt)3) We temporarily completely emptied the robots.txt, submitted it in search console and uploaded sitemap again and same warnings even though no robots.txt was online4) We run screaming frog crawl on whole website and it indicates that there is no page blocked by robots.txt5) We carefully revised the whole robots.txt and it does not contain any row that blocks relevant content on our site or our root domain. (same robots.txt was online for last decade in http version without problem)6) In big webmaster tools I could upload the sitemap and so far no error reported.7) we resubmitted sitemaps and same issue8) I see our root domain already with https in google SERPThe site is https://www.languagecourse.netSince the site has significant traffic,  if google would really interpret for any reason that our site is blocked by robots we will be in serious trouble.
      This is really scary, so even if it is just a bug in search console and does not affect crawling of the site, it would be great if someone from google could have a look into the reason for this since for a site owner this really can increase cortisol to unhealthy levels.Anybody ever experienced the same problem?Anybody has an idea where we could report/post this issue?

      1 Reply Last reply Reply Quote 0
      • DonnaDuncan
        DonnaDuncan last edited by

        Have you checked with your website hosting provider? They may be blocking bots at a server level. I know when I tried to scan your site I got a connection timeout error.

        lcourse 1 Reply Last reply Reply Quote 0
        • ViviCa1
          ViviCa1 last edited by

          I recently updated a Wordpress website from noindex to wanting it indexed and I still had a warning in Search Console for a day or two and the homepage was initially indexed with the meta description saying "A description for this website ...", even though the actual Fetch and Render and also Robots.txt test was just fine. If you are absolutely sure there isn't anything wrong, I would maybe give it a couple of days. In our case, I resubmitted the homepage to Google to speed up the process and that fixed it.

          1 Reply Last reply Reply Quote 1
          • lcourse
            lcourse @DonnaDuncan last edited by

            Hi Donna, we are using cloudflare which may have blocked your scan/bot.
            We disabled cloudflare temporarily and submitted new robots.txt and new sitemap and still get the same warnings.

            DonnaDuncan 1 Reply Last reply Reply Quote 0
            • DonnaDuncan
              DonnaDuncan @lcourse last edited by

              I'd still speak with the hosting provider. It may be a firewall setting, not the CDN.

              1 Reply Last reply Reply Quote 1
              • Christy-Correll
                Christy-Correll last edited by

                Hi icourse, thanks for your question! You've received some thoughtful responses. Did any of them help you sort your issue out? If so, please mark one or more as a "Good Answer." Thanks!

                Christy

                1 Reply Last reply Reply Quote 0
                • 1 / 1
                • First post
                  Last post
                • How long after https migration that google shows in search console new sitemap being indexed?
                  lcourse
                  lcourse
                  0
                  6
                  621

                • Pages getting into Google Index, blocked by Robots.txt??
                  Devanur-Rafi
                  Devanur-Rafi
                  0
                  10
                  673

                • MOZ crawl report says category pages blocked by meta robots but theyr'e not?
                  SamCUK
                  SamCUK
                  0
                  5
                  315

                • Robots.txt file - How to block thosands of pages when you don't have a folder path
                  Klarke
                  Klarke
                  0
                  4
                  264

                • Search Engine Blocked by robots.txt for Dynamic URLs
                  KeriMorgret
                  KeriMorgret
                  0
                  2
                  689

                • Negative impact on crawling after upload robots.txt file on HTTPS pages
                  ShaMenz
                  ShaMenz
                  0
                  2
                  892

                • 10,000 New Pages of New Content - Should I Block in Robots.txt?
                  EGOL
                  EGOL
                  0
                  2
                  709

                • Block all search results (dynamic) in robots.txt?
                  onwebtoday
                  onwebtoday
                  0
                  9
                  4.8k

                Get started with Moz Pro!

                Unlock the power of advanced SEO tools and data-driven insights.

                Start my free trial
                Products
                • Moz Pro
                • Moz Local
                • Moz API
                • Moz Data
                • STAT
                • Product Updates
                Moz Solutions
                • SMB Solutions
                • Agency Solutions
                • Enterprise Solutions
                • Digital Marketers
                Free SEO Tools
                • Domain Authority Checker
                • Link Explorer
                • Keyword Explorer
                • Competitive Research
                • Brand Authority Checker
                • Local Citation Checker
                • MozBar Extension
                • MozCast
                Resources
                • Blog
                • SEO Learning Center
                • Help Hub
                • Beginner's Guide to SEO
                • How-to Guides
                • Moz Academy
                • API Docs
                About Moz
                • About
                • Team
                • Careers
                • Contact
                Why Moz
                • Case Studies
                • Testimonials
                Get Involved
                • Become an Affiliate
                • MozCon
                • Webinars
                • Practical Marketer Series
                • MozPod
                Connect with us

                Contact the Help team

                Join our newsletter
                Moz logo
                © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                • Accessibility
                • Terms of Use
                • Privacy