The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Search Engine Trends
    4. Indexed, though blocked by robots.txt: Need to bother?

    Indexed, though blocked by robots.txt: Need to bother?

    Search Engine Trends
    2 2 121
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • vtmoz
      vtmoz last edited by

      Hi,

      We have intentionally blocked some of the website files which were indexed for years. Now we receive a message "Indexed, though blocked by robots.txt" in GSC. We can ignore as per my knowledge? Are any actions required about this? We thought of blocking them with meta tags but these are PDF files.

      Thanks

      1 Reply Last reply Reply Quote 1
      • GastonRiera
        GastonRiera last edited by

        Hi there!

        What Google is telling you is that you are indexing URLs that you probably are  not wanting to be indexed, or the other way around, that important pages are being blocked but indexed for other reasons.

        If I might ask, why did you blocked through robots.txt those files?
        There most 2 answers are:
        1- Wanted to remove those from search results. If this is your case, you've solved only a part of the problem. What you should have done is (previously allowing robots to crawl those urls) apply noindex rules (keep in mind that can be set up in the HTTP header, as long as not html files cant have meta robots tag), then after a sufficient time block them in robots.txt. 
        _2- Optimize how GoogleBot (crawiling) time. _Being this case, then you've done it correctly and there is nothing to worry.

        Hope this help.
        Best luck.
        GR

        1 Reply Last reply Reply Quote 1
        • 1 / 1
        • First post
          Last post
        • Need only tens of pages to be indexed out of hundreds: Robots.txt is Okay for Google to proceed with?
          GastonRiera
          GastonRiera
          0
          2
          24

        • What happens when a de-indexed subdomain is redirected to another de-indexed subdomain? What happens to the link juice?
          vtmoz
          vtmoz
          1
          3
          70

        • Meta robots at every page rather than using robots.txt for blocking crawlers? How they'll get indexed if we block crawlers?
          ThompsonPaul
          ThompsonPaul
          0
          3
          253

        • Google indexing site content that I did not wish to be indexed
          David-E-Carey
          David-E-Carey
          0
          6
          144

        • What is the appropriate Robot.txt to unblock if Google cannot get all the resources from my homepage?
          OlegKorneitchouk
          OlegKorneitchouk
          0
          2
          5.7k

        • Googlebot soon to be executing javascript - Should I change my robots.txt?
          danatanseo
          danatanseo
          0
          3
          504

        • Will we no longer need Location + Keyword? Do we even need it at all?
          SanketPatel
          SanketPatel
          0
          6
          196

        • Is it hurting my seo ranking if robots.txt is forbidden?
          Sebes
          Sebes
          0
          6
          1.3k

        Get started with Moz Pro!

        Unlock the power of advanced SEO tools and data-driven insights.

        Start my free trial
        Products
        • Moz Pro
        • Moz Local
        • Moz API
        • Moz Data
        • STAT
        • Product Updates
        Moz Solutions
        • SMB Solutions
        • Agency Solutions
        • Enterprise Solutions
        • Digital Marketers
        Free SEO Tools
        • Domain Authority Checker
        • Link Explorer
        • Keyword Explorer
        • Competitive Research
        • Brand Authority Checker
        • Local Citation Checker
        • MozBar Extension
        • MozCast
        Resources
        • Blog
        • SEO Learning Center
        • Help Hub
        • Beginner's Guide to SEO
        • How-to Guides
        • Moz Academy
        • API Docs
        About Moz
        • About
        • Team
        • Careers
        • Contact
        Why Moz
        • Case Studies
        • Testimonials
        Get Involved
        • Become an Affiliate
        • MozCon
        • Webinars
        • Practical Marketer Series
        • MozPod
        Connect with us

        Contact the Help team

        Join our newsletter
        Moz logo
        © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
        • Accessibility
        • Terms of Use
        • Privacy