The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. On-Page / Site Optimization
    4. I have more pages in my site map being blocked by the robot file than I have being allowed to be crawled. Is Google going to hate me for this?

    I have more pages in my site map being blocked by the robot file than I have being allowed to be crawled. Is Google going to hate me for this?

    On-Page / Site Optimization
    3 3 103
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • absoauto
      absoauto last edited by

      Using some rules to block all pages which start with "copy-of" on my website because people have a bad habit of duplicating new product listings to create our refurbished, surplus etc. listings for those products. To avoid Google seeing these as duplicate pages I've blocked them in the robot file, but of course they are still automatically generated in our sitemap. How bad is this?

      1 Reply Last reply Reply Quote 0
      • Asher
        Asher last edited by

        It is not bad, just not best practices because Google will still index the URL's if they are mentioned on other pages.  Just to quote them:

        "While Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web. As a result, the URL of the page and, potentially, other publicly available information..."

        What I would do instead is either use rel="canonical" or 301 redirects.  I hope that helps.

        1 Reply Last reply Reply Quote 0
        • SVmedia
          SVmedia last edited by

          When you say "people," are you saying your own web team duplicates content to make their job easier? Or am I missing something?...

          If that's the case, you really should create unique URL's with unique page titles, product info, etc.  That's the correct way to avoid getting hit for duplicate content - don't create it.  It seems like what you're doing now is more of a band-aid solution to the problem.

          I'd consider that even though creating unique content in situations like this can seem daunting and/or be more expensive, there's probably huge long-term gains to made if you do it right.

          1 Reply Last reply Reply Quote 0
          • 1 / 1
          • First post
            Last post
          • When making content pages to a specific page; should you index it straight away in GSC or let Google crawl it naturally?
            Jacksons_Fencing
            Jacksons_Fencing
            0
            5
            57

          • Google directs people to the wrong page on my site.
            bradgmo
            bradgmo
            0
            3
            96

          • When You Add a Robots.txt file to a website to block certain URLs, do they disappear from Google's index?
            Saijo.George
            Saijo.George
            0
            3
            205

          • How much content does Google Crawl on your site?
            NatanValencia
            NatanValencia
            0
            3
            297

          • Why isn't our site being shown on the first page of Google for a query using the exact domain, when its pages are indeed indexed by Google
            featherseo
            featherseo
            0
            3
            144

          • Blocking Pages E-Commerce Site
            BlueprintMarketing
            BlueprintMarketing
            0
            6
            397

          • Stumped on why Google is not showing main site pages anymore
            Luia
            Luia
            0
            3
            248

          • Best way to optimize site for Google Maps?
            MiriamEllis
            MiriamEllis
            0
            5
            566

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy