The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. On-Page / Site Optimization
    4. I have more pages in my site map being blocked by the robot file than I have being allowed to be crawled. Is Google going to hate me for this?

    I have more pages in my site map being blocked by the robot file than I have being allowed to be crawled. Is Google going to hate me for this?

    On-Page / Site Optimization
    3 3 103
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • absoauto
      absoauto last edited by

      Using some rules to block all pages which start with "copy-of" on my website because people have a bad habit of duplicating new product listings to create our refurbished, surplus etc. listings for those products. To avoid Google seeing these as duplicate pages I've blocked them in the robot file, but of course they are still automatically generated in our sitemap. How bad is this?

      1 Reply Last reply Reply Quote 0
      • Asher
        Asher last edited by

        It is not bad, just not best practices because Google will still index the URL's if they are mentioned on other pages.  Just to quote them:

        "While Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web. As a result, the URL of the page and, potentially, other publicly available information..."

        What I would do instead is either use rel="canonical" or 301 redirects.  I hope that helps.

        1 Reply Last reply Reply Quote 0
        • SVmedia
          SVmedia last edited by

          When you say "people," are you saying your own web team duplicates content to make their job easier? Or am I missing something?...

          If that's the case, you really should create unique URL's with unique page titles, product info, etc.  That's the correct way to avoid getting hit for duplicate content - don't create it.  It seems like what you're doing now is more of a band-aid solution to the problem.

          I'd consider that even though creating unique content in situations like this can seem daunting and/or be more expensive, there's probably huge long-term gains to made if you do it right.

          1 Reply Last reply Reply Quote 0
          • 1 / 1
          • First post
            Last post
          • When making content pages to a specific page; should you index it straight away in GSC or let Google crawl it naturally?
            Jacksons_Fencing
            Jacksons_Fencing
            0
            5
            57

          • If Product Pages Perform Well In Google then Is It Possible That Category Page Can perform Well In Google?
            badgergravling
            badgergravling
            0
            2
            52

          • Meta Titles On Site Different From Google Index Page
            HiveDigitalInc
            HiveDigitalInc
            0
            11
            196

          • Robot.txt file issue on wordpress site.
            AlanMosley
            AlanMosley
            0
            8
            229

          • Blocking Subdomain from Google Crawl and Index
            odmsoft
            odmsoft
            0
            8
            33.7k

          • Description tag not showing in the SERPs because page is blocked by Robots, but the page isn't blocked. Any help?
            DavidFaltz
            DavidFaltz
            0
            9
            183

          • Blocking Pages E-Commerce Site
            BlueprintMarketing
            BlueprintMarketing
            0
            6
            397

          • Best Way to Add Google Map to Site
            evolvingSEO
            evolvingSEO
            0
            3
            662

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy