The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Moz Tools
    4. Why does my crawl diagnostics show duplicate content

    Why does my crawl diagnostics show duplicate content

    Moz Tools
    5 4 422
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • MSSBConsulting
      MSSBConsulting last edited by

      My crawl diagnostics show duplicate content at mysite.com and mysite.com/index.html which are essentially the same file.

      1 Reply Last reply Reply Quote 0
      • CollinJarman
        CollinJarman last edited by

        "Essentially" the same file isn't the same as "the same file."  Your best bet is probably to mark one of them (probably mysite.com) with rel=canonical.

        MSSBConsulting 1 Reply Last reply Reply Quote 0
        • mozzello
          mozzello last edited by

          If the crawl report found those two URLs, then your website has at least one link to each of those URLs (otherwise Rogerbot wouldn't have found them).

          You should follow Collin's advice to define the canonical page.

          It also won't  hurt to figure out where those links are being used in your content, and then make sure you only use one to point to your page.

          Cheers

          Michel

          Dr-Pete 1 Reply Last reply Reply Quote 1
          • MSSBConsulting
            MSSBConsulting @CollinJarman last edited by

            mysite.com is a domain not a file with mysite.com/index.html being the home page. Not sure how I would do what you suggest.

            1 Reply Last reply Reply Quote 0
            • Dr-Pete
              Dr-Pete @mozzello last edited by

              Michel is right - Google doesn't care that they're one template - if both URLs are being crawled, then they'll see that as two "pages". Every unique, crawlable URL can become an indexed page. That's why duplicate content problems are so common.

              The good news is that you can put a canonical tag on just the one template/file and it will cover all of the paths/URLs that land on that file. The tag goes in your section and looks like:

              I'd check the internal links, though, and see if you're linking to both versions. It's best to use one, consistent URL in your internal links for any given page.

              1 Reply Last reply Reply Quote 0
              • 1 / 1
              • First post
                Last post
              • Duplicate content flagged by Moz that's not actually duplicate content at all
                Caro-O
                Caro-O
                1
                4
                89

              • Blog Page URLs Showing Duplicate Content
                Mike.Goracke
                Mike.Goracke
                0
                4
                195

              • Why does Crawl Diagnostics report this as duplicate content?
                ChiarynMiranda
                ChiarynMiranda
                0
                5
                378

              • Crawl Diagnostics Warnings - Duplicate Content
                skehoe
                skehoe
                0
                3
                349

              • How to resolve Duplicate Content crawl errors for Magento Login Page
                afmaury
                afmaury
                1
                5
                2.5k

              • Duplicate page content showing up with proper use of canonical tag
                Leighm
                Leighm
                0
                7
                665

              • Crawl reports urls with duplicate content but its not the case
                Cyrus-Shepard
                Cyrus-Shepard
                0
                6
                543

              • My crawl diagnostic is showing 2 duplicate content and titles.
                JasonHegarty
                JasonHegarty
                0
                9
                846

              Get started with Moz Pro!

              Unlock the power of advanced SEO tools and data-driven insights.

              Start my free trial
              Products
              • Moz Pro
              • Moz Local
              • Moz API
              • Moz Data
              • STAT
              • Product Updates
              Moz Solutions
              • SMB Solutions
              • Agency Solutions
              • Enterprise Solutions
              • Digital Marketers
              Free SEO Tools
              • Domain Authority Checker
              • Link Explorer
              • Keyword Explorer
              • Competitive Research
              • Brand Authority Checker
              • Local Citation Checker
              • MozBar Extension
              • MozCast
              Resources
              • Blog
              • SEO Learning Center
              • Help Hub
              • Beginner's Guide to SEO
              • How-to Guides
              • Moz Academy
              • API Docs
              About Moz
              • About
              • Team
              • Careers
              • Contact
              Why Moz
              • Case Studies
              • Testimonials
              Get Involved
              • Become an Affiliate
              • MozCon
              • Webinars
              • Practical Marketer Series
              • MozPod
              Connect with us

              Contact the Help team

              Join our newsletter
              Moz logo
              © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
              • Accessibility
              • Terms of Use
              • Privacy