The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Moz Pro
    4. Crawl Diagnostics returning duplicate content based on session id

    Crawl Diagnostics returning duplicate content based on session id

    Moz Pro
    3 2 386
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • csingsaas
      csingsaas last edited by

      I'm just starting to dig into crawl diagnostics and it is returning quite a few errors. Primarily, the crawl is indicating duplicate content (page titles, meta tags, etc), because of a session id in the URL.

      I have set-up a URL parameter in Google Webmaster Tools to help Google recognize the existence of this session id. Is there any way to tell the SEOMoz spider the same thing? I'd like to get rid of these errors since I've already handled them for the most part.

      1 Reply Last reply Reply Quote 0
      • Martijn_Scheijbeler
        Martijn_Scheijbeler last edited by

        Hi Cody,

        The best way is to block Rogerbot within your Robots.txt from crawling specific pages of your site. In your case protecting Rogerbot from seeing the pages with a session ID.
        More information could be found here on Rogerbot.

        Be cautious and test it out, but the lines you would have to add to your Robots.txt are probably:

        User-agent: rogerbot 
        Disallow: /*sessionid

        Hope this helps!

        csingsaas 1 Reply Last reply Reply Quote 2
        • csingsaas
          csingsaas @Martijn_Scheijbeler last edited by

          You the man! Thanks!

          1 Reply Last reply Reply Quote 0
          • 1 / 1
          • First post
            Last post
          • Pagination Issues on E-commerce Site: Duplicate Page Title and Content on Moz Crawl
            Dr-Pete
            Dr-Pete
            0
            3
            1.4k

          • Crawl diagnostics incorrectly reporting duplicate page titles
            ChiarynMiranda
            ChiarynMiranda
            0
            4
            176

          • Crawl Diagnostics: Next crawl date is in the past
            Abe_Schmidt
            Abe_Schmidt
            0
            4
            291

          • Duplicate content error?
            BlueLinkERP
            BlueLinkERP
            0
            3
            1.5k

          • "Issue: Duplicate Page Content " in Crawl Diagnostics - but sample pages are not related to page indicated with duplicate content
            cbielich
            cbielich
            0
            2
            345

          • How to remove Duplicate content due to url parameters from SEOMoz Crawl Diagnostics
            dfeg
            dfeg
            0
            7
            2.3k

          • About Duplicate Content found by SEOMOZ... that is not duplicate
            nuxeo
            nuxeo
            0
            4
            614

          • SEOmoz crawl diagnostics report - what are the duplicate pages urls?
            SteveOllington
            SteveOllington
            0
            4
            959

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy