The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Moz Tools
    4. Can I exclude pages from my Crawl Diagnostics?

    Can I exclude pages from my Crawl Diagnostics?

    Moz Tools
    3 2 556
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • JonathanGoodman
      JonathanGoodman last edited by

      Right now my crawl diagnostic information is being skewed because it's including the onsite search from my website. Is there a way to remove certain pages like search from the errors and warnings of the crawl diagnostic? My search pages are coming up as:

      • Long URL
      • Title Element Too Long
      • Missing Meta Description
      • Blocked by meta-robots (Which is how I want it)
      • Rel Canonical

      Here is what the crawl diagnostic thinks my page URL looks like:

      website.com/search/gutter%25252525252525252525252525252525252525252525252525252525

      252525252525252525252525252525252525252525252525252525252525252

      525252525252525252525252525252525252525252525252525252525252525

      252525252525252525252525252525252525252525252525252525252525252

      52525252525252525252525252525252525252525252525252Bcleaning/

      Thank you,

      Jonathan

      1 Reply Last reply Reply Quote 0
      • AlanMosley
        AlanMosley last edited by

        You can do so in robots.txt

        SeoMoz bot is called rogerbot

        JonathanGoodman 1 Reply Last reply Reply Quote 1
        • JonathanGoodman
          JonathanGoodman @AlanMosley last edited by

          Thanks! That was easy.

          1 Reply Last reply Reply Quote 0
          • 1 / 1
          • First post
            Last post
          • After I make corrections of my crawl diagnostics report, how can I tell is those corrections "took". Is there a way to immediatly refresh that report. Will it eventually refresh?'
            Brian-H
            Brian-H
            0
            3
            200

          • Only few pages (308 pages of 1000 something pages) have been crawled and diagnosed in 4 days, how many days till the entire website is crawled complete?
            DarinPirkey
            DarinPirkey
            0
            4
            288

          • In my crawl diagnostics, there are links to duplicate content. How can I track down where these links originated in?
            kirklandsl
            kirklandsl
            0
            3
            307

          • Is it possible to exclude pages from Crawl Diagnostic?
            GCSMasone
            GCSMasone
            1
            2
            425

          • Crawl Diagnostics and missing meta tags on noindex blog pages
            ShaMenz
            ShaMenz
            0
            2
            835

          • My website has 18500 pages but my SEO MOZ campaign is limited to a 10,000 page crawl. How can I get the other 8500 pages crawled? Can I use one of my 3 spare campaigns?
            kenneth_martin
            kenneth_martin
            0
            5
            1.0k

          • "Issue: Duplicate Page Content " in Crawl Diagnostics - but these pages are noindex
            AaronWheeler
            AaronWheeler
            0
            2
            1.1k

          • Can you help me get started using the crawl diagnostics report?
            KeriMorgret
            KeriMorgret
            0
            4
            1.1k

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy