The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. Should search pages be disallowed in robots.txt?

    Should search pages be disallowed in robots.txt?

    Technical SEO Issues
    3 2 335
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • MichaelWeisbaum
      MichaelWeisbaum last edited by

      The SEOmoz crawler picks up "search" pages on a site as having duplicate page titles, which of course they do.  Does that mean I should put a "Disallow: /search" tag in my robots.txt?  When I put the URL's into Google, they aren't coming up in any SERPS, so I would assume everything's ok.  I try to abide by the SEOmoz crawl errors as much as possible, that's why I'm asking.  Any thoughts would be helpful.

      Thanks!

      1 Reply Last reply Reply Quote 0
      • Kotkov
        Kotkov last edited by

        What engine are you running and what is the exact url you see when you perform search?

        MichaelWeisbaum 1 Reply Last reply Reply Quote 0
        • MichaelWeisbaum
          MichaelWeisbaum @Kotkov last edited by

          Using Google and I'm entering the exact URL in the search.  No results are found.  I know this means it's not indexed, but why does the SEOmoz crawler pick it up?

          1 Reply Last reply Reply Quote 0
          • 1 / 1
          • First post
            Last post
          • Disallow wildcard match in Robots.txt
            effectdigital
            effectdigital
            0
            3
            1.0k

          • Should I block Map pages with robots.txt?
            imaginex
            imaginex
            0
            3
            73

          • How to use robots.txt to block areas on page?
            LauraHT
            LauraHT
            0
            8
            225

          • Block or remove pages using a robots.txt
            OlegKorneitchouk
            OlegKorneitchouk
            0
            2
            422

          • Allow or Disallow First in Robots.txt
            Net66SEO
            Net66SEO
            0
            12
            27.8k

          • Should I set up a disallow in the robots.txt for catalog search results?
            AlanBleiweiss
            AlanBleiweiss
            0
            5
            1.7k

          • Robots.txt File Redirects to Home Page
            NickPateman81
            NickPateman81
            0
            6
            4.1k

          • Search Engine blocked by robots.txt
            Goetzman
            Goetzman
            0
            7
            1.2k

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy