The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. How to block search bots in crawling my site except for homepage?

    How to block search bots in crawling my site except for homepage?

    Intermediate & Advanced SEO
    3 3 145
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • esiow2013
      esiow2013 last edited by

      Thanks in advance!

      1 Reply Last reply Reply Quote 0
      • GPainter
        GPainter last edited by

        You can use robots meta great guide here - http://moz.com/learn/seo/robotstxt

        or if you wanted to be a bit different you can hide content behind a login screen or flash etc. things robots cant get through, I'd recommend the robots though

        Hope that helps.

        1 Reply Last reply Reply Quote 0
        • JaneCopland
          JaneCopland last edited by

          Robots.txt exclusion is definitely the easiest way to go. The URLs within the site might still be "indexed", but they will not be crawled and if they ever showed up in a search, they would look like something like this: http://i.imgur.com/xU6mDYA.png

          1 Reply Last reply Reply Quote 0
          • 1 / 1
          • First post
            Last post
          • Website Indexing Issues - Search Bots will only crawl Homepage of Website, Help!
            LoganRay
            LoganRay
            0
            2
            278

          • Google cache is showing my UK homepage site instead of the US homepage and ranking the UK site in US
            adzhass
            adzhass
            0
            12
            828

          • Robots.txt - Do I block Bots from crawling the non-www version if I use www.site.com ?
            PatrickDelehanty
            PatrickDelehanty
            0
            2
            479

          • When Mobile and Desktop sites have the same page URLs, how should I handle the 'View Desktop Site' link on a mobile site to ensure a smooth crawl?
            DirkC
            DirkC
            0
            3
            1.4k

          • I want to block search bots in crawling all my website's pages expect for homepage. Is this rule correct?
            GPainter
            GPainter
            0
            7
            135

          • Duplicate site (disaster recovery) being crawled and creating two indexed search results
            Dr-Pete
            Dr-Pete
            0
            4
            195

          • Should product searches (on site searches) be noindex?
            NakulGoyal
            NakulGoyal
            0
            5
            680

          • Best way to block a search engine from crawling a link?
            deltasystems
            deltasystems
            0
            3
            1.0k

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy