The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. Google crawling the demo site, robots exclusion already placed

    Google crawling the demo site, robots exclusion already placed

    Intermediate & Advanced SEO
    2 2 541
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • dotsquares
      dotsquares last edited by

      This post is deleted!
      1 Reply Last reply Reply Quote 0
      • donford
        donford last edited by

        HI Bankim

        Google will allow you to remove your own pages. Here is some help with that..

        Also what we do with our test sites is lock them down with .htaccess and .htpasswd by requiring a login you will never have to worry about any crawler indexing your test environment and thus worry about duplicate content when you go live.

        1 Reply Last reply Reply Quote 0
        • 1 / 1
        • First post
          Last post
        • Google Mobile site crawl returns poorer results on 100% responsive site
          WebQuest
          WebQuest
          0
          4
          37

        • How to make Google index your site? (Blocked with robots.txt for a long time)
          SanjidaKazi
          SanjidaKazi
          0
          3
          128

        • Proper 301 in Place but Old Site Still Indexed In Google
          DougRoberts
          DougRoberts
          0
          6
          10.6k

        • Google attempting crawl of URLs with parameter randomly placed in URL
          CleverPhD
          CleverPhD
          0
          5
          1.6k

        • Google Places Listing Active In Two Seperate Google Places Accounts?
          MiriamEllis
          MiriamEllis
          0
          2
          118

        • Should I let Google crawl my production server if the site is still under development?
          DoItHappy
          DoItHappy
          0
          4
          291

        • Is My site seeing a Google Dance ? - Rankings all over the place
          cbielich
          cbielich
          0
          2
          651

        • Working out exactly how Google is crawling my site if I have loooots of pages
          wazza1985
          wazza1985
          0
          3
          875

        Get started with Moz Pro!

        Unlock the power of advanced SEO tools and data-driven insights.

        Start my free trial
        Products
        • Moz Pro
        • Moz Local
        • Moz API
        • Moz Data
        • STAT
        • Product Updates
        Moz Solutions
        • SMB Solutions
        • Agency Solutions
        • Enterprise Solutions
        • Digital Marketers
        Free SEO Tools
        • Domain Authority Checker
        • Link Explorer
        • Keyword Explorer
        • Competitive Research
        • Brand Authority Checker
        • Local Citation Checker
        • MozBar Extension
        • MozCast
        Resources
        • Blog
        • SEO Learning Center
        • Help Hub
        • Beginner's Guide to SEO
        • How-to Guides
        • Moz Academy
        • API Docs
        About Moz
        • About
        • Team
        • Careers
        • Contact
        Why Moz
        • Case Studies
        • Testimonials
        Get Involved
        • Become an Affiliate
        • MozCon
        • Webinars
        • Practical Marketer Series
        • MozPod
        Connect with us

        Contact the Help team

        Join our newsletter
        Moz logo
        © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
        • Accessibility
        • Terms of Use
        • Privacy