The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. Robots.txt - Do I block Bots from crawling the non-www version if I use www.site.com ?

    Robots.txt - Do I block Bots from crawling the non-www version if I use www.site.com ?

    Intermediate & Advanced SEO
    2 2 479
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • morg45454
      morg45454 last edited by

      my site uses is set up at http://www.site.com I have my site redirected from non- www to the www in htacess file. My question is... what should my robots.txt file look like for the non-www site? Do you block robots from crawling the site like this? Or do you leave it blank?

      User-agent: *

      Disallow: /

      Sitemap: http://www.morganlindsayphotography.com/sitemap.xml

      Sitemap: http://www.morganlindsayphotography.com/video-sitemap.xml

      1 Reply Last reply Reply Quote 0
      • PatrickDelehanty
        PatrickDelehanty last edited by

        Hi there

        If you configured this properly, I wouldn't worry about this at all.

        Check your internal links and sitemap to make sure that your URLs listed as a reflection of this www. version.

        Beyond that, you're all good, no need to block non www.

        Hope this helps! Good luck!

        1 Reply Last reply Reply Quote 2
        • 1 / 1
        • First post
          Last post
        • We are redirecting http and non www versions of our website. Should all versions http (non www version and www version) and https (non www version) should just have 1 redirect to the https www version?
          BlueprintMarketing
          BlueprintMarketing
          0
          8
          437

        • Our Web Site Is candere.com. Its PA and back link status are different for https://www.candere.com, http://www.candere.com, https://candere.com, and http://candere.com. Recently, we have completely move from http to https.
          94501
          94501
          0
          3
          102

        • Switching site from non-www to www
          MonicaOConnor
          MonicaOConnor
          0
          7
          3.7k

        • I've submitted a disavow file in my www version account, should I submit one in my non-www version account as well?
          MonicaOConnor
          MonicaOConnor
          0
          3
          108

        • Robots.txt Blocked Most Site URLs Because of Canonical
          0
          1
          117

        • Can URLs blocked with robots.txt hurt your site?
          workzentre
          workzentre
          0
          4
          302

        • Www. & Non-www versions problem
          SanketPatel
          SanketPatel
          0
          5
          201

        • Could you use a robots.txt file to disalow a duplicate content page from being crawled?
          KyleChamp
          KyleChamp
          0
          11
          1.3k

        Get started with Moz Pro!

        Unlock the power of advanced SEO tools and data-driven insights.

        Start my free trial
        Products
        • Moz Pro
        • Moz Local
        • Moz API
        • Moz Data
        • STAT
        • Product Updates
        Moz Solutions
        • SMB Solutions
        • Agency Solutions
        • Enterprise Solutions
        • Digital Marketers
        Free SEO Tools
        • Domain Authority Checker
        • Link Explorer
        • Keyword Explorer
        • Competitive Research
        • Brand Authority Checker
        • Local Citation Checker
        • MozBar Extension
        • MozCast
        Resources
        • Blog
        • SEO Learning Center
        • Help Hub
        • Beginner's Guide to SEO
        • How-to Guides
        • Moz Academy
        • API Docs
        About Moz
        • About
        • Team
        • Careers
        • Contact
        Why Moz
        • Case Studies
        • Testimonials
        Get Involved
        • Become an Affiliate
        • MozCon
        • Webinars
        • Practical Marketer Series
        • MozPod
        Connect with us

        Contact the Help team

        Join our newsletter
        Moz logo
        © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
        • Accessibility
        • Terms of Use
        • Privacy