The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. Sub Domains and Robot.txt files...

    Sub Domains and Robot.txt files...

    Technical SEO Issues
    3 2 160
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Vizergy
      Vizergy last edited by

      This is going to seem like a stupid question, and perhaps it is but I am pulling out what little hair I have left.

      I have a sub level domain on which a website sits.  The Main domain has a robots.txt file that disallows all robots.  It has been two weeks, I submitted the sitemap through webmaster tools and still, Google has not indexed the sub domain website.  My question is, could the robots.txt file on the main domain be affecting the crawlability of the website on the sub domain?  I wouldn't have thought so but I can find nothing else.

      Thanks in advance.

      1 Reply Last reply Reply Quote 0
      • TakeshiYoung
        TakeshiYoung last edited by

        The way that Google finds robots.txt files is by taking your URL, and adding /robots.txt to it. So a good way to see if the robots.txt file is affecting your subdomain is to go to subdomain.domain.com/robots.txt. If the file exists, then it is affecting your subdomain. If it doesn't, then it's only active on your main domain.

        Getting indexed is function of having unique content and pagerank, so make sure your subdomain has unique content and links if you're having trouble getting it indexed. Submitting a sitemap is no guarantee that Google will index your site.

        Vizergy 1 Reply Last reply Reply Quote 1
        • Vizergy
          Vizergy @TakeshiYoung last edited by

          Thank you, Mr. Young.  I believed this to be the case (that it wasn't the robots.txt file) but I could think of nothing else.  I have since been indexed.

          1 Reply Last reply Reply Quote 0
          • 1 / 1
          • First post
            Last post
          • Is repurposing an old sub domain better than creating a new sub domain?
            VeroBrain
            VeroBrain
            1
            2
            66

          • Robots.txt blocking Addon Domains
            eglove
            eglove
            0
            10
            556

          • One server, two domains - robots.txt allow for one domain but not other?
            LesleyPaone
            LesleyPaone
            0
            3
            551

          • Does this robots.txt file look right?
            ThompsonPaul
            ThompsonPaul
            0
            8
            188

          • How to prevent duplicat content issue and indexing sub domain [ CDN sub domain]?
            Dr-Pete
            Dr-Pete
            0
            3
            328

          • Robots.txt file
            Asher
            Asher
            0
            3
            261

          • Using hyphenated sub-domains or non-hyphenated sub-domains? What is the question! I Any takers?
            dcmike
            dcmike
            0
            11
            5.5k

          • Robots.txt File Redirects to Home Page
            NickPateman81
            NickPateman81
            0
            6
            4.1k

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy