The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. Sub Domains and Robot.txt files...

    Sub Domains and Robot.txt files...

    Technical SEO Issues
    3 2 160
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Vizergy
      Vizergy last edited by

      This is going to seem like a stupid question, and perhaps it is but I am pulling out what little hair I have left.

      I have a sub level domain on which a website sits.  The Main domain has a robots.txt file that disallows all robots.  It has been two weeks, I submitted the sitemap through webmaster tools and still, Google has not indexed the sub domain website.  My question is, could the robots.txt file on the main domain be affecting the crawlability of the website on the sub domain?  I wouldn't have thought so but I can find nothing else.

      Thanks in advance.

      1 Reply Last reply Reply Quote 0
      • TakeshiYoung
        TakeshiYoung last edited by

        The way that Google finds robots.txt files is by taking your URL, and adding /robots.txt to it. So a good way to see if the robots.txt file is affecting your subdomain is to go to subdomain.domain.com/robots.txt. If the file exists, then it is affecting your subdomain. If it doesn't, then it's only active on your main domain.

        Getting indexed is function of having unique content and pagerank, so make sure your subdomain has unique content and links if you're having trouble getting it indexed. Submitting a sitemap is no guarantee that Google will index your site.

        Vizergy 1 Reply Last reply Reply Quote 1
        • Vizergy
          Vizergy @TakeshiYoung last edited by

          Thank you, Mr. Young.  I believed this to be the case (that it wasn't the robots.txt file) but I could think of nothing else.  I have since been indexed.

          1 Reply Last reply Reply Quote 0
          • 1 / 1
          • First post
            Last post
          • Is repurposing an old sub domain better than creating a new sub domain?
            VeroBrain
            VeroBrain
            1
            2
            66

          • Multiple robots.txt files on server
            peakdistrictseo
            peakdistrictseo
            0
            8
            2.4k

          • One robots.txt file for multiple sites?
            RyanPurkey
            RyanPurkey
            0
            4
            1.1k

          • Meta Robots Noindex and Robots.txt File
            Devanur-Rafi
            Devanur-Rafi
            0
            2
            125

          • Robots.txt file
            Asher
            Asher
            0
            3
            261

          • Does Bing ignore robots txt files?
            Nightwing
            Nightwing
            0
            3
            2.8k

          • How does robots.txt affect aliased domains?
            Dr-Pete
            Dr-Pete
            0
            2
            1.3k

          • Using hyphenated sub-domains or non-hyphenated sub-domains? What is the question! I Any takers?
            dcmike
            dcmike
            0
            11
            5.5k

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy