The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. Sub Domains and Robot.txt files...

    Sub Domains and Robot.txt files...

    Technical SEO Issues
    3 2 160
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Vizergy
      Vizergy last edited by

      This is going to seem like a stupid question, and perhaps it is but I am pulling out what little hair I have left.

      I have a sub level domain on which a website sits.  The Main domain has a robots.txt file that disallows all robots.  It has been two weeks, I submitted the sitemap through webmaster tools and still, Google has not indexed the sub domain website.  My question is, could the robots.txt file on the main domain be affecting the crawlability of the website on the sub domain?  I wouldn't have thought so but I can find nothing else.

      Thanks in advance.

      1 Reply Last reply Reply Quote 0
      • TakeshiYoung
        TakeshiYoung last edited by

        The way that Google finds robots.txt files is by taking your URL, and adding /robots.txt to it. So a good way to see if the robots.txt file is affecting your subdomain is to go to subdomain.domain.com/robots.txt. If the file exists, then it is affecting your subdomain. If it doesn't, then it's only active on your main domain.

        Getting indexed is function of having unique content and pagerank, so make sure your subdomain has unique content and links if you're having trouble getting it indexed. Submitting a sitemap is no guarantee that Google will index your site.

        Vizergy 1 Reply Last reply Reply Quote 1
        • Vizergy
          Vizergy @TakeshiYoung last edited by

          Thank you, Mr. Young.  I believed this to be the case (that it wasn't the robots.txt file) but I could think of nothing else.  I have since been indexed.

          1 Reply Last reply Reply Quote 0
          • 1 / 1
          • First post
            Last post
          • Blocking subdomains with Robots.txt file
            PaulM01
            PaulM01
            0
            3
            641

          • Multiple robots.txt files on server
            peakdistrictseo
            peakdistrictseo
            0
            8
            2.4k

          • One server, two domains - robots.txt allow for one domain but not other?
            LesleyPaone
            LesleyPaone
            0
            3
            551

          • Meta Robots Noindex and Robots.txt File
            Devanur-Rafi
            Devanur-Rafi
            0
            2
            125

          • Will Links to one Sub-Domain on a Site hurt a different Sub-Domain on the same site by affecting the Quality of the Root Domain?
            ifbyphone
            ifbyphone
            0
            5
            153

          • Is my robots.txt file working?
            Resultify
            Resultify
            0
            5
            1.1k

          • Does Bing ignore robots txt files?
            Nightwing
            Nightwing
            0
            3
            2.8k

          • Using hyphenated sub-domains or non-hyphenated sub-domains? What is the question! I Any takers?
            dcmike
            dcmike
            0
            11
            5.5k

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy