The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. Should I disallow via robots.txt for my sub folder country TLD's?

    Should I disallow via robots.txt for my sub folder country TLD's?

    Intermediate & Advanced SEO
    2 2 241
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Shawn124
      Shawn124 last edited by

      Hello,

      My website is in default English and Spanish as a sub folder TLD.  Because of my Joomla platform, Google is listing hundreds of soft 404 links of French, Chinese, German etc. sub TLD's. Again, i never created these country sub folder url's, but Google is crawling them.  Is it best to just "Disallow" these sub folder TLD's like the example below, then "mark as fixed" in my crawl errors section in Google Webmaster tools?:

      User-agent: *

      Disallow: /de/

      Disallow: /fr/

      Disallow: /cn/

      Thank you,

      Shawn

      1 Reply Last reply Reply Quote 0
      • eyepaq
        eyepaq last edited by

        Joomla will do that to you 🙂

        To answer your questions: Yes, the use of robots.txt in this case makes sense. You will save some crawling budget that can be spent by Google's bot somewhere else.

        I would't worry about the WMT errors though - nothing bad can happen if you have them there and if you solve the issue those will go away - no need to spent time on those - it dosen't affect your performance in any way.

        Hope it helps.

        Cheers.

        1 Reply Last reply Reply Quote 1
        • 1 / 1
        • First post
          Last post
        • Can 'follow' rather than 'nofollow' links be damaging partner's SEO
          LureCreative
          LureCreative
          0
          4
          112

        • Migrating From Parameter-Driven URL's to 'SEO Friendly URL's (Slugs)
          PatrickDelehanty
          PatrickDelehanty
          0
          2
          292

        • Google's 'related:' operator
          EpicWebStudios
          EpicWebStudios
          0
          5
          209

        • Robots.txt file - How to block thosands of pages when you don't have a folder path
          Klarke
          Klarke
          0
          4
          264

        • Soft 404's from pages blocked by robots.txt -- cause for concern?
          CleverPhD
          CleverPhD
          1
          6
          2.0k

        • Disallow my store in robots.txt?
          AlanMosley
          AlanMosley
          0
          2
          308

        • Is there any negative SEO effect of having comma's in URL's?
          gcdtechnologies
          gcdtechnologies
          0
          3
          2.6k

        • Robots.txt disallow subdomain
          oznappies
          oznappies
          0
          7
          1.9k

        Get started with Moz Pro!

        Unlock the power of advanced SEO tools and data-driven insights.

        Start my free trial
        Products
        • Moz Pro
        • Moz Local
        • Moz API
        • Moz Data
        • STAT
        • Product Updates
        Moz Solutions
        • SMB Solutions
        • Agency Solutions
        • Enterprise Solutions
        • Digital Marketers
        Free SEO Tools
        • Domain Authority Checker
        • Link Explorer
        • Keyword Explorer
        • Competitive Research
        • Brand Authority Checker
        • Local Citation Checker
        • MozBar Extension
        • MozCast
        Resources
        • Blog
        • SEO Learning Center
        • Help Hub
        • Beginner's Guide to SEO
        • How-to Guides
        • Moz Academy
        • API Docs
        About Moz
        • About
        • Team
        • Careers
        • Contact
        Why Moz
        • Case Studies
        • Testimonials
        Get Involved
        • Become an Affiliate
        • MozCon
        • Webinars
        • Practical Marketer Series
        • MozPod
        Connect with us

        Contact the Help team

        Join our newsletter
        Moz logo
        © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
        • Accessibility
        • Terms of Use
        • Privacy