The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. Robots.txt: Can you put a /* wildcard in the middle of a URL?

    Robots.txt: Can you put a /* wildcard in the middle of a URL?

    Intermediate & Advanced SEO
    2 2 410
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • IHSwebsite
      IHSwebsite last edited by

      We have noticed that Google is indexing the language/country directory versions of directories we have disallowed in our robots.txt.

      For example:

      Disallow: /images/ is blocked just fine

      However, once you add our /en/uk/ directory in front of it, there are dozens of pages indexed.

      The question is: Can I put a wildcard in the middle of the string, ex. /en/*/images/, or do I need to list out every single country for every language in the robots file. Anyone know of any workarounds?

      1 Reply Last reply Reply Quote 0
      • irvingw
        irvingw last edited by

        Yes, wildcards work, thank god.

        1 Reply Last reply Reply Quote 1
        • 1 / 1
        • First post
          Last post
        • Block session id URLs with robots.txt
          Mat_C
          Mat_C
          1
          4
          130

        • Robots.txt & Disallow: /*? Question!
          BabaBha0173
          BabaBha0173
          0
          8
          321

        • If I block a URL via the robots.txt - how long will it take for Google to stop indexing that URL?
          GastonRiera
          GastonRiera
          0
          3
          98

        • Domain.com/old-url to domain.com/new-url
          Edzjus333
          Edzjus333
          0
          10
          162

        • Should /node/ URLs be 301 redirect to Clean URLs
          MargaritaS
          MargaritaS
          0
          3
          172

        • URL Redirect: http://www.example.net/ vs. http://www.example.net
          dotfly
          dotfly
          0
          2
          705

        • Blocking out specific URLs with robots.txt
          Modi
          Modi
          0
          3
          133

        • Reciprocal Links and nofollow/noindex/robots.txt
          DanCrean
          DanCrean
          0
          4
          1.2k

        Get started with Moz Pro!

        Unlock the power of advanced SEO tools and data-driven insights.

        Start my free trial
        Products
        • Moz Pro
        • Moz Local
        • Moz API
        • Moz Data
        • STAT
        • Product Updates
        Moz Solutions
        • SMB Solutions
        • Agency Solutions
        • Enterprise Solutions
        • Digital Marketers
        Free SEO Tools
        • Domain Authority Checker
        • Link Explorer
        • Keyword Explorer
        • Competitive Research
        • Brand Authority Checker
        • Local Citation Checker
        • MozBar Extension
        • MozCast
        Resources
        • Blog
        • SEO Learning Center
        • Help Hub
        • Beginner's Guide to SEO
        • How-to Guides
        • Moz Academy
        • API Docs
        About Moz
        • About
        • Team
        • Careers
        • Contact
        Why Moz
        • Case Studies
        • Testimonials
        Get Involved
        • Become an Affiliate
        • MozCon
        • Webinars
        • Practical Marketer Series
        • MozPod
        Connect with us

        Contact the Help team

        Join our newsletter
        Moz logo
        © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
        • Accessibility
        • Terms of Use
        • Privacy