The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. International Issues
    4. Robots.txt issue with indexation

    Robots.txt issue with indexation

    International Issues
    4 2 210
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • profesia
      profesia last edited by

      Hello

      i have a problem with one of the rules for robots.txt

      i have a multilingual mutation of entire page on www.example.com/en/

      I want to make indexable /allow/ the main page under /en/

      but not indexable /disallow/ everything else under /en/*

      Please help me how to write the rule.

      1 Reply Last reply Reply Quote 0
      • SEOKeith
        SEOKeith last edited by

        Name the page you want indexing something and you can use the following:

        Disallow: /en/

        Allow: /en/index.html

        Always test robots.txt in google webmaster tools.

        Hope that helps,

        Keith

        profesia 1 Reply Last reply Reply Quote 0
        • profesia
          profesia @SEOKeith last edited by

          There is no option like

          /en/index.html

          The only adress where you can reach the english main page version is www.example.com/en/

          SEOKeith 1 Reply Last reply Reply Quote 0
          • SEOKeith
            SEOKeith @profesia last edited by

            Well put the rest of the content in a different directory then and disallow that, thats the only other solution I can think of...

            1 Reply Last reply Reply Quote 0
            • 1 / 1
            • First post
              Last post
            • How to check Robots.txt File
              fghgkjh1233
              fghgkjh1233
              1
              5
              194

            • Hreflang tags and canonical tags - might be causing indexing and duplicate content issues
              gfiorelli1
              gfiorelli1
              0
              9
              3.0k

            • Low Index: 72 pages submitted and only 1 Indexed?
              DSM_UK
              DSM_UK
              0
              10
              263

            • Backlinks issue
              DirkC
              DirkC
              0
              7
              112

            • Blocking domestic Google's in Robots.txt
              Bush_JSM
              Bush_JSM
              0
              5
              274

            • Can I add more than one XML sitemap in my Robots.txt File?
              gfiorelli1
              gfiorelli1
              0
              3
              3.7k

            • Geotargetting Issues
              EricaMcGillivray
              EricaMcGillivray
              0
              2
              527

            • Geolocation and Indexing
              RyanKent
              RyanKent
              0
              4
              755

            Get started with Moz Pro!

            Unlock the power of advanced SEO tools and data-driven insights.

            Start my free trial
            Products
            • Moz Pro
            • Moz Local
            • Moz API
            • Moz Data
            • STAT
            • Product Updates
            Moz Solutions
            • SMB Solutions
            • Agency Solutions
            • Enterprise Solutions
            • Digital Marketers
            Free SEO Tools
            • Domain Authority Checker
            • Link Explorer
            • Keyword Explorer
            • Competitive Research
            • Brand Authority Checker
            • Local Citation Checker
            • MozBar Extension
            • MozCast
            Resources
            • Blog
            • SEO Learning Center
            • Help Hub
            • Beginner's Guide to SEO
            • How-to Guides
            • Moz Academy
            • API Docs
            About Moz
            • About
            • Team
            • Careers
            • Contact
            Why Moz
            • Case Studies
            • Testimonials
            Get Involved
            • Become an Affiliate
            • MozCon
            • Webinars
            • Practical Marketer Series
            • MozPod
            Connect with us

            Contact the Help team

            Join our newsletter
            Moz logo
            © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
            • Accessibility
            • Terms of Use
            • Privacy