The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. On-Page / Site Optimization
    4. Can we listed URL on Website sitemap page which are blocked by Robots.txt

    Can we listed URL on Website sitemap page which are blocked by Robots.txt

    On-Page / Site Optimization
    7 4 482
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Internet-Marketing-Profs
      Internet-Marketing-Profs last edited by

      Hi,

      I need your help here.

      I have a website, and few pages are created for country specific. (www.example.com/uk).

      I have blocked many country specific pages from Robots.txt file. It is advisable to listed those urls (blocked by robots.txt) on my website sitemap. (html sitemap page)

      I really appreciate your help.

      Thanks,

      Nilay

      1 Reply Last reply Reply Quote 0
      • JarnoNijzing
        JarnoNijzing last edited by

        Nilay,

        if you have blocked them from your robots.txt but you do enclude them in your sitemap xml or html then they will be indexed unless you enclode a meta robots in it with a noindex tag. If that tag is not in the pages and you enclude that page in your sitemap Google will feel it as important content in list it in the SERPs.

        Hope this helps

        Regards

        Jarno

        Internet-Marketing-Profs 1 Reply Last reply Reply Quote 0
        • Travis-W
          Travis-W last edited by

          Hi Nilay,

          I actually did this yesterday by accident.
          I recommend you remove the blocked pages from your XML sitemap, otherwise Google will display a "warning" after you submit it.

          As far as your HTML sitemap, it does not really matter.
          I think you are okay to keep the links there.

          Internet-Marketing-Profs 2 Replies Last reply Reply Quote 0
          • Internet-Marketing-Profs
            Internet-Marketing-Profs @JarnoNijzing last edited by

            Hi Jarno,

            Thanks for your time.

            Should i put URLs anywhere in my websites which are being excluded by Robot.txt?

            Thanks,

            Nilay

            1 Reply Last reply Reply Quote 0
            • Internet-Marketing-Profs
              Internet-Marketing-Profs @Travis-W last edited by

              Hi Zora,

              Thanks for your time.

              1 Reply Last reply Reply Quote 0
              • Internet-Marketing-Profs
                Internet-Marketing-Profs @Travis-W last edited by

                Hi Zora,

                But the pages which i have blocked are only visible in specific country. Addition, i have blocked theme also, so you think it's good to put those url on website or link them in website?

                irvingw 1 Reply Last reply Reply Quote 0
                • irvingw
                  irvingw @Internet-Marketing-Profs last edited by

                  if the content is of benefit to the user then include them in your navigation. Why are you blocking them in the first place, duplicate content?

                  1 Reply Last reply Reply Quote 0
                  • 1 / 1
                  • First post
                    Last post
                  • Description tag not showing in the SERPs because page is blocked by Robots, but the page isn't blocked. Any help?
                    DavidFaltz
                    DavidFaltz
                    0
                    9
                    183

                  • How to exclude URL filter searches in robots.txt
                    Everett
                    Everett
                    0
                    9
                    8.6k

                  • When You Add a Robots.txt file to a website to block certain URLs, do they disappear from Google's index?
                    Saijo.George
                    Saijo.George
                    0
                    3
                    205

                  • How can I reduce Too Many On-Page Links? I am looking for best method through which I can reduce by on page link.
                    dawnieando
                    dawnieando
                    0
                    6
                    173

                  • Can Sitemap Be Used to Manage Canonical URLs?
                    ThompsonPaul
                    ThompsonPaul
                    0
                    2
                    167

                  • New CMS system - 100,000 old urls - use robots.txt to block?
                    Blenny
                    Blenny
                    0
                    8
                    745

                  • Can bad text URLs hurt pages?
                    loopyal
                    loopyal
                    0
                    5
                    439

                  • How do you block development servers with robots.txt?
                    JustinTaylor88
                    JustinTaylor88
                    0
                    7
                    5.2k

                  Get started with Moz Pro!

                  Unlock the power of advanced SEO tools and data-driven insights.

                  Start my free trial
                  Products
                  • Moz Pro
                  • Moz Local
                  • Moz API
                  • Moz Data
                  • STAT
                  • Product Updates
                  Moz Solutions
                  • SMB Solutions
                  • Agency Solutions
                  • Enterprise Solutions
                  • Digital Marketers
                  Free SEO Tools
                  • Domain Authority Checker
                  • Link Explorer
                  • Keyword Explorer
                  • Competitive Research
                  • Brand Authority Checker
                  • Local Citation Checker
                  • MozBar Extension
                  • MozCast
                  Resources
                  • Blog
                  • SEO Learning Center
                  • Help Hub
                  • Beginner's Guide to SEO
                  • How-to Guides
                  • Moz Academy
                  • API Docs
                  About Moz
                  • About
                  • Team
                  • Careers
                  • Contact
                  Why Moz
                  • Case Studies
                  • Testimonials
                  Get Involved
                  • Become an Affiliate
                  • MozCon
                  • Webinars
                  • Practical Marketer Series
                  • MozPod
                  Connect with us

                  Contact the Help team

                  Join our newsletter
                  Moz logo
                  © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                  • Accessibility
                  • Terms of Use
                  • Privacy