The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. How to stop pages being crawled from xml feed?

    How to stop pages being crawled from xml feed?

    Intermediate & Advanced SEO
    2 2 368
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • jazavide
      jazavide last edited by

      We have a site that has an xml feed going out to many other sites. 
      The xml feed is behind a password protected page so cannot use a cannonical link to point back to original url.

      How do we stop the pages being crawled on all of the sites using the xml feed? as with hundreds using it after launch it will cause instant duplicate content issues?

      Thanks

      1 Reply Last reply Reply Quote 0
      • CoreyNorthcutt
        CoreyNorthcutt last edited by

        You'll probably want to disallow spiders from crawling them with robots.txt:

        http://www.robotstxt.org/robotstxt.html

        1 Reply Last reply Reply Quote 1
        • 1 / 1
        • First post
          Last post
        • Will google be able to crawl all of the pages given that the pages displayed or the info on a page varies according to the city of a user?
          0
          3
          17

        • Crawl Stats Decline After Site Launch (Pages Crawled Per Day, KB Downloaded Per Day)
          R0bin_L0rd
          R0bin_L0rd
          0
          4
          158

        • Best way to link to 1000 city landing pages from index page in a way that google follows/crawls these links (without building country pages)?
          lcourse
          lcourse
          0
          7
          54

        • How can I stop my facets being crawled?
          MeganSingley
          MeganSingley
          0
          4
          97

        • Xml sitemap Issue... Xml sitemap generator facilitating only few pages for indexing
          Paddy_Moogan
          Paddy_Moogan
          0
          6
          153

        • An affiliate website uses datafeeds and around 65.000 products are deleted in the new feeds. What are the best practises to do with the product pages? 404 ALL pages, 301 Redirect to the upper catagory?
          Everett
          Everett
          0
          5
          135

        • Large number of pages crawled.
          Cyrus-Shepard
          Cyrus-Shepard
          0
          3
          89

        • XML question - not finding all of the pages
          SanketPatel
          SanketPatel
          0
          4
          238

        Get started with Moz Pro!

        Unlock the power of advanced SEO tools and data-driven insights.

        Start my free trial
        Products
        • Moz Pro
        • Moz Local
        • Moz API
        • Moz Data
        • STAT
        • Product Updates
        Moz Solutions
        • SMB Solutions
        • Agency Solutions
        • Enterprise Solutions
        • Digital Marketers
        Free SEO Tools
        • Domain Authority Checker
        • Link Explorer
        • Keyword Explorer
        • Competitive Research
        • Brand Authority Checker
        • Local Citation Checker
        • MozBar Extension
        • MozCast
        Resources
        • Blog
        • SEO Learning Center
        • Help Hub
        • Beginner's Guide to SEO
        • How-to Guides
        • Moz Academy
        • API Docs
        About Moz
        • About
        • Team
        • Careers
        • Contact
        Why Moz
        • Case Studies
        • Testimonials
        Get Involved
        • Become an Affiliate
        • MozCon
        • Webinars
        • Practical Marketer Series
        • MozPod
        Connect with us

        Contact the Help team

        Moz logo
        © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
        • Accessibility
        • Terms of Use
        • Privacy