How to stop pages being crawled from xml feed?
-
We have a site that has an xml feed going out to many other sites.
The xml feed is behind a password protected page so cannot use a cannonical link to point back to original url.How do we stop the pages being crawled on all of the sites using the xml feed? as with hundreds using it after launch it will cause instant duplicate content issues?
Thanks
-
You'll probably want to disallow spiders from crawling them with robots.txt: