SEO impact on archives and mirror services?

[mandaloriansecurity]

Hi,

We're looking at setting up an email archive site for certain lists related to our niche. We've been considering using archives.ourdomain.com versus ourdomain.com/archives. Part of the problem is that while the archives won't initially have huge amounts of content, over time it'll outstrip the amount of content we produce and I was worried about whether or not this would cause a duplicate content penalty as the emails will also be archived elsewhere.

If the volume of mails over time combined with duplication penalties are going to penalise the site, I'd rather we used a subdomain or a separate domain. If not, then subdomains might be the way to go.

[yeagerd]

You have a few choices here to avoid that.

If you want to avoid it completely, simply include this entire subdomain.domain.tld in the robots.txt as:

User-agent: *
Disallow: /
Disallow: *

Then place it in the root of your subdomain folder, for example if you sub was at subdomain.domain.tld you would normally have a folder called subdomain within your normal root folder.

Do not place this robots.txt file in your main folder or the whole site will not be indexed.

Extra measure, you can place a:

in the head section of each page.

[yeagerd]

I meant include this robot in the subdomain not include the subdomain in the robot.  Was answering an employee at the same time and crossed some wires.