Could you use a robots.txt file to disalow a duplicate content page from being crawled?

[gregelwell]

A website has duplicate content pages to make it easier for users to find the information from a couple spots in the site navigation. Site owner would like to keep it this way without hurting SEO.

I've thought of using the robots.txt file to disallow search engines from crawling one of the pages. Would you think this is a workable/acceptable solution?

[KyleChamp]

The best way would be to use the Rel canonical tag

On the page you would like to rank for put the Rel canonical tag in

This lets google know that this is the original page.

Check out this link posted by Rand about the Rel canonical tag [http://www.seomoz.org/blog/canonical-url-tag-the-most-important-advancement-in-seo-practices-since-sitemaps](http://www.seomoz.org/blog/canonical-url-tag-the-most-important-advancement-in-seo-practices-since-sitemaps)

[anthonytjm]

Well, the answer would be yes and no. A robots.txt file would stop the bots from indexing the page, but links from other pages in site to that non indexed page could therefor make it crawlable and then indexed. AS posted in google webmaster tools here:

"You need a robots.txt file only if your site includes content that you don't want search engines to index. If you want search engines to index everything in your site, you don't need a robots.txt file (not even an empty one).

While Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web. As a result, the URL of the page and, potentially, other publicly available information such as anchor text in links to the site, or the title from the Open Directory Project (www.dmoz.org), can appear in Google search results."

I think the best way to avoid any conflict is applying the rel="canonical"  tag to each duplicate page that you don't want indexed.

You can find more info on rel canonical here

Hope this helps out some.

[Adam.Whittles]

I'm not sure I understand why the site owner seems to think that the duplicate content is necessary?

If I was in your situation I would be trying to convince the client to remove the duplicate content from their site, rather than trying to find a way around it.

If the information is difficult to find then this may be due to a problem with the site architecture. If the site does not flow well enough for visitors to find the information they need, then perhaps a site redesign is necessary.

[gregelwell]

Anthony, Thanks for your response. See Kyle, he also felt using the rel canonical tag was the best thing to do. However he seemed to think you'd put it on the original page - the one you want to rank for. And you're suggesting putting on the duplicate page. Should it be added to both while specifying which page is the 'original'?

Thanks!

Greg

[gregelwell]

Thanks Kyle. Anthony had a similar view on using the rel canonical tag. I'm just curious about adding it to both the original page or duplicate page? Or both?

Thanks,

Greg

[anthonytjm]

per google webmaster tools:

If Google knows that these pages have the same content, we may index only one version for our search results. Our algorithms select the page we think best answers the user's query. Now, however, users can specify a canonical page to search engines by adding a element with the attribute rel="canonical" to the section of the non-canonical version of the page. Adding this link and attribute lets site owners identify sets of identical content and suggest to Google: "Of all these pages with identical content, this page is the most useful. Please prioritize it in search results."

[gregelwell]

Next time I'll read the reference links better

Thank you!

[Dr-Pete]

Generally agree, although I'd just add that Robots.txt also isn't so great at removing content that's already been indexed (it's better at prevention). So, I find that it's not just not ideal - it sometimes doesn't even work in these cases.

Rel-canonical is generally a good bet, and it should go on the duplicate (you can actually put it on both, although it's not necessary).

[gregelwell]

Peter, Thanks for the clarification.

[KyleChamp]

Yeah, sorry for the confusion. I put the tag on all the pages (Original and Duplicate). I sent you a PM with another good article on Rel canonical tag