Robots.txt on refinements

[Gordian]

In dealing with Panda do you think it is a good idea to put all refinements for category pages in the robots.txt file? We already have a lot as noindex, follow but I am wondering if it would be better to address from a crawl perspective as the pages are probably thin duplicate content to Google.

[LesleyPaone]

I don't know if you have those taco commercials where you live that have the little girl in them that says "Why not both!", but you might do that, it would not hurt and it would make you sleep better at night.

Oh, here is a link to the commercial, https://www.youtube.com/watch?v=vqgSO8_cRio

[ThompsonPaul]

One of the most common mistakes I see in SEO... There's nothing about the robots.txt that keeps pages from being indexed. In fact, just the opposite. If you have existing pages to which you've added no-index, but you also block them with robots.txt, then the search crawler will never see them to pick up the no-index and therefore won't know it's supposed to remove them. So they would still count against you as thin content even though they're not being crawled. NOT the result you're looking for.

If you can no-index them, great. If not, at least use canonical tags to point them to the primary version of the category page. (Remember no-index is a FAR stronger command to the search engines then canonical tags, which they take as "suggestions".)

The only time it's appropriate to block no-indexed pages with robots is if you're absolutely certain the pages have never made it into the index in the first place. If they've never been indexed, you can no-index them for security, and then drop them behind the robots.txt to save crawl budget.

Hope that makes sense?

Paul

[evolvingSEO]

Hi There

In general you probably don't need to do that. Here's how I would normally deal with indexation in WordPress (assuming you're using WordPress);

• Categories - index
• Tags - noindex
• Date archives - noindex
• Author (single author blogs) - noindex
• Author (multi-author) - index
• Subpages - noindex

Basically all these settings are shown in my post here on setting up WordPress: http://moz.com/blog/setup-wordpress-for-seo-success

Yoast is the best plugin to do all this with!