Robots.txt assistance

[theLotter]

I want to block all the inner archive news pages of my website in robots.txt - we don't have R&D capacity to set up rel=next/prev or create a central page that all inner pages would have a canonical back to, so this is the solution.

The first page I want indexed reads:
http://www.xxxx.news/?p=1

all subsequent pages that I want blocked because they don't contain any new content read:
http://www.xxxx.news/?p=2
http://www.xxxx.news/?p=3
etc....

There are currently 245 inner archived pages and I would like to set it up so that future pages will automatically be blocked since we are always writing new news pieces. Any advice about what code I should use for this?

Thanks!

[Andy.Drinkwater]

I haven't actually done this myself, but I suspect that pattern matching is your solution here.

However, what you want to be able to do is disallow the whole pattern and then allow just the first page:

Allow: /?p=1
Disallow: /?p=*

The thing I don't have the answer to, is if this will work by first allowing the page 1, and then blocking all others. I don't have a method for this in blocking via robots as this is normally handed with other solutions you mention.

You can try it though through Webmaster tools:
https://support.google.com/webmasters/answer/156449?hl=en

1. On the Webmaster Tools Home page, click the site you want.
2. Under Crawl, click Blocked URLs.
3. If it's not already selected, click the** Test robots.txt** tab.
4. Copy the content of your robots.txt file, and paste it into the first box.
5. In the URLs box, list the site to test against.
6. In the User-agents list, select the user-agents you want.

-Andy

[Martijn_Scheijbeler]

I think it has to be the other way around: Disallow: /?p=* Allow: /?p=1 as you want to first disallow everything with the P parameter but then allow the first page. You should test it but I think in Andy's example you will still block the first page which you've just allowed.

[Andy.Drinkwater]

Definitely something to test. I'm not sure of the rules that Google will apply with this and which way round works.

-Andy

[CleverPhD]

I think you are missing something here if you want to get these pages out of the index. Plus, your use of Robots may harm how Google finds and ranks your actual news items.

First, you have to add the noindex meta tag to pages 2-N in your pagination.  Let Google crawl them and take them out of the index.

If you just add them to robots.txt, Google will not crawl, but will also not remove them from the index.

Once you get them out of the index, keeping those tags in place will prevent reindexation and you don't have to add them to Robots.txt.

More importantly, you want pages 2-N being spidered but not indexed.  You want Google to crawl your paginated pages to find all of your deep content.  Otherwise, unless you have a XML or HTML sitemap, or some other crawlable navigational aid, you are actually preventing Google from crawling and then ranking your content.

Read this Moz post

http://moz.com/learn/seo/robotstxt

There is a section titled "Why Meta Robots is Better than Robots.txt" that will confirm my points.

Lastly. Step back a second. If you are a news/content site and this helps you to generate revenue, and you have a bunch of news pages, and this is important content, spend some money on Development to implement the rel=next/prev.  It is worth it to get Google crawling your stuff properly.

Good luck!

[Andy.Drinkwater]

If you read the original post again, Sara says "we don't have R&D capacity".

They wouldn't be able to do all this.

-Andy

[CleverPhD]

Thanks Andy.  I did see that and that is why I mentioned at the end that being a content site and if that generates revenue that they should consider investing some money in that direction.

If they are short on money/resources/capacity and the robots.txt solution could actually negatively impact indexation of content that is producing/justifying the current level of money/resources/capacity they could end up in worse position than where they started, i.e. having less money/resources/capacity.

[Andy.Drinkwater]

"I mentioned at the end that being a content site and if that generates revenue that they should consider investing some money in that direction"

Absolutely.

[theLotter]

Thanks for all the input and advice!

We are a gaming site that publishes industry news 2-3 times a week, but that is not our main source of income