Wp-login.php

[MyAllenMedia]

To block or not to block with robots.txt?

[EugenOprea]

Do you have any links from your main pages pointing to wp-login.php?

If not, the search engines will not know about that URL, so you can choose to not block it.

But if you have links to wp-login.php and don't want the page indexed, then you should block it.

[MyAllenMedia]

Thanks for the reply. Makes sense. What about tags? Should they be blocked in robots.txt too? Reason I ask is that i have indexed tag content in Google and if I blog /tags/ from being indexed then I'm worried that I'll lose that indexed content. The reason for the blocking of /tags/ is because seomoz is finding duplicate content from /tags/ and I don't want that to hurt me.

[EugenOprea]

You may want to add a short description and custom titles for those tags. There are some ways to do that in WordPress.

This way your content will have something unique and will not be 100% duplicate.

[MattAntonino]

I block wp-admin and wp-includes just because they don't need to be indexed.

As far as tags, you can easily no-follow all of those with any half decent Wordpress SEO plugin like Yoast or All in One SEO.  I would definitely noindex them.

[MyAllenMedia]

Not sure I follow. I have indexed tag results on Google that have the /tag/ in the url. If i noindex them, won't i lose those urls that get indexed?

[MagicDude4Eva]

I would start with a robots.txt like the one below and add in anything else you don't want to be crawled/indexed:

Sitemap: http://www.example.com/sitemap.xml

# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*

# Google AdSense
User-agent: Mediapartners-Google*
Disallow:

# digg mirror
User-agent: duggmirror
Disallow: /

# global
User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /trackback/
Disallow: /feed/
Disallow: /comments/
Disallow: /category/*/*
Disallow: */trackback/
Disallow: */feed/
Disallow: */comments/
Disallow: /*?
Allow: /wp-content/uploads/

[MattAntonino]

You don't want to index tags in Wordpress.  You'll have duplicate content on any post that has more than one tag on it (which is almost all of them.)   You want to get the canonical permalinks indexed and that's it.  Search for a post you have indexed under a tag and then do a keyword search to see if it appears a few times (archives, multiple tag pages, etc.)  That's bad - you want to fix that.

[MyAllenMedia]

I understand that but what's happening is that i have tag results in the top 10 and not the post result. The post result is indexed but not anywhere close to the tag. If i block tags, in my head, I think i run the risk of losing traffic. I have to be 100% certain of this for I'm getting and been getting a lot of traffic from those "tag" results.

[MyAllenMedia]

With all the talk here about not indexing /tag/ why do you not include it in your robots?

[MattAntonino]

If you're happy with your results, why block anything in Robots?  The short term answer is tags are working for you.  The long term answer is you're creating duplicate content.  So it just depends - are you optimizing for today and tomorrow's sales or for long term success.  There is no "right" answer to that but that's the question.

[MyAllenMedia]

I get that. Just not sure about blocking them. A lot of sites don't block them and yoast.com is one of them. So, if big seo guys aren't blocking tag then are we 100% sure that Google sees "/tag/" as duplicate content?

[EugenOprea]

Michael, the best thing to do for you would be to check Google Webmasters Tools and see what Google sees there as duplicate content.

This will help you decide whether you want or not to block /tag/.