Default Robots.txt in WordPress - Should i change it??

[rootwaysinc]

I have a WordPress site as using theme Genesis  i am using default robots.txt. that has a line Allow: /wp-admin/admin-ajax.php, is it okay or any problem. Should i change it?

[Martijn_Scheijbeler]

Hi Peter,

Usually I would say it's not enough as the robots.txt is forgeting about excluding the search pages and in most cases you want to make sure the WP core files are not included + tag pages. Take a look at our robots.txt to see what we've included there: http://thenextweb.com/robots.txt then you'll notice we include for example these:

User-agent: *
Disallow: ?p=
Disallow: /wp-includes/
Disallow: /wp-login.php
Disallow: /wp-admin/*
Disallow: /wp-register.php
Disallow: /wp-content/themes/icetea/includes/*
Disallow: /tag/
Disallow: ?s=
Disallow: /search/*

Other cases in our robots.txt are very specifically in there because of our site and may not apply to others.

[Mobilio]

In addition of Martijn here is mine robots.txt:

User-agent: *
Disallow:

Sitemap: http://peter.nikolow.me/sitemap_index.xml

But using Yoast - categories, tags, most of archives and other generated pages are disabled for indexing.

[rootwaysinc]

Thank you both for your response.

@Martijn your robots.txt is really a nice example but for my new site is it good practice to block this areas??

@Peter To be a safe side I was using the same robots.txt...

[Martijn_Scheijbeler]

Yes, we're a news site as well and in our case we want to make sure the low quality pages on TNW aren't indexed.