Wordpress error

[SEOKeith]

I gave you an example of a basic robots.txt file that I use on one of my Wordpress sites above, I would suggest using that for now.

I would not bother messing around with crawl delay in robots.txt as Peter said above there are better ways to achieve this... Plus I doubt you need it any way.

Google caches the robots.txt info for about 24hrs normally in my experience... So it's possible the old cached version is still being used by Google.

[NileCruises]

Thanks Keith.

Only part of our site is WP based. Would that be a problem using the example you kindly suggested?

[SEOKeith]

Use this:

**User-agent: *
Disallow: /blog/wp-admin/
Disallow: /blog/wp-includes/

Sitemap: http://nile-cruises-4u.co.uk/sitemap.xml**

Any FYI, you have the following error on your blog:

Warning: is_readable() [function.is-readable]: open_basedir restriction in effect. File(D:\home\nile-cruises-4u.co.uk\wwwroot\blog/wp-content/plugins/D:\home\nile-cruises-4u.co.uk\wwwroot\blog\wp-content\plugins\websitedefender-wordpress-security/languages/WSDWP_SECURITY-en_US.mo) is not within the allowed path(s): (D:\home\nile-cruises-4u.co.uk\wwwroot) in D:\home\nile-cruises-4u.co.uk\wwwroot\blog\wp-includes\l10n.php on line **339 **

Get your web guy to look at that, it appears at the top of every blog page for me...

Hope that helps,

Keith

[NileCruises]

Thanks very much Keith. I've just edited the file as suggested.

I see the error but as I am the web guy I cant' figure out how to get rid of it.

I think it might be a plugin that's causing it so I'm going to disable the and re-able them one as a time.

I've just PM'd you by the way.

Thanks for your help Keith.

Colin

[NileCruises]

Mind you the whole blog is now showing an error message and cant' be viewed so looks like an afternoon of trial and error!

[SEOKeith]

Looks like a 403 permissions problem, that's a server side error... Make sure you have the correct permissions set on the blog folder in IIS Personally I always host on Linux...

[NileCruises]

Thanks Keith. Just contacting out hosts.

Nightmare!

[NileCruises]

Blog isn't' showing now and my hosts say that the index.php file is missing from the directory but I can see it.

Strange.

Have contacted them again to see what the problem can be.

Bit of a wasted Saturday!

[Dr-Pete]

Google is seeing the same Robots.txt content (in GWT) that you show in the physical file, right? I just want to make sure that, when the site was hacked, no changes were made that are showing different versions of files to Google. It sounds like that's not the case here, but it definitely can happen.

[NileCruises]

Hi Peter,

I've tested the robot.txt file in Webmaster Tools and it now seems to be working as it should and it seems Google is seeing the same file as I have on the server.

I'm afraid this side of things isn't' my area of expertise so it's been a bit of a minefield.

I've taken a subscription with sucuri.net and taken various other steps that hopefully will hel;p with security. But who knows?

Thanks,

Colin

[SEO-Pump.com]

This will be my first post on SEOmoz so bear with me

The way I understand it is that robots read the robots.txt file from top to bottom, and once they find a rule that applies to them they stop reading and begin crawling.  So basically the robots.txt written as:

User-agent:*

Disallow:

Crawl-delay: 20

User-agent: 008

Disallow: /

would not have the desired result as user-agent 008 would first read the top guideline:

User-agent: *

Disallow:

Crawl-delay: 20

and then begin crawling your site, as it is first being told that All user-agents are disallowed to crawl no pages or directories.

The corrected way to write this would be:

User-agent: 008

Disallow: /

User-agent: *

Disallow:

Crawl-delay: 20