Does this robots.txt file look right?

[Furious-D]

This post is deleted!

[Martijn_Scheijbeler]

Hi Sean,

Like you already said, I wouldn't recommend your current robots.txt as it would indeed block all files ending with .html. So I would go with your own robots.txt file with only the User Agent.

Good luck!

[Furious-D]

Thank you, Martijn!

[ThompsonPaul]

Actually, while I agree this is not a worthwhile robots.txt file, it's not blocking all html files. In order to do that, it would need to be using the " ***** " wildcard, not the slash.

I suspect this was a miguided attempt to use the robots.txt to deal with the .html version of your homepage. (Which needs to be done using redirects in the .htaccess file instead.)

You do have an issue with your home page though.

www.rfsystemlab.us has been 301-redirected to www.rfsystemlab.us/home.html. This is the opposite of what you want, unless you have some very unusual requirements that aren't visible from the site. You always want the raw domain www.rfsystemlab.us to resolve as your home page. The www.rfsystemlab.us/home.html should actually be redirected back to the primary domain URL, instead of vice versa as you have it now. (You also need the non-www version rfsystemlab.us to 301-redirect to the www.rfsystemlab.us as well.)

It looks like this was a recent change as your primary URL is still indexed, but I'd get that corrected asap before you start messing up your home page rankings.

Paul

P.S. The most universally accepted robots file that doesn't block anything is

User-agent: *
Disallow:

By having nothing after the disallow directive, it essentially means allow everything to be indexed. Google and Bing also recognize the Allow directive, even though it's not standard, so you could also use:

User-agent: *
Allow: /

But both these do the same thing and the first is more standards compliant, so that's my preference. I also like to add in the path to my xml sitemap as standard as well, so the final file would be:

User-agent: *
Disallow:

Sitemap: http://www.yoursite.com/path/to/sitemap.xml

The advantage to having a default sitemap is it will help keep your error logs cleaner, since a call to a non-existent robots.txt will show as a constant error in your error logs, making it harder to spot the real errors.

[Furious-D]

Paul,

Thank you for your answer.  I agree with everything you are saying.  I am not sure if we can setup the redirects properly.  The site is built on the SiteKreator.com platform.

I will take your advice on the proper use or the robots.txt fil and also include a link to my xml sitemap.

[Furious-D]

Another question here regarding the raw domain:

How crucial is it to have the bare domain version of the site redirected to the www version?

[ThompsonPaul]

The problem with having both the www version and the non-www version resolving is that the search engines consider those to be two separate sites, F-D. Which means they can be considered duplicate content and competing against each other. And it means links to one don't count as links to the other.

So, absolute best practice is to pick one or the other as your primary, and redirect the secondary to it. (And any platform that calls it'self SEO-friendly should have that built in, or at least manually configurable)

That said, if there's no way you can get this done on the SiteKreator platform, there are a few things you can do to try to mitigate the problem: (in descending order of importance/ease)

1. Decide for yourself which is the primary version (in your case it should be the www version) and ONLY ever use that address when referencing your site. E.g. make sure any links you create use the full www version, whenever you type out your site address, use the full version etc.

2. Verify the two sites in Google Webmaster Tools - one for the www address and one for the non-www version (you have to create them as separate sites because  as mentioned, the SEs consider the two URLs to be separate sites) Once verified, use the Configuration -> Settings page to set the Preferred domain to the primary you have chosen in both sites.

3. If there's any way to insert canonical tags into the header of your pages, set   for the home page.

4. Occasionally check the incoming links to the non-www pages and if they're high-link-juice-value, see if you can get the linking site to update their link with the full URL.

Fortunately, at the moment you only have a couple dozen incoming links from a couple of root domains coming to the non-primary URLs so it's not a massive yet. The problem will build if others who you can't control link to you using the non-primary URL. Which is why best practice is to redirect to take care of even those instances.

Do the steps as listed above and you'll mitigate the problem as much as possible within the limitations of your platform.

Hope that all makes sense?

Paul

[ThompsonPaul]

Thanks for the backup, Cyrus! Totally agree w/ you on the preference for using blank Disallow - as I mentioned about it being more standards-complaint.

Paul