What's wrong with this robots.txt

Leonie-Kramer

Hi. really struggling with the robots.txt file
this is it:

User-agent: *
Disallow: /product/

#old sitemap
Disallow: /media/name.xml

When testing in w3c.org everything looks good, testing is okay, but when uploading it to the server, Google webmaster tools gives 3 errors. Checked it with my collegue we both don't know what's wrong.

Can someone take a look at this and give me the solution.
Thanx in advance!

Leonie

Martijn_Scheijbeler

Hi Leonie, what are the 3 errors as it seems that the robots.txt file syntax is correct.

Leonie-Kramer

Hi, sorry forgot to mention that

syntax error @ User-agent: *

no user agent @ Disallow: /product/

no user agent @ Disallow: /media/name.xml

Thanx, Leonie

DeanAndrews

Hi,
Lines containing only a comment are discarded completely, and therefore do not indicate a record boundary however you may need to remove the line break (not 100% sure but worth testing):

User-agent: *
Disallow: /product/
Disallow: /media/bcc.xml

Martijn_Scheijbeler

Sounds more like a bug in the tool that you're as I tested the syntax just now in Google Webmaster Tools and it's not causing any issues there.

Leonie-Kramer

Okay i got these errors in webmaster tools, very strange it is

Leonie-Kramer

if i test the blocked url's they are blocked so it looks like the file is doing what's supposed to do. but still is strange i got these errors.

@Dean Andrews, thanx i will test it without empty lines, though have to wait for another deployment

BlueprintMarketing

if you want to find out anything that could possibly be wrong with that this tool is the holy grail of finding out what's wrong with robots.txt issues in my opinion just expect a lot more info than a simple response from it.

http://tools.seochat.com/tools/robots-txt-validator/

Sincerely,

Thomas

DeanAndrews

Thomas,

That's an awesome tool, thank you for sharing.

Leonie-Kramer

Thanx for the url: it gives a warning on

Disallow: /product/
and
Disallow: /media/bcc.xml

i wonder why?

BlueprintMarketing

Hi Dean happy to be of help!

BlueprintMarketing

Hi Leonie,

I believe that you should create a robots.txt file that allows for a user agent disallow a folder /media/ and /.xml file. make the Unwanted xml file a 410 it will be dead to Google. however I think I have come up with a solution below please try pasting that in if it does not work.

A another tool for building robots.txt files and comparing them to the existing file from the same company believe it or not is right here.

http://www.internetmarketingninjas.com/seo-tools/robots-txt-generator/

please note you are disallowing more than just media you are disallowing something that should be more like this is for the xml sitemap why not just set it to a 410 killing the link in Google's eyes then you will not have to Disallow.

User-agent: *
Disallow: /product/
Disallow: /media/
Disallow: /bcc.xml

Sitemap: http://example.com/sitemap_index.xml

putting your new site map in where I have placed a site map or where the rule above will give you the spot to put it will help you tell Google where your new site map resides along with of course submitting it to Google Webmaster tools and fetching it as a Google bot.

I would like to look at the architecture of your site if you're getting errors with what you showed me you can send me a private message and I promise I will respond if you are not comfortable showing the URL on Q&A.

I hope this is of help,

Thomas

BlueprintMarketing

By the way here is an outdated site map that has when it looks like errors that really is telling me the protocol for putting a site map inside a robots.txt file is not endorsed by Google or Bing however I truly feel it is helpful so I do it. I've also added extra video site maps from an external host which is what's throwing out the errors the red color of the disallows is not a error it is just letting you know they are being blocked. Hopefully this will be of help

bigger photo is right here as well please give me a look at what errors are getting

http://i.imgur.com/Xg7EXwO.png

http status: 200

Syntax check robots.txt on http://www.blueprintmarketing.com/robots.txt (359 bytes)

| Line | Severity | Code |
| 6 | Warning | The official standard does not include Sitemap support even though major crawlers (Google and Bing) support it. It is still nonstandard. |
| 7 | Warning | The official standard does not include Sitemap support even though major crawlers (Google and Bing) support it. It is still nonstandard. |
| 8 | Warning | The official standard does not include Sitemap support even though major crawlers (Google and Bing) support it. It is still nonstandard. |
| 9 | Warning | The official standard does not include Sitemap support even though major crawlers (Google and Bing) support it. It is still nonstandard. |
| 10 | Warning | The official standard does not include Sitemap support even though major crawlers (Google and Bing) support it. It is still nonstandard. |

Warnings Detected: 5

Errors Detected: 0

robots.txt source code for http://

Xg7EXwO.png

Leonie-Kramer

Hi, Thanx for your reply, i'm not sure i understand you by "please note you are disallowing more than just media"

the thing with this is the xml file is an old file but somewhere in the google archive. i tried do remove it with the wmt, but returns. It's not on the server anymore. the directory "media" doesn't exist anymore, also from an old website.

Because the file still returns in wmt i thought let's try it with the robots.txt

new robots.txt not tested waiting for deployment

Oh call me stupid, but how do i make a 410?

Grtz, Leonie

Leonie-Kramer

Hi ,

i got it working with a proper sitemap. Special thanks to Thomas for the great effort in his answers!

BlueprintMarketing

Hi Leonie,

That's very kind of you I am very happy that you got it working correctly.

All the best,

Thomas

BlueprintMarketing

Believe me it took me plenty of time to realize how to do this but if you're handy with SFTP or SSH you can change the

http://stackoverflow.com/questions/1975904/htaccess-to-redirect-all-traffic-to-one-page-410-gone?rq=1

And for the ultimate in ease if you're using WordPress there is actually a plug-in for 410s so it wasn't something anyone found easy to do.

https://wordpress.org/plugins/wp-410/

Sincerely,

Thomas

Leonie-Kramer

Ah thanks, it's an Azure platform, so no SFTP, SSH or .htaccess. but i'll give the stack link to the technical guys then they have to translate it to our environment ( .net)

BlueprintMarketing

I think thats a great Idea .net is not my thing.

All the best!

Tom

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.