Google robots.txt test - not picking up syntax errors?

McTaggart

I just ran a robots.txt file through "Google robots.txt Tester" as there was some unusual syntax in the file that didn't make any sense to me...

e.g. /url/?*
/url/?
/url/*

and so on. I would use ? and not ? for example and what is ? for! - etc.

Yet "Google robots.txt Tester" did not highlight the issues...

Can anybody explain why Google didn't - or perhaps it isn't supposed to pick up such errors?

Thanks, Luke

beaupedraza

Hey Luke,

It appears that in each of the three examples, there was a plausible case for each example. Let's cover each:

For /url/?* , it can be expressed that a URL can offer a trailing slash and then a query string, see examples here.
with /url/? , this covers examples of the above and in addition, would plausibly block product pages that generate query strings, similar to this example from H&M. In essence, only allowing the product page to be seen.
/url/* , well, that's just anything and everything after the trailing slash.

I guess the question you should ask yourself is "Is this the best approach for the issue?"

McTaggart

Many thanks Beau - much appreciated.

Welcome to the Q&A Forum