Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Does RogerBot read URL wildcards in robots.txt

Technical SEO Issues

2 2 928

Log in to reply

AspenFasteners last edited by

I believe that the Google and Bing crawlbots understand wildcards for the "disallow" URL's in robots.txt - does Roger?
1 Reply Last reply
Reply Quote 0
jennita last edited by

Hi! Yes, our crawler does read wildcards in a robots.txt file. This change was made in the past 3 months or so. Hope that helps!

Jen
1 Reply Last reply
Reply Quote 2

Browse Questions

View

From

Sorted by

With category