How to allow bots to crawl all but WP-content

[Tom3_15]

Thank you for the response. I'm still a little uncertain, does the version you wrote allow the bots to crawl the css and js as well?

Best

[GastonRiera]

Yes it does.

As I said earlier. Copy and paste that code into the robot.txt tester in any of your search console and try with some name.css or testing.js just for testing.
Check the image i've attached.

Hope it helps.
Best luck
GR

btsycPz

[Tom3_15]

Awesome. Thanks, Gaston!

[Tom3_15]

Hi Gaston,

I just wanted to follow up with you with one last question if possible. Would this allow my images and PDF's to be crawled & indexed still?

Thanks!

[GastonRiera]

Hi Tom,

Yes, this config will allow images to be crawled,

No, this config will block images to be crawled,as long as your wordpress has the defalt folder for images: /wp-content/uploads/year/month/image-name.png

How to know, super easy, where your images are stored? Go to the web where you can find an image... Then right clic and then copy link address. With that link you will find that folder structure.

Hope it helps.
Best luck.
GR

[Tom3_15]

Gaston,

Thanks for the fast reply! My images folder does follow that format, which is what makes me worrisome as we are blocking the wp-conent folder.

Thanks!

[GastonRiera]

Oh god, my mistake!
Im deeply sorry, yes, this configuration will block images! that follow that folder structure!

I'll correct myself.
Thanks for pointing it out!

[Tom3_15]

Thanks, Gaston. I should have been more clear about what I am looking to do. I currently am having an indexation issue. Somehow, pages are being automatically generated by WordPress.

These pages are often .txt files of information or code from plugins, all beginning with /wp-content/uploads/ in their URL. I have been manually removing them from the index and would like to now have them be uncrawlable.

Best

[Tom3_15]

Can I do so with:

Allow: *.jpg

Allow: *.png

[GastonRiera]

Yeap, with that you are allowing every file ending with that extension

[Tom3_15]

Thank you for the help, Gaston!