Removing CSS & JS Files from Index

[kirmeliux]

Hi,

Google has indexed a few .CSS and .JS files that belong to our WordPress plugins and themes. I had them blocked via robots, but realized this doesn't prevent indexation (and can likely hurt us since Google wants to access these files).

I've since removed the robots instructions, submitted a removal request via Search Console, but want to make sure they don't come back.

Is there a way to put a noindex tag within .CSS and .JS files? Or should I do something with .htaccess instead?

[Mobilio]

TL;DR - You're hesitated about problem that doesn't exist.

Googlebot doesn't index CSS or JS files. They index text files, HTML, PDF, DOC, XLS, etc. But doesn't index style sheets or javascript files.

All you need in WordPress is to create blank robots.txt file where WP is installed with this content:

User-agent: *
Disallow:
Sitemap: http://site/sitemap-file-name.xml

And that's all. This is explain many times:

http://googlewebmastercentral.blogspot.bg/2014/05/understanding-web-pages-better.html
http://googlewebmastercentral.blogspot.bg/2014/10/updating-our-technical-webmaster.html

[kirmeliux]

Unless I'm severely misreading the links provided, which I've read before, it seems Google is stating that they read, render, and sometimes index .CSS and .JS files. Here's an article written a week after the second article you posted.

The aforementioned WordPress plugin and theme files hosted on my server are indeed showing up in Google SERPs.

I do not want to prevent Googlebot from reaching these files as they're needed for optimal site performance, but I do want them to be no-indexed. Thus, I don't want robots.txt to prevent crawling, only indexing.

Let me know if I'm misunderstanding.

[Niels.V]

I haven't encountered undesirable .css or .js indexing myself (yet), but as you surmised, maybe this htaccess directive might be worth trying?

<filesmatch ".(txt|log|xml|css|js)$"="">Header set X-Robots-Tag "noindex"</filesmatch>

Google seems to support it

[TimHolmes]

I would seriously think again when it comes to blocking/no-indexing your CSS and JS files - Google has in the past stated that if they cannot fully render your site properly then this could lead to poorer rankings.

You will also likely get notifications in your Search Console as errors for this too.

Check out this great article from July this year which goes into more details. 

[Niels.V]

There's quite a big difference between crawling directives, which block and indexing directives. This article by (former?) Moz user S_ebastian_ is a good foundation read.

This article at developers.google.com is a good second read. If I'm understanding it right, Google thinks in terms of crawling directives vs indexing / serving directives.

My attempt at <tl rl="">:</tl>

crawling = looking, using in any way :: controlled via robots.txt

indexing / serving = indexing, archiving, displaying snippets in results, etc :: controlled via html meta tags or web server htaccess (or similar for other web servers).

I'm not convinced yet, that asking for noindex via htaccess causes the same sort of grief that deny in robots.txt causes.

[kirmeliux]

Hi Tim,

Assigning a noindex tag to these files will not block them, only prevent them from showing in SERPs. This is the intended goal and the reason I deleted my robots.txt file which prevented crawling.

[kirmeliux]

I figured .htaccess would be the best route. Thank you for researching and confirming. I appreciate it.