No index

[Happy-SEO]

Screaming frog spider does index pages on our website like:

• wp-content/plugins/woocommerce/assets/js/frontend/jquery-ui-touch-punch.min.js?ver=2.3.9
• wp-content/plugins/mailchimp-for-wp/assets/css/checkbox.min.css?ver=2.3.2

Is it a bad/good idea to set my parameters in Webmastertools and tell Google not to crawl pages that begin with wp/content?

Thanks!

[UmarKhan]

I will not suggest you to apply parameters in your robots.txt/WMT unless you're quite sure that your "wp-content" does not contain any content that you want to get indexed.

If you have any problem with Screaming Frog, you may restrict it to not crawl your site. If you're on WordPress try out this plugin "WP Ban",
https://lesterchan.net/portfolio/programming/php/#wordpress-wp-ban

Hope this helps!

[DirkC]

Hi,

If these files are used to render the content on screen - it's not a good idea to block these files from indexing.

Official Google recommendation is to allow these files to be indexed:
"If resources like JavaScript or CSS in separate files are blocked (say, with robots.txt) so that Googlebot can’t retrieve them, our indexing systems won’t be able to see your site like an average user. We recommend allowing Googlebot to retrieve JavaScript and CSS so that  your content can be indexed better.  "

Source: http://googlewebmastercentral.blogspot.be/2014/05/understanding-web-pages-better.html

Rgds,

Dirk

[Happy-SEO]

Hi Dirk,

Good explanation, that makes sense.

Thanks!