AJAX and High Number Of URLS Indexed

[RMATVMC]

I recently took over as the SEO for a large ecommerce site. Every Month or so our webmaster tools account is hit with a warning for a high number of URLS. In each message they send there is a sample of problematic URLS. 98% of each sample is not an actual URL on our site but is an AJAX request url that users are making. This is a server side request so the URL does not change when users make narrowing selections for items like size, color etc. Here is an example of what one of those looks like

Tire?0-1.IBehaviorListener.0-border-border_body-VehicleFilter-VehicleSelectPanel-VehicleAttrsForm-Makes

We have over 3 million indexed URLs according to Google because of this. We are not submitting these urls in our site maps, Google Bot is making lots of AJAX selections according to our server data. I have used the URL Handling Parameter Tool to target some of those parameters that are currently set to let Google decide and set it to "no urls" with those parameters to be indexed. I still need more time to see how effective that will be but it does seem to have slowed the number of URLs being indexed.

Other notes:

1. Overall traffic to the site has been steady and even increasing.

2. Google bot crawls an average of 241000 urls each day according to our crawl stats. We are a large Ecommerce site that sells parts, accessories and apparel in the power sports industry.

3. We are using the Wicket frame work for our website.

Thanks for your time.

[AxialDev]

You could try using a robots.txt with a wildcard to stop Google from visiting those URLs :

Disallow: /*Tire?

or

Disallow: /*?0

It would help to see a full URL example (and matching categories).

See: https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt : URL matching based on path values

[RMATVMC]

Axial Dev,

Thanks for responding I have considered the Robots disallow however my worry has been several Videos by Matt Cutts talking about how now that the Google Bot can make AJAX requests that the best practice is to allow it to do so. So that is why I have not thrown on all around disallow addition to our robots.txt file, but It is clearly having issues on our site distinguishing the difference between a server side AJAX request on our site vs an actual real URL that should be indexed

Below is Matt Cutts plea to allow Java script to be allowed to be crawled there are a few others out there as well.

Does anyone else have experience with AJAX server side requests being indexed and how they combated the issue?

watch?v=B9BWbruCiDc