How to fix google index filled with redundant parameters

[iragless]

Hi All

This follows on from a previous question (http://moz.com/community/q/how-to-fix-google-index-after-fixing-site-infected-with-malware) that on further investigation has become a much broader problem.  I think this is an issue that may plague many sites following upgrades from CMS systems.

First a little history.  A new customer wanted to improve their site ranking and SEO.  We discovered the site was running an old version of Joomla and had been hacked.  URL's such as http://domain.com/index.php?vc=427&Buy_Pinnacle_Studio_14_Ultimate redirected users to other sites and the site was ranking for buy adobe or buy microsoft.  There was no notification in webmaster tools that the site had been hacked.  So an upgrade to a later version of Joomla was required and we implemented SEF URLs at the same time.  This fixed the hacking problem, we now had SEF url's, fixed a lot of duplicate content and added new titles and descriptions.  Problem is that after a couple of months things aren't really improving.  The site is still ranking for adobe and microsoft and a lot of other rubbish and the urls like http://domain.com/index.php?vc=427&Buy_Pinnacle_Studio_14_Ultimate are still sending visitors but to the home page as are a lot of the old redundant urls with parameters in them.  I  think it is default behavior for a lot of CMS systems to ignore parameters it doesn't recognise so http://domain.com/index.php?vc=427&Buy_Pinnacle_Studio_14_Ultimate displays the home page and gives a 200 response code.

My theory is that Google isn't removing these pages from the index because it's getting a 200 response code from old url's and possibly penalizing the site for duplicate content (which don't showing up in moz because there aren't any links on the site to these url's)  The index in webmaster tools is showing over 1000 url's indexed when there are only around 300 actual url's.  It also shows thousands of url's for each parameter type most of which aren't used.

So my question is how to fix this, I don't think 404's or similar are the answer because there are so many and trying to find each combination of parameter would be impossible.  Webmaster tools advises not to make changes to parameters but even so I don't think resetting or editing them individually is going to remove them and only change how google indexes them (if anyone knows different please let me know)

Appreciate any assistance and also any comments or discussion on this matter.

Regards, Ian

[AlanMosley]

If the site is returning 200's then that is where the problem lies, you need to find out why.

I can see any other fix, removing the urls is only a temp fix, you must make them return 404's

[iragless]

Thanks for your input Alan

There lies my problem.  The URL's don't exist but give a 200 response.

http://domain.com/index.php?vc=427&Buy_Pinnacle_Studio_14_Ultimate is the same as

http://domain.com/index.php which is the same as

http://domain.com/?type_anything_here_and it still gives a 200 response.  Joomla seems to just ignore parameters from non existing pages after the ?.  I found a lot of people are having similar problems here http://forum.joomla.org/viewtopic.php?f=618&t=699954\.

Once in googles index I can't see a way of getting rid of thousands or redundant entries.  I have the added problem of the site being hosted on a Zeus Web Server which isn't as well documented as apache.

I'm currently looking into wild cards in robots.txt. It will be a slow process to get rid of them all but might finally help me clean up the index.

Ian

[iragless]

OK Might have a solution that would at least work for my situation.

Since implementing SEF URL's on the site I have no real need for any URL's with parameters.  By adding the following to robots.txt it should prevent any indexing of old pages or pages with parameters.

Disallow: /index.php?*

Tested it in webmaster tools with some of the offending URL's and it seems to work.  I'll wait until the next indexing and post back or mark it as answered.

[AlanMosley]

I can see it fixed your problem, but its a ugly fix, you mean need to use parameters in the future, you may already be using them but unaware.

[modernmagic]

the ability to remove the index.php is built into the stock joomla .htaccess file.

In the joomla backend, global config / site tab/ seo settings > enable "Use URL rewriting".

[iragless]

Thanks again Alan.

I've checked the site with screaming frog and it doesn't return any url's with parameters so at this stage I might be ok.  I am getting a message in webmaster tools saying "severe health issues" but it doesn't appear to be affecting the urls I want to keep.  I'll likely remove the entry once things have cleared up some more.

Thanks Jeff

At the moment I'm stuck with Zeus web server (insert expletives here) so no htaccess file or I'd be in a better position.  After messing around with it and very limited documentation I can only get the site operating with index.php in the url but with SEF url's for the remainder of it.  I'm investigating migration to an apache server so that might make it easier.

Regards

Ian