How to fix Google index after fixing site infected with malware.

[iragless]

Thanks Alan

This seems to be done by the combination of Joomla/Zeus and the redirection manager.  No longer infected, the only visits are from organic searches from google and it's been a couple of months.  Whatever the reason Joomla feels it shouldn't 404 these pages and just displays (not 301 redirects them) to the home page.

My feeling is that these URL's in the index and the visits from them probably aren't doing the site any good.

[AlanMosley]

These pages may have links from other spam sites, you don't want them to return a 200.
You want them to 404, in joomla you can make the site use htaccess or not, make sure it dose and 404 the pages there.

[iragless]

Thanks Paul

I've checked analytics and the only source of these url's is google organic searches, not external sites.  I think unfortunately my problem is the dynamic nature of Joomla and a combination of a number of factors that are causing it to do this in an SEO unfriendly way.

I think my biggest challenge is getting the URL's to 404 before I submit them to the web master removal tool (which my research tells me needs to be done before you submit).  I think I read there might be a robots.txt option so I'll look into that.

Ian

[AlanMosley]

That means no body clicks on them, but how did google find them? This is not evidence there is no links, just that no one has visited your site thought them

[iragless]

I've checked the source of the visits and they are only coming form google searches for "cheap adobe" and the like.  The original malware used the site to get these searches into the index and then direct them to other sites/pages.

Being a Zeus server it doesn't use htaccess, my task would be a lot simple if it did.  It has an alternative rewrite file but documentation is scarce on using it for 404's.

I'll keep researching.

[iragless]

You're right, I guess I was focused on the index.  Moz isn't showing any external links to these pages and neither is webmaster tools.  My feeling is that google is retaining them for some reason, maybe just the keywords in the url?

[ThompsonPaul]

The search engines are retaining the indexing of the links because following them through the redirect returns a 200 server header - which to the SEs means all is well and there is a page there to index. As you note in other responses - the only way to change that is to force the server to return a 404 header as a signal to the SEs to eventually drop it.

Yes, you could use a robots.txt directive to block those specific URLs that are the target of the spam links, in order to satisfy the URL Removal Tool's requirement for allowing a removal request. That should work as a quicker solution than trying to make coding changes in Joomla (sorry, it's been about 3.5 yrs since I've done any Joomla work).

Good luck!

Paul

[EDIT: Gah...ignore the P.S. as I didn't notice you don't have an easy way to get redirects into the Zeus server before Joomla kicks in. Sorry]

P.S. A final quick option would be to write a redirect in htaccess to 301-redirect the fake URLs to a real 404 page. This would kick in before Joomla got a chance to interfere with its pseudo-redirect.

[iragless]

Thanks all for you help

A little more information and maybe a little more advice required.

Since fixing the malware http://domain.com/index.php?vc=201&Cheap_Adobe_Acrobat_xi and similar are actually no longer pages.  Joomla actually sees anything after ? as a parameter and just ignores it because it no longer matches a page and hence the reason it just defaults to the home page http://domain.com/index.php.  This is Joomla and probably most other content management systems default behavior.  The problem here lies in the fact that google indexed that page when it was infected and it remains in the index because to google it sees a status code of 200 when re-indexing this page.

The problem is now a bit broader and has more ramifications than first thought.  Any pages from the previous system that used parameters would receive a 200 status code and remain in the index.  Checking url parameters in web master tools confirms this with various paramaters showing thousands of url's monitored.  Keep in mind google is showing a message that there are no problems with parameters for this site.

So the advice I need now is related to url parameters in Web Master tools.  The new site uses SEF URLS and so makes much less use of paramaters.  How can I ensure that the old redundant pages with parameters are dropped from the index.  This would involve thousands of 301's or 404's let alone trying to work them all out.  There is a reset link for each parameter in webmaster tools but not much documentation as to what it does.  If I reset all the parameters would that clean up the index?

I'd be interested in what others think about this issue because I feel that this might be a common problem with cms based platforms and after major changes, thousands of paramater based url's just defaulting to home and other pages probably affects the site and page ranking.

Ian

[iragless]

OK Might have a solution that would at least work for my situation.

Since implementing SEF URL's on the site I have no real need for any URL's with parameters.  By adding the following to robots.txt it should prevent any indexing of old pages or pages with parameters.

Disallow: /index.php?*

Tested it in webmaster tools with some of the offending URL's and it seems to work.  I'll wait until the next indexing and post back or mark it as answered.

[BlueprintMarketing]

I would make them return a 410 not 404

410's are dead links if you use a 404 google will keep coming back to see if you fixed the 404

sending google to a 410 lets them know it's gone

http://moz.com/learn/seo/http-status-codes

all the best,

tom

[iragless]

Thanks Tom

That's a good point.  Part of my problem lies in the number of URL's with parameters (thousands).  Applying status codes of any type isn't really viable.

Starting to see the url's clean up with the addition of the entries in robot.txt.

Regards

Ian