Sudden influx of 404's affecting SERP's?

[ChimplyWebGroup]

Hi Mozzers,

We've recently updated a site of ours that really should be doing much better than it currently is. It's got a good backlink profile (and some spammy links recently removed), has age on it's side and has been SEO'ed a tremendous amount. (think deep-level, schema.org, site-speed and much, much more).

Because of this, we assumed thin, spammy content was the issue and removed these pages, creating new, content-rich pages in the meantime.

IE:

• We removed a link-wheel page; <a>https://www.google.co.uk/search?q=site%3Asuperted.com%2Fpopular-searches</a>, which as you can see had a **lot **of results (circa 138,000).

• And added relevant pages for each of our entertainment 'categories'.
  <a>http://www.superted.com/category.php/bands-musicians</a> - this page has some historical value, so the Mozbar shows some Page Authority here.
  <a>http://www.superted.com/profiles.php/wedding-bands</a> - this is an example of a page linking from the above page. These are brand new URLs and are designed to provide relevant content.

The old link-wheel pages contained pure links (usually 50+ on every page), no textual content, yet were still driving small amounts of traffic to our site. 
The new pages contain quality and relevant content (ie - our list of Wedding Bands, what else would a searcher be looking for??) but some haven't been indexed/ranked yet.

So with this in mind I have a few questions:

• How do we drive traffic to these new pages? We've started to create industry relevant links through our own members to the top-level pages. (http://www.superted.com/category.php/bands-musicians) The link-profile here _should _flow to some degree to the lower-level pages, right? We've got almost 500 'sub-categories', getting quality links to these is just unrealistic in the short term.
• How long until we should be indexed? We've seen an 800% drop in Organic Search traffic since removing our spammy link-wheel page. This is to be expected to a degree as these were the only real pages driving traffic. However, we saw this drop (and got rid of the pages) almost exactly a month ago, surely we should be re-indexed and re-algo'ed by now?!
• **Are we still being algor****hythmically penalised? **The old spammy pages are still indexed in Google (138,000 of them!) despite returning 404's for a month. When will these drop out of the rankings? If Google believes they still exist and we were indeed being punished for them, then it makes sense as to why we're still not ranking, but how do we get rid of them? I've tried submitting a manual removal of URL via WMT, but to no avail. Should I 410 the page?
• Have I been too hasty? I removed the spammy pages in case they were affecting us via a penalty. There would also have been some potential of duplicate content with the old and the new pages.
  _popular-searches.php/event-services/videographer _may have clashed with _profiles.php/videographer, _for example.
  Should I have kept these pages whilst we waited for the new pages to re-index?

Any help would be extremely appreciated, I'm pulling my hair out that after following 'guidelines', we seem to have been punished in some way for it. I assumed we just needed to give Google time to re-index, but a month should surely be enough for a site with historical SEO value such as ours? 
If anyone has any clues about what might be happening here, I'd be more than happy to pay for a genuine expert to take a look. If anyone has any potential ideas, I'd love to reward you with a 'good answer'.

Many, many thanks in advance.

Ryan.

[MonicaOConnor]

How do we drive traffic: In light of removing these old pages, and seeing a tremendous influx of 404 errors, I am assuming the pages that were removed were not 301 redirected. If that is the case I would strongly encourage you to redirect those old URLs to your new pages. This will help get traffic to the new pages which will eventually help them rank on their own. It sounds like your pages just dropped off the face of the planet and because the other pages are so new, you are losing all of your organic rankings and subsequently, organic traffic.

How long should you wait to be indexed: Did you submit a new sitemap to Google? I would make sure you have done that. After that, it shouldn't take very long, 2 weeks is the longest I have waited for an index after a sitemap submission.

As far as a penalty goes, check WMT. If you see nothing in there from Google I think you are safe on the penalty side. However, the sudden changes, the large amount of changes and the influx in 404 pages might have moved your site back in rankings while Google takes a look to make sure there isn't any nefarious activity. I wouldn't worry about a penalty unless you actually receive one.

If you are worried about duplicate content, try researching Rel canonical tags to see if they will be helpful to you. It sounds like you made a lot of changes quickly, and that Google needs time to investigate. Unfortunately, you have to kind of wait a little bit. Try the things I listed here, I hope that it helps!

[ChimplyWebGroup]

Hi Monica,

Thanks for the fast response.

I'm a bit wary of 301 redirecting the old pages -- this would be extremely easy to do - however: if we were being penalised on those old pages, wouldn't this just redirect all the penalties to our new squeaky clean pages?

I submitted a new sitemap to Google the day we made the changes -- probably about three weeks - a month ago including the new URLs (or as many as we could include with the 500 URL limit) and removing the old spammy ones.

Penalty-wise, we've never had a manual warning or anything in WMT. However, this doesn't discount the idea that we may have been suffering an algorithm penalty, right?

It'd be great to hear from anyone about their experience with 301's and the likeliness of passing on 'bad' linkjuice from old pages (does this even happen?).
Also whether a 410 would help - stops all the 404 errors from continuously occurring and Google assuming there's something bizarre going on.

Thank you again.