I'm pulling my hair out trying to figure out why google stopped crawling.. any help is appreciated

[malady]

When I post something on the internet, freaked out or not, that's never an invite to go looking for my personal contact information to contact me outside of this medium.

Anyway.. thanks for your help.. My situation might not be like yours, I was looking for actual advice tips and knowledge on things that might really be wrong. What I got was google hates your guts, but hey I'm in the same boat maybe they just manually told you to screw off.

How does that help me in any way when what I'm looking for are things on my site that I can use to improve my situation.  I know you're well meaning, but you're not giving me any answers here.  I'm trying to improve the SEO on my site.. not get phone calls from guys telling me their story in the middle of the night because I made a mistake in putting my url. I'll ask my question in another thread elsewhere.

Thanks

[loopyal]

well, I'm sorry I wasted your time, like you, I'm looking for answers, and  thats all. Don't worry, I won't respond to any of your threads again, I did the best I could.

If you look back through the things I said, you may find something useful. Don't underestimate your own powers of investigation and testing.

I can almost guarantee you this - there is no simple answer. Nobody has it, other than google. The only way to work out what it is - is to ask, listen, experiment and test. There are a lot of smart people in SEOMoz, and if they haven't responded, it is because they don't have "the answer" you want.

And just remember, nobody responding to you here, if they do respond, is making any money from giving you suggestions, nor are they trying to wind you up. This is a voluntary thing of people helping people. The only possible reward is a thumbs up on a post, but I see I didn't give you anything that was even remotely useful.

I hope you figure it out.

[Dr-Pete]

Unfortunately, it can be very difficult to separate a bad history from a penalty from a large-scale technical problem, especially on large sites. I've seen many people assume they got hit by Panda when it was really a link-based penalty, and vise-versa. The site's history makes this go from difficult to nearly impossible, at least without a very deep dive, but I'll see what I can see.

Alan's right on one thing - Google Webmaster Tools has huge gaps in what they warn you about, and it's typically only manual penalties. Many sites have massive problems that never trigger a warning from Google.

I notice that you're NOINDEX'ing even high-level pages (in the navigation), such as:

http://dajaz1.com/music/alternative/

That seems like a bad message to Google - if it's important enough to appear in navigation, it's important enough to index. That's a pretty extreme culling of pages.

The paginated content is a bit of a mess, such as:

http://dajaz1.com/page/350/

In some cases, these don't even seem to return any results, so I'm not sure how they got crawled in the first place. The trick with META NOINDEX here is that, until Google re-crawls, they won't process the tag. This gets tricky, but I'd recommend a couple of possibilities:

(1) If the page returns no results, 301-redirect to the last page of search that has results.

(2) If none of these pages have search value, you could block "/page" as a folder in Google Webmaster Tools. This is a bit dangerous, so I'd want to make sure none of these pages had search value.

Are you getting any page load (speed) warning? Hitting your site intially is massive - about 4MB, by my count, with a ton of JavaScript, most of which just fuels the top, rotating images (which are loading very slowly on my machine). It seems like overkill, both from an SEO and usability standpoint, and is probably in your way to recovery. I'd seriously consider stripping down the size of the code and pruning back some of the active elements for a while.

If you can re-open the important paths, get rid of the thin content (this is going to be complicated and probably involves multiple steps), and speed up the site, you'll know enough to see if this is a technical issue (such as Panda).

There is certainly weak ranking even on your indexed pages, which could indicate a penalty, but it's really tough to tell. Too much of your content is competitive or uses shared phrases or videos, so it's hard to see whether a search for:

"Dwayne Wade 70-Foot Buzzer Beater" ...has you in 6th place for competitive reasons or because your site has been devalued. I don't think it's a penalty, at least in this case. It's a YouTube video and there are other, similar videos for a fairly recent, competitive term, so this may be an accurate ranking (in Google's POV).

The history is a lot tougher, and Q&A just isn't adequate to comment on a situation that complicated, as there are not only SEO but legal ramification. Honestly, I'd have to know a lot more details on that. If you suspect the history has hit you permanently, there may come a time when you have to completely re-brand and re-launch under a new domain.

I suspect, though, that cleaning up the crawl problems, removing the thinnest content, speeding up the site, and generally fixing some technical issues could help quite a bit. It's going to be a difficult process, though. The thing about changes like the Panda Update, is that it's not just one factor. I can't point to one thing and say "fix this" - you have to aggressively attack multiple factors, since Google is wrapping multiple signals into Panda and won't tell you which one is the problem.

I should say that I'm not saying this is Panda, but that it's a Panda-like situation - you've got a lot of crawl/index issues that are going to cause you problems. The question is whether those are compounded by your history (and, unfortunately, they probably are). The combination means that you have to be even more aggressive with the clean-up.

[malady]

Thanks for the response. Honestly I don't know why it's indexing the pages like that. I was told to make the categories and tags noindex, follow because google is indexing us, but it's indexing our pages, categories and tags. We've even had search results pages show up.. But it doesn't seem to like our actual posts.

I've turned the wp seo plugin off because honestly I'm just confused and not sure what to do.  The site was innocent, so I would think that it wouldn't be penalized for it. It was (and still sorta is) a case that's mainstream media and they are fully aware of the situation all the way up the food chain at google. We were very active in the anti SOPA/PIPA and used by companies like google and the eff as an example of why it was a bad law. I can't see google penalizing us for that.

searching site:dajaz1.com  none of our blog posts come up. I don't understand why it's not ranking them, but it will rank our pages with numbers behind them, or categories and tags nor do I understand how to fix it.

we've been noindex,follow on the categories, tagged, pages, search, and author pages for about a week now, and it's not indexing new stuff, and it's still not indexing our actual blog posts.

So if II should put them to index our categories in the nav bar, what's to stop them from indexing them over our blog posts.. because our posts are not ranking at all. (but our category pages are? )

That scroller at the top I think you're talking about is a gawd awful Nike Ad. We're on a ad network so we're stuck with it.. however that occurred after all this ranking stuff started.

I'm confused about the site speed, it loads very quickly for me, scores a 86/100 on google page test, 87/100 on webpagetest.org.. but yet shows slow load in analytics, webmaster tools, and occasionally people will tell me it loads slow for them.  Unless it's that ad, in which case there's not much I can do about it. We're in a contract with the ad network (which is very  a reputable and well known one)

[Dr-Pete]

The category/tag issue is tricky - sometimes, it does make sense to NOINDEX/FOLLOW, if those categories and tags spin out into a ton of internal search pages. My worry is that you're noindex'ing main navigation links - which is really a mixed signal. Those pages are important enough to get top billing, but not to index (?)

I have a feeling the best answer is a balance, but that takes a really deep knowledge of the site to sort out. If it's only been a week, it's really tough to tell. On a big site and deeper pages, it can take weeks to sort out, I'm afraid to say. NOINDEX is far from instantaneous.

Your main categories could potentially get ranked over your blog posts in some cases, but again, that's partially the site architecture. If you don't want them to rank, why are they main navigation links? Another, more radical options would be to kill the drop-down menus and add a layer (each main-nav home page would have left-side links or something to that effect) - that would drive more link-juice to the main sections and articles, and less to the sub-categories.

There's something very wrong with your article/blog navigation. If I click the ">>", I jump to page 417, but then there are dozens of empty pages. This is part of why all those paginated versions are getting indexed. I don't know if this is a WP issue or a quirk of your implementation (maybe a bunch of blank entries in the database), but it's creating dozens or hundreds of empty pages for Google to crawl. Since that's available right from the home-page, it's definitely a problem.

[malady]

I removed the pagi navigation and just left it to the older and newer links to take care of that issue. Lets see if that works at all. I removed all the drop down navigation, though to be honest I'd added that in after this all started hoping to set some type of site structure.. which I clearly didn't do a good job of.

I'm ok with radical at this point, I just want our posts to start getting indexed again.  I really appreciate the suggestions..

[Dr-Pete]

SORRY: SEE NEXT COMMENT

The current and prev/next link are just showing up as blank for me now (no articles). Something's off in the database, I suspect. It's almost like the "newest" articles are blank.

[Dr-Pete]

Sorry - my browser auto-completed the "/350" paginated URL. That change looks good, actually - it'll simplify the crawl path and should prevent those weird, empty results pages. Not sure why those are happening (and you may still need to fix something), but sticking to "Older" / "Newer" should help. I doubt this is the whole problem, but it's definitely an issue.

[malady]

Well google made a strong return yesterday.. so that definitely helped. Not sure if they're indexing us (though my sitemap no longer says 16 indexed and now 2000 indexed. And still not positive how many of those are pages and categories, but it looks like some of the posts are starting to make it under site:dajaz1.com

Thank you so much for your help. I'm not sure the issue is fixed ( I currently have all SEO plugins disabled and I'm too scared to turn them on) but at least google is crawling again and the next few days should tell the tale.

Another thing I did was got rid of the sub categories and turned them into tags. Hopefully this will help with the site structure, even if we take an initial hit with 404's at this point I don't want the tags indexed anyway since I want google to focus on the posts.

jeuWV

[Dr-Pete]

Glad to hear there's at least some progress. So often in these cases, I'm seeing that there are multiple things going on, and all you can do is fix them one at a time until the situation gets clearer. Give it a little time and then make the next set of changes - hopefully, it's all cumulative. Just make sure you measure progress and give things time to work. Where I see the most problems is when someone makes a change, it doesn't work in 3-4 days and then they reverse it or pile on new changes. Half the time, Google hasn't even re-cached 90% of the pages at that point.