Welcome to the Q&A Forum

I don't think we have a satisfactory answer for you or for ourselves. So many things can affect when/whether/how metrics change that I'd be lying if I gave a single answer. It could be:

• Some sites earn more links on pages we do/don't crawl

• Some sites earn links from places that our metrics count less/more

• Some sites lose existing links

Which particular competitors have precisely which issues happening and what's a result of changes in PA/DA calculation or mozRank/mozTrust scaling is something we can't say today.

We could do this if we only crawled the same pages every index, but we know from experience this produces terrible results (as we both miss new pages and as in 1 month, ~15-20% of what we crawl decays with 404s/500s/30xs/etc). For now, our plan of attack is improving size and continuing to tweak the algorithms to get closer and more accurate correlations with Google's rankings.

I'm sorry this is so frustrating - it's a massively complex problem and we're doing the best we can, and working as hard as we can on improving. I don't think everything will be miraculously fixed in 3 months or 9 months or 24 months, but it will keep getting better, and we do believe we can make link research, link analysis and competitive comparison functionality very good over time. Even today, I think we've got the best product on the market for many of the functions here.

Thanks Ed! Britney (Muller), Russ (Jones), and I have been pushing the team really hard to get keyword rankings data alongside link data so you can see how the two correlate and connect up. I totally agree that's where the magic comes from in SEO -- accurate rankings + comprehensive link graph = SEO heaven. Hopefully Moz will get there soon!

I'm working on trying to bring that practice to light and, hopefully, help stop it (if possible): http://www.webpronews.com/can-your-site-lose-its-rankings-because-of-competitors-negative-seo-2012-04

Funny you should ask Russ Jones from the Moz team has been working on testing a mobile-first type of index for link data to see what that does to the link graph. He should have a post out before not too long on that topic.

As for what mobile en-GB means as a setting in Moz Analytics, it's basically where/how we rank track - the geography and language that's used to check rankings (in your case "En - English" and "GB - Great Britain").

Hi Donnie - my point with today's WB Friday video is to hopefully suggest and help site owners to make their sites shiny and clean, so that even if some bad links point your way (everyone gets some bad links - they're a natural part of the web) or someone from Google manually reviews your site or a compeitor sends a spam report to Google about you, you'll be largely immune.

Having a site that focuses on fantastic user experience and mixes in a balance of good keyword targeting, accessibility and SEO best practices is the best way to avoid any untoward action. I'd only be really worried about this update if you've been doing serious manipulative link acquisition and spammy on-site tactics that you haven't cleaned up.

Best of luck!

Thanks Andrew! We're very glad to have that in the product, too. Been far too long

Hi Jody - feel free to drop me a line directly (rand at moz dot com) and I can make sure you're part of it in the future. We won't have another until next year (2015), just FYI.

Cheers!

I think TOC (ala Wikipedia like EGOL recommended) could work just fine, though you may want to do a UX/usability test with some of your audience.

Re: the content - if you think/know that searchers are seeking long-form, highly detailed stuff, go ahead and give it to them! You'll know your audience far better than I. Only point I might make is that visuals, video, diagrams, and charts/data (e.g. maybe something like a visual showing what muscles are needed for each type of move and the exercises that simulate them) can go a long way, sometimes more powerfully than text alone.

Hi Sandi - Moz won't know about all the links to your site. We don't crawl as fast or comprehensively as Google, so Open Site Explorer will always be showing a sample set rather than a complete picture of every link.

I still use it on occasion, but we recognize that it was far more of a research project than anything else. I will say the Virante implementation might be worth considering, as it's both good for thinking through topic modeling stuff and potentially helpful on the pure keyword research/targeting front, too: http://www.virante.com/seo-tools/lda-content-optimizer

Our goal is to bring aboard some more folks with serious topic modeling experience and, hopefully, find even better methodologies and implementations for this project so we can eventually start grading pages and making recommendations closer to how Google/Bing do.

The engineer who developed the tool is now working inside Google, which we'll take as a compliment, though it's also a bit sad to lose him

We've found this to be very close as well! I believe the correlation coefficient was around 0.85 last time we ran it.

In terms of the value of mozRank - our intent was always to mimic Google's PageRank algo, not to build a metric that best represents how a page/site might rank. You're totally spot on to say PA/DA are a good choice for that (and we have some tests running to make them even better in the near future).

BTW - I should mention here since it's relevant that Linkscape's estimated index update of July 11 is now moved back to July 20th due to some processing errors on the Amazon cluster (and some mistakes on our end, too). Sorry for the delay, but on the plus side, we do expect it to be quite a high quality index update (deeper crawls and better domain diversity, too).

Hey gang - sorry for my delay getting in here. We had a crazy day.

Here's what happened and why OSE report downloads have been slow:

• Last week, we had some volume spikes. The new OSE V3 offers a lot more download functionality, and everyone's been using that functionality a lot (awesome, but also a big load)
• We had some technical operational issues that slowed things up on OSE for about 24 hours, creating a huge backlog of reports
• We tried to simultaneously scale the capacity and segment the downloaded requests so they wouldn't interfere with load on the rest of the platform
• Today, we finally cleared the queue fully and reports should be back to fairly speedy

I'm really sorry about this. The tech behind scaling is challenging and our engineering team worked their butts off, but I know there's no excuse when you need a report or to do some work. We're going to work very hard over the next few weeks to try and make sure we get better with uptime/responsiveness/speed/etc.

Thanks for the patience - and keep your eyes out for more improvements/upgrades in data quality, size and speed coming in the next few months.

Cheers,

Rand

Hi Dylan - I'm pretty sure they can be influenced, but to say precisely how is tough.

For those who are curious about what these are - this query currently shows some under the first result: https://www.google.com/search?q=site+explorer

I'd say they're likely triggered by some of the same things that make sitelinks happen on a broader scale:

• User/usage data showing navigation from that page to other subpages on the site
• Link connections from folks who link to the main ranking page and subpages
• Relevance overlap of content between the pages
• Link popularity/importance of the subpages
• Some inclination from Google that this particular site/set-of-pages are potentially a navigational result for the query (i.e. the user is searching specifically for that stuff, not necessarily in discovery mode for broader results)

Hope that helps!

Thanks a ton Sean! We have considered distributed computing as a way to help crawl, index, process, etc. It's so flattering and humbling to hear that you'd be willing to help out and that the community would, too

For now, we believe we can get to the index size/quality/freshness using our hosted system, but the engineering team will certainly be encouraged to hear that folks in our community might contribute to this. Distributed systems present their own challenges, and we'd have to write that code from scratch, but if we find that we can't do what we want with our existing network, we might reach out.

BTW - I wanted to let folks know that the team here does feel very confident that come December/January, we're going to be producing indices that reach exceptional quality bars. The problems we face are largely known, and we now have the team and the solutions to tackle it, so we're pretty excited.

Richard - this is just my personal opinion, and it's not based on a ton of recent experience (I haven't had sites banned/penalized by Google in a very long time now). However, my feeling is that if you've been hit by this update, it's a great catalyst for immediate change. I'd be thinking about all of the following:

• Is now the time to start a new site that's exclusively white hat?
• Can I salvage this site and what's involved in that?
• How do I get rid of the bad links?
• How do I explain all the things I've done to Google (through reconsideration) and are they likely to let me back in?

Obviously, it depends on your long/short term goals, why you're doing web marketing in the first place, what you want to accomplish in the next few years, etc. If you're in the short-term, make money on the web fast with as little effort as possible and don't want to build a brand, then it's a question of whether 301'ing the domain to a new one and trying some tactics Google's not-yet-devalued is better than trying reconsideration and removal of bad links.

If you're thinking long term and brand building and want to put in sweat equity now that might take years to pay off, I'd be thinking around the new domain vs. trying to salvage this one (or two).

Wish you luck whatever you choose!

Hey LL - totally hear you on both fronts. With Linkscape, Ryan's precisely right. We went crawling deeper on large sites, and, as I noted in the blog posts around this issue, encountered these binary files. We attempted a fix for the past index using a blacklist of file types. It only caught about 40% of them. We were hoping that would be closer to 90%.

We've got two new indices running. One that launches mid-November that will have more of these files removed but not all, and another in December when we think we'll have them all cleaned out.

In terms of Google's search query data in Keyword Difficulty, that does suck. Google turned off our API access and we've been going back and forth with them to try to turn it on. We, along with a few hundred other tools, got shut off all together and trying to get back in has been a long, painful process. However, the tool does not rely on this data to get accurate reports (difficulty doesn't care how many times something's searched, just how competitive the top results are, which is what we look at). Thus, the tool's still fully operational, and you can still grab the KW data from Google's AdWords tool or your own API access (not as convenient, I agree).

TL;DR - We're fixing the OSE binary files issue by December (maybe sooner) and working with Google to try to get the Adwords API data back in KW Diff (but no guarantees).

Hope that helps and a million thanks to Ryan for jumping in and helping out on this. He's not an official rep or a Moz employee, but seriously appreciate him going above and beyond.

I'm surprised at how well this still works, but it does:

1. Use an incognito browser window to remove account personalization
2. Use a query string like this: https://google.co.nz/search?q=your+keyword+terms&gl=us

With 2) above, you're removing the geographic bias of any particular region/IP address by searching in Google New Zealand, then re-geo-locating the search to the US. This will give you non-geo-biased results.

If you want to see how specific results look from a particular region, there's two semi decent options:

A) Use Google's Ad Preview Tool: https://adwords.google.com/apt/anon/AdPreview?__u=1000000000&__c=1000000000
B) Use the &near parameter, e.g. https://google.co.nz/search?q=your+keyword+terms&gl=us&near=seattle+wa

I don't know if I'd particularly recommend my beard style - I do it because I have no jawline

Certainly cool with the blog post, though I might have to do a follow-up on some real SEO heroes and why I'm not sure I belong in that group; have made plenty of mistakes, failed to scale a consulting business in SEO, had years of debt, etc.

Thanks for posting this Pete. I've been hearing from folks about these issues, too, and obviously a lot of questions and speculation is flying around Twitter.

I wonder if Google thinks the rankings are unconnected to the date issues, but searcher behavior or some other indirect attribute is having this commonly (but not universally) observable impact on sites employing the embeds/dates... Wouldn't be the first time (e.g. subdomains, rich snippets, CTR, etc. supposedly not having "direct impact" but then having obvious observable/testable impact).

Some good things to know about DA: - It's using an algorithm that examines PA across a domain - It's not a perfect metric by any means (though it is, on average, the best correlated domain-wide metric for estimating a site's performance in Google's SERPs) - The range of correlations is in the 0.15-0.25 area depending on the index update and on Google's changes, so we're still talking about something that only explains a small portion of why a site may rank (page-level factors, anchor text, user/usage data, social data, personalization, goegraphy, etc. are not taken into account) - When a site's DA is higher, but other metrics are lower, it probably has something to do with the link distribution across the site and/or the types of links pointing at the site. That said, definitely will ask Matt to jump in here and have a look. In terms of your request for a $100/month service having better accuracy - as I noted, DA is the best metric we've seen (or any webmaster/SEO I know has seen/measured). It's incredibly hard to build a link index, very hard to calculate a link graph, and then super tough to build a ranking model against thousands of Google's results. I agree that we should be better and we're always working toward that (see http://www.seomoz.org/blog/introducing-seomoz-updated-page-authority-and-domain-authority); this is just an insanely hard problem, due to the massive technology and algorithmic requirements involved in understanding and mathematically quantifying how Google ranks pages.