September's Mozscape Update Broke; We're Building a New Index

[randfish]

Yeah - the new links you see via "just discovered" will take longer to be in the main index and impact metrics like MozRank, Page Authority, Domain Authority, etc. It's not that they're not picked up or not searched, but that they don't yet impact the metrics.

And yes - will check out the other question now!

[LayGiri]

I did notice no new links added to a number of projects in the last 2 months and I was wondering what went wrong. Thanks for clearing up the issue with this post. We look forward to the resolution.

[Highland]

And they would have gotten away with it too if weren't for those meddling kids and their pesky subdomains

[randfish]

Two potential solutions for you - 1) watch "Just Discovered Links" in Open Site Explorer - that tab will still be showing all the links we find, just without the metrics. And 2) Check out Fresh Web Explorer - it will only show you links from blogs, news sites, and other things that have feeds, but it's one of the sources I pay attention to most, and you can set up good alerts, too.

[Joe.Robison]

Thanks for the transparency as usual. A question I've always been wondering:

Moz seems to have much more stature, clout, and maybe funding compared to many other SEO software companies based around the world. And of course you offer more of a suite of products rather than just focusing on Open Site Explorer. But to me one of the most important SEO tools is the backlink explorer tools that companies offer, and it seems like OSE, although one of the first, lags compared to a few others. I've read that OSE isn't looking to just grab all the links, but only the most important ones. It seems though that there's been lots of technical challenges, and I can't help but think that there are other companies that have already solved their indexing challenges or are a few steps ahead of OSE.

Would Moz ever go out an buy a pretty good backlink explorer company like Ahrefs or Majestic or some other upstart that's solved that piece of the puzzle? Combining that new technology that's solved the indexing part with your DA algorithm seems like a match made in heaven. I'm sure you guys have considered this years ago internally, but it's a question I've always pondered...

[LayGiri]

Talk about reading everyone's mind... I should point out though that Rand mentioned above that moz was working on a new real time tool like the ones we have seen elsewhere. I think a little patience might solve everyone's problems.

[randfish]

Hi Joe - fair question.

The basic story is - what the other link indices do (Ahrefs and Majestic) is unprocessed link crawling and serving. That's hard, but not really a problem for us. We do it fairly easily inside the "Just Discovered Links" tab. The problem is really with our metrics, which is what makes us unique and, IMO, uniquely useful.

But, metrics like MozRank, MozTrust, Spam Score, Page Authority, Domain Authority, etc. require processing - meaning all the links needed to be loaded into a series of high-powered machines and iterated on, ala the PageRank patent paper (although there are obviously other kinds of ways we do this for other kinds of metrics). Therein lies the rub. It's really, really hard to do this - takes lots of smart computer science folks, requires tons of powerful machines, takes a LONG time (17 days+ of processing at minimum to get all our metrics into API-shippable format). And, in the case where things break, what's worse is that it's very hard to stop and restart without losing work and very hard to check our work by looking at how processing is going while it's running.

This has been the weakness and big challenge of Mozscape the last few years, and why we've been trying to build a new, realtime version of the index that can process these metrics through newer, more sophisticated, predictive systems. It's been a huge struggle for us, but we're doing our best to improve and get back to a consistent, good place while we finish that new version.

tl;dr Moz's index isn't like others due to our metrics, which take lots of weird/different types of work, hence buying/partnering w/ other indices wouldn't make much sense at the moment.

[KevnJr]

Thanks Rand for the update.  We have hired a full time marketing manager  and he has been working hard the past month, I know he's excited to see the new results.  "Putty & Paint does not a NEW Boat make"  Fixing is a painstaking reality compared to building.  Moz is great, so we will wait

[Mysites]

Hey Rand, is this why my crawl reports are saying that i have some 404 client errors on pages where I can't see any issues? Or is this another issue that I'm incurring?

Thanks in advance

[DavidLee]

Hi Lehia

Crawl reports are separate from our Mozscape indexes. Also any delays with our index only impact the ability to access new data. With your crawl reports I have a suspicion the URLs with the 404s are ones with trailing slashes e.g. domain.com/ and not domain.com

If not, send us your account info and some examples at help@moz.com and we can take a look!

[DmitriiK]

Hello, Rand. I just noticed that yesterday new update was scheduled for October 8th. And just now it says October 14th! What's going on? I hope it didn't break again...

[randfish]

It didn't break, but it is taking longer to process than we hoped. Very frustrating, but we have a plan that, starting in a few more weeks, should get us to much more consistent index releases (and better quality ones, too).

[DmitriiK]

Thanks!

I have been noticing for quite some time that last minute changes in update release dates are becoming "normal". Is there way you guys can make those changes in update dates be announced earlier than on the expected update release date?

[randfish]

Sometimes yes. Sometimes, we don't know until we reach the last stages of processing whether it's going to finish or take longer. We're trying to get better at benchmarking along the way, too, and I'll talk to the team about what we can do to improve our metrics as an index run is compiling.

[DmitriiK]

Thanks for clarifying!

[inkyj]

It's been about a month since the last update on this and just curious if there's any news on the progress?  I'm still seeing the same results in OSE that I've been seeing for the last couple of months so it appears it's not fixed yet, but is there any indication of when it might be?

[DmitriiK]

The same story here. Quite honestly, I think that last index was very much messed up, probably due to broken index before that. So, I gave up on it and simply waiting for next scheduled one - on 11/17.

[Joe.Robison]

Gotcha - makes a lot more sense now. Moz's DA/PA is clearly the gold standard that most in the industry rely on, but I didn't realize the extent of the processing required to make that happen. Even more props to Moz for the BHAG of taking on such a complex task all these years.

[randfish]

I hope we might actually have that 11/17 index out a little bit early. We've made a lot of fixes and optimizations, and, fingers crossed, it looks (so far) like it's making a difference in terms of speed to index processing completion.