Mozscape Index update frequency problems?

[kevin.kembel]

I'm new to Moz, only a member for a couple months now.  But I already rely heavily on the mozscape index data for link building, as I'm sure many people do.  I've been waiting for the latest update (due today after delay), but am not seeing any mention of the data yet - does it normally get added later in the day?

I'm not that impatient that I can't wait until later today or tomorrow for this index update, but what I am curious about is whether Moz is struggling to keep up, and if updates will continue to get more and more rare?  For example, in 2013 I count 28 index updates.  In 2014 that number dropped to 14 updates (50% drop).  In 2015, there was only 8 (another 43% drop), and so far this year (until the March 2nd update is posted) there has only been 1.

This isn't just a complaint about updates, I'm hoping to get input from some of the more experienced Moz customers to better understand (with the exception of the catastrophic drive failure) the challenges that Moz is facing and what the future may hold for update frequency.

[randfish]

Hi Kevin - the index update should be live as of right now (probably only a few hours after you posted this message). We aim to have one index update per month, so 12 per year. We had a catastrophic failure on our early January index, so it had to be abandoned (noted here: https://moz.com/products/api/updates), but the team has been working hard to fix issues and prevent others from arising. Unfortunately, it's often the case that we encounter new/unexpected/never-before-seen issues that need to be addressed. Frustrating, but unavoidable as best we can tell. Obviously, we will continue to do our best to get these indices out on time.

As far as the future goes, it's hard to say. We have a bigger team now than we last year -- 4 folks work full time on the Mozscape index and 4 are working on the next-generation version of the index (which will update in near-real-time), and we certainly have much better monitoring and operational structures in place. But, as I noted above, it seems that the errors/issues we face are always new and unique - things we've never seen before in the 100+ index runs we've had over the last 8 years. I can tell you we're building processes to identify problems before they happen, and that we're better staffed, and that we have engineers on-call 24/7 to fix issues if they crop up, but processing full-graph metrics on a full-scale web index whose shape and composition can vary wildly means there's still uncertainty and probably always will be. Our job is to keep reducing that uncertainty and finding optimizations, while we rebuild the full system in the background to eventually replace the old, batch-processing system that's at the core of so many of our challenges.

Hope that helps.

[kevin.kembel]

Hi Rand,

I appreciate the official response, and I enjoyed reviewing the new data shortly after my post.

It's especially great to hear about the dedicated team working on the next generation of analytics gathering, I think that will go a long way to silencing critics of the current index processing system (like me).  I can't wait, are you able to give us any kind of timeline for that migration?

[randfish]

I wish I had an answer to that If I had to guess, I'd say sometime in the next 2 years, but not anytime in the next 9 months. We've stopped asking the team for a final delivery date because it's just too hard to estimate all the work required, and past estimations have been so far off target. Instead, we just try to estimate the next quarter worth of sprints and then measure how we perform against those.

It turns out, replicating a processing system like Google runs without billions in revenue is really hard