What's the Story on Mozscape Updates?

[DonnaDuncan]

So Rand, does the engineering collective know what needs to be done or do you need a technical architect to offer an assessment of the situation and recommendations? I don't have that skill set, but perhaps there's someone in your vast audience that does and would be willing to help out.

[randfish]

The good news is we do know what we need to do and we have the people at the architecture level to make smart decisions (IMO). However, finding folks to help us execute on this plan has been insanely hard. We've had open positions on this team for literally 2 years, so if you know anyone, send 'em our way! I'm certainly working my networks to try and help, too.

[randfish]

My guess and hope right now is that we don't have another delayed index. However, based on the inconsistencies and not-totally-clear diagnosis of issues, I'd say there's some risk associated with the next index.

Re: crawl updates - those have been behind, but should be catching up. I think the Moz Analytics team said 2-3 more days for completion of all the lagging reports.

[randfish]

Moz has a hybrid cloud we built ourselves on a datacenter with hardware we customized. We were previously using AWS, but moved to give ourselves greater control and huge cost savings.

[LesleyPaone]

I deal with a few enterprise grade clients on the server configuration end, just out of curiosity what language platform do you run on?

Also, while you are here I would like to make a feature request or inquire if it is in the pipeline.

I would like to see the index split into two subsets that are used to generate the rank / trust. I would like to see it use the GWT links and also the GA referrers. It could show a better scope and at the same time keep that data private.

[DonnaDuncan]

A suggestion.

Put out a call in Q&A and social channels. Be specific about the skill set needed. Link to job descriptions. Ask for referrals and shares. Use social channels to the do the same thing. Surely we can get a few viable candidates to step forward...

Edited for one final thought. Are you open to remote candidates?

[MattAntonino]

A giant game of Whack-a-Moz ... yeah, that's fun until it's not. I can feel the ... what is it? Distress? Worry? I know you want these things to come out on time & exactly how you want but that's rare even in the land of Google & Facebook.

You guys have great company benefits. It's amazing you find it so hard to hire the right people and as you said, have had openings for 2+ years.

Glad to hear the issues are at least unrelated. That gives us hope for the site crawls & other bugs to get fixed whatever happens to the link index.

Thanks again for the update & good luck getting through it all!

[keboydjr]

So what are the accurate numbers, last time or this time? My total links went from 15,000 down to 3,000. I really depend on the consistency and whatever changes were made, problems encountered, have really screwed up my reporting to the executive team.

[randfish]

I'll ask Martin from the Big Data team to answer the first question - I believe we use a number of different languages across the various services that power Mozscape.

On the split-index, I don't think I quite understand? Do you mean there'd be a unique version of our index that crawls the links we see from GA referrers (via the connected profiles in Moz Analytics)? That would be possible, but a ton of infrastructure work, and we'd likely need to build a unique version of the index for each customer to keep that data private.

As for GWMT - we don't have access to those, as folks don't connect/oAuth their GWMT accounts to Moz.

[randfish]

Well, there's probably only a few thousand people who've worked on big data issues of this scale and complexity, and hiring ex-search-engine employees who've already made their millions in stock is pretty hard

Thanks for the positive wishes - we'll keep working hard to try and get this to where it should be.

[randfish]

Unfortunately, our social channels and Q+A reaches the Moz audience, which is mostly marketers and very, very few software engineers (even fewer with the right big data skills). We've got a recruiting team that's going hard at this problem, though, and yes, we've just changed to accept remote candidates for technical roles, so hopefully that will help.

[DonnaDuncan]

"our social channels and Q+A reaches the Moz audience, which is mostly marketers and very, very few software engineers (even fewer with the right big data skills)"

True. But we might know some. I know some. We're all well connected.

[LesleyPaone]

Yeah basically on the split index. I think it would give people a better view of how they are actually doing without exposing their data across the internet. It also seems like it could be a way of more accurately determining the DA, PA, and trust as well. And could also add more value to the product. Right now the link index only picks up a portion of the total links and then tried to determine a ranking method off of that. But in GWT there is a totally different set of links usually.

I think if you merged the GWT data, GA data, and the Moz data together things could be very powerful.

I realize a lot of people want to see just what they can do to improve their ranking and nothing more. But I tend to look at SEO as a whole picture and just want more traffic. I could really careless if it comes from Google, Bing, or some guys blog. So I think a link section would also help too. This is what I imagine in my head. Another section in our account that scrapes the referral pages out of our GA accounts and shows us how our backlinks are performing over time. That would be nice, because part of what a lot of us do is buy links in some way or another. It would let us know at a glance if a blog we want to target again is still pulling the traffic it was before. Things of that nature. But I think a real value is putting everything under one roof. GWT and GA are totally separate, having the information from both of those platforms merged into one would be awesome for me.

[DonnaDuncan]

Hi Ken,

It's normal to see fluctuations in link counts from update to update, especially if a large number of your links are coming from relatively few domains. Take a look at this Q&A - http://moz.com/community/q/why-did-my-da-and-links-go-down. See if it helps shed light on why this happens.

[YairSpolter]

Thanks for the update, Rand, and good luck on getting Mozscape V2 up and running ASAP. I'm sure I'm not alone in wanting to see Moz emerge as the leader in this industry, bringing the tools up to par with the amazing community assets that you guys provide.

Looking forward to seeing the mustache disappear!

-Yair