Link analysis going Crazy. Next Linkscape update. Multiple Problems
-
Hey gang - there's a thread going around the SEOmoz engineering + help teams on this topic today. We're researching what happened right now, and Kate Matsudaira, our VP Engineering, has promised to leave a reply once she's got the full story. We'll try to be as transparent as possible here and as fast as we can as well, but Linkscape investigation can take some time due to the massive complexity of the system.
Thanks for posting responses and please do keep suggesting sites we're missing, pages we might not have crawled but should have, large drops in metrics (particularly if/when they're outliers to the rest of your competitors/other sites in your sphere).
Thanks much!
Randp.s. Normally, I'd be much more involved in this myself, but today's my anniversary with my wife and we're on vacation in Southern Oregon. Don't worry, though, I only take one each year, so typically I'm better able to respond fast.
-
Hi everyone!
I just wanted to add a quick response to shed a bit more light on the situation.
Last year we started a on a project to drastically improve our index. The first part of that was to make our crawler discover more of the web - this included crawling deeper on domains, discovering more links faster (freshness), and contain more links overall.
Background
To understand the changes, it might help if I explain how our crawler used to work and how we changed.
Our crawler used to crawl the web (for 3-4 weeks), then we would compute the link graph and create all the lists of links, and metrics you see in Open Site Explorer - this is what we called processing (and it would take 2-3 weeks). As part of processing we would select the top 10 billion urls to crawl, and then start crawling those.
The problem with this system was that the data was could be 7-8 weeks old (crawling time + processing + deployment to the API and OSE). It also wasn't recursive - meaning that we would only discover new links when we did the processing of that crawl, so it could take us several months before we would see new links that were deeper in domains.
The changes
We modified our crawler so we were crawling all the time - we crawl sites every day, or week, or month - based on authority. As we crawl those site, any new links that we find are added to one of the buckets, and will be crawled typically within that same index. This is exciting because we can go deeper, discover more links, and produce a higher quality index. The other benefit, is that since we are crawling all the time, we can just take a snapshot of that crawl and run processing - without waiting for the last round of processing to finish - and this means we can update the index more often.
However, in June, we had a problem with the old crawlers, and we had to roll out our new version of the crawl and index with the OSE launch on July 27th. So even though our testing looked good when we released the new index, and correlations were higher than the old crawl, we got complaints about things that were wrong.
The issues
Binary files were in the index - There are normally only supposed to be links in the index, but because the new crawler went very deep on some domains we started discovering all sorts of binary files, which when parsed, produced lots of weird links. So domains had all these links from sites that didn't link to them. We fixed this issue, and this is the first index with the fix.
We went too deep on big domains - There are a lot of knobs to turn on the new crawlers - from the number of sites we crawl daily/weekly/month to how many links we keep for different domains. One of the first things we noticed with this new crawl, was that we had less domains in our index. So we dialed down how many urls could come from a domain - and this new index also contains that change.
What we are doing
We recognize that all of you depend on this data. And we take the index quality very seriously.
We have already made a lot of other changes, increasing the overall size and adjusting how we crawl. However, since it still takes 2-4 weeks to process an index, so some of those changes won't be seen for another 2-4 weeks yet.
We are also working on an updated, higher correlating Page Authority/Domain Authority that should be out in a month or two - but also may jump around a bit.
What you can do
Definitely keep sending us feedback. It really helps us understand where we may have missed in our testing, and what we can do to fix it.
And thanks again for your patience - we really want to deliver the best possible Linkscape for you, and I assure the team is working nights and weekends to address these concerns.
And if anyone has questions you can always email me or our help team (which tend to respond to emails much faster), as all of us care a lot and really want to hear your feedback.
Thanks again,
Kate -
Hey Rand, Thanks for the response. I think we all appreciate the complexities involved. You go and have yourself a good anniversary. Cheers, Brendan.
-
And btw, we are still investigating the differences between indexes and will continue to update this thread as we have more information.
Thanks!
-
Happy Anniversary, what are you doing replying to posts!?
I'm sure the team will work it out.
-
Thanks for the detailed info! It'd be great to receive updates like this.
-
Thanks you guys for the follow up and explanation but as some of the PRO member have already mentioned in this thread, most of us are not running large sites and using OSE for large sites. Seeing such huge changes without a heads up, is something that has taken many of us by surprise. There is not one single metrics in the latest updates that is not way off compared to the previous updates for our site. Some of my competitors have seen many changes but some have not? What should i then trust?
Let me be honest with you guys the responses you have provided so far have not been satisfactory.
I know you guys are working hard at this but i can't stress enough how urgent you need to come up with an "official" point of view moving forward.
Thanks,
Olivier
-
Thanks Gyorgy - I am glad you found it useful.
For what it is worth, we have another index update planned in 2-3 weeks, and then another 3 weeks out - each index should get progressively better.
The team is working over time here though - the hard part is that the changes we make now can take 2+ months to propagate.
All the domains people sent us yesterday helped us identify another bug with our index, so we have a fix for that too. But since it takes 3-5 weeks to crawl, and then another 2-3 weeks to process you won't be able to see those improvements for another 2+ months. However, by December, the index will be better than it has ever been - with more domains and links.
Thanks again for your patience and all the details - it has really helps us track down issues.
-
Rand,
Thanks for keeping us up to date with your latest post, Linkscape September Update in the SEOmoz Blog.
Sha
-
Wow! You caught that fast
Thanks Sha - glad we can keep in close touch with everyone around this issue. -
LOL...hanging on your every tweet!
but seriously though... it's much appreciated.
-
A simple asterisk next to the new results linking to a similar explanation as above would have made the difference for many members. The staff's view of the situation in-house is more nuanced, of course, but you guys aren't seeing the forest for the trees: As the Linkscape system becomes more refined, more and more of the userbase will come to regard seomoz's stats as the gospel truth.
So? Give users the statistical qualifications they'll need when changes occur; provide clear, up-front documentation alongside all significant algorithmic changes, and consider the plight of those SEOs whose expectations of your service are very simple: to see their numbers go up, relative to the work they've put in. You just can't remain blind to that. Kate's response here is a good step, but still incomplete as Oliver notes above.
SEOmoz is a killer service, one I'd invest in if the opportunity arose, but this lack of support hints at a company culture out of touch with the tenets of their own business model's success. So, stay mindful of the little guys. Their audience you'll need to satisfy in the long run to growth & profitability.
-
Hi Olivier - sorry the responses to date haven't been to your satisfaction. Unfortunately, there's not a ton I can do to provide more detail/color. Metrics are going to change every index, sometimes substantially when we change what we're crawling or how we calculate metrics. The relative comparison between self/competitors/broader field should remain fairly useful/usable as a way to think about data between indices, but I can't argue the point that right now, comparing/thinking historically about Linkscape metrics is rarely useful.
We do hope that as we get more stable, make greater improvements and achieve wider/larger indices, the historical comparisons will become more useful and usable, but that's still many months off.
-
Everything i have read about this issue has never come close to answering one of the most fundamental question for a small business owner like me:
Some of my competitors have seen many changes but some have not? What should i then trust?
The answers being given that all should be set in 2/3 months is not acceptable & just as an FYI there has been no changes in any other link metrics from google webmaster tools etc.
Finally i will conclude by saying that I run a site that is 7 months old so the idea that I have old links and that the value of my links today are lower than 7 months ago make me crazy. In a launch phase like mine, all these metrics are key and i still remain skeptical that maybe SEOmoz is not for me the little guy but for older larger site that have a lot more history.
Bottom line all this is disappointing.
Olivier
-
I don't think we have a satisfactory answer for you or for ourselves. So many things can affect when/whether/how metrics change that I'd be lying if I gave a single answer. It could be:
-
Some sites earn more links on pages we do/don't crawl
-
Some sites earn links from places that our metrics count less/more
-
Some sites lose existing links
Which particular competitors have precisely which issues happening and what's a result of changes in PA/DA calculation or mozRank/mozTrust scaling is something we can't say today.
We could do this if we only crawled the same pages every index, but we know from experience this produces terrible results (as we both miss new pages and as in 1 month, ~15-20% of what we crawl decays with 404s/500s/30xs/etc). For now, our plan of attack is improving size and continuing to tweak the algorithms to get closer and more accurate correlations with Google's rankings.
I'm sorry this is so frustrating - it's a massively complex problem and we're doing the best we can, and working as hard as we can on improving. I don't think everything will be miraculously fixed in 3 months or 9 months or 24 months, but it will keep getting better, and we do believe we can make link research, link analysis and competitive comparison functionality very good over time. Even today, I think we've got the best product on the market for many of the functions here.
-
-
Rand
The one thing that I always appreciated is that you always take the time to explain and give your point of view.
Especially as the head of my company and as an entrepreneur myself i can only thank you for taking the time and energy to do that, it must be recognized and is a big part if the success of your company. As I said before despite this OSE/Linkscape situation, I still like you guys (my co worker offered me a moz sweater for holidays bc I always refer to your metrics/tool
& they have little to no SEO knowledge).BTW i will take this opportunity to ask (rather than request) that you provide some SEO blog specifically in the "consumer health information online space" which is dominated by 3 to 5 key players (which for most of them use very little or have simply ZERO social integration due to the highly regulated market place) yet still enjoy top rankings while guys like us are doing the best to change things. I know it takes time but some SEO perspective on this would be interesting
Thanks again
Olivier
-
Thanks Olivier - if you shoot me an email with 3-5 search queries you're hoping to analyze, I think it would be fun to do an analysis/plan-of-attack blog post in that space.
-
Thanks for responding and I will email you and take you on your offer.
-
Thanks for responding and I will email you and take you on your offer.
-
Hi Oliver,
We use OSE for benchmarking and for report too; however, we also have a link building spreadsheet for every client. In case something happens with OSE or any other third party tool, we still have a reliable source of information and proof for the clients, that we're continuously working.
When I get a new link, I record the page, the domain, PA and DA values, etc., so later if OSE changes, I can still show the clients the number of links we collected and the value of the websites.
Always have a backup solution, don't rely on OSE only. If OSE fails, it's inconvenient, but not tragic.
-
03107
-
04131
-
04393
-
03288
-
05318
-
13479
-
0101.4k