Welcome to the Q&A Forum

Hey Lawrence,

The changes made to the PA model do not treat 301 redirects any different - we haven't made any updates to how we handle redirects. However, if you've seen a drop in your links, we probably didn't crawl the pages redirecting to you. This would definitely affect your PA score.

I hope this helps, but let me know if you have more questions! If you want to find out more information on the new PA model, Matt Peters, the engineer who worked on the new model, has a great blog post here:

http://www.seomoz.org/blog/introducing-seomoz-updated-page-authority-and-domain-authority

Thanks,

Carin

Hey Berend van Bon,

The Back Burner has an interesting thought on the decrease in your DA. It's tough to say for sure why your DA has dropped - it could be due to the updated DA model or we might not have crawled as many sites linking to you. If the latter is the case, you would see a decrease in link count, thereby lowering your DA.

Most likely if you see a drop in your linking domains and/or URL counts, this was what caused your DA to decrease. If these metrics look about the same as last index, the drop would probably be due to the updated DA model.

I hope that helps, but let me know if you have any other questions!

Thanks,

Carin

Hey Zack,

Sorry to hear you're still having problems - we've seen an improvement on most sites at this point. Would you want to send me info on the site you're searching and any filters you are using?

If you don't feel comfortable posting that info on this thread, feel free to email me directly: carin@seomoz.org.

Thanks!

Carin

Thanks Ryan for the great answer! We do have the new social features in Open SIte Explorer that display the Facebook shares, collected from their FQL API.

We are also in development of a new tool in the PRO app offering Social Analytics metrics. Here is Rand's blog post about it!

Hope that helps, but let me know if you have any more questions!

Thanks,

Carin

For sure! I figured it's a question that has crossed people's minds before (I know it has for me!) so if you see anyone else wondering, I wanted you to have something to point them to

Thanks!

Carin

Hey! I just saw this post - not sure if you filed a ticket the Customer Service team, but I wanted to see if I can help explain what's going on.

I wanted to make sure I had a full handle of what was going on so I talked with one of the Open Site Explorer developers. So my understanding is it will automatically redirect when Linkscape has identified it as a redirect, but if Open Site Explorer requests data from Linkscape and is returned a 301, then the message alert will show up and Open Site Explorer will show the redirected URL metrics.

Two main reasons this could happen - the redirect wasn't in place when we crawled the page so it wasn't recorded as a redirect. The other reason could be when Open Site Explorer requests Linkscape metrics for a given URL, it does not explicitly say if the URL is a redirect - it only tells us if the searched form is canonical. If the search form does not match the canonical, we assume it redirects to the canonical.

I hope that helps explain a bit, but let me know if you have more questions! The customer service team is awesome and they will be able to help you at as well!

Thanks,

Carin

Hey PSV,

Sorry I'm jumping in so late to the game on this! The response from the Customer Service team by Kenny below is correct - with the adjustments we've made to the new crawler, we're going less deep this crawl than we did in the July crawl. This is because we're still trying to make tweaks to get the domain diversity back up to what is what before. It looks like you had a major change from July to August which is strange - I haven't seen such a drastic change pointed out by any other users.

If I could get the actual domain, I can look into this for you. Is this discrepancy being reported by Open Site Explorer or are you querying the API directly?

It's hard to speculate without knowing which domain you're referring to, but you may see a decrease in inbound links if you had a lot of links from a high MozRank page. If you had a large amount of links on one domain and we didn't crawl that domain as deeply this past index where we went MUCH deeper the index before, many of the links deeper down in the domain won't be in this latest index.

Please let me know which domain you're referring to and I can try to track down some information for you!

Thanks,

Cairn

Hey guys,

Yep, Keri is correct, unfortunately We found a bug in ourJuly index with our new crawlers - they were crawling binary files as if they were links and, since they are not normal links, the crawler couldn't handle them very well.

We have made some updates to our crawling so it will go deeper into sites. The reason for these odd inbound links from high-authority sites is due to the crawler reaching much deeper into sites where there are more download (i.e. binary) links. The first issue is the crawler is counting a binary file as a link, but the larger issue, is that the crawler doesn’t really know how to handle these types of files. This bug is causing some links to be improperly associated with certain domains. This is why you’re seeing inbound links to pages that don’t really exist.

There are two steps to addressing this issue: changing how the crawler sees these file types and then fixing how the crawler handles these file types. We have made improvements to our algorithm so that we will be able to handle the majority of these files correctly, however, this update will need a few more weeks to propagate. The fix for this issue probably won’t be seen for another update, meaning late September. Our improvements should catch most of the issues, but there still could be a few cases we haven't addressed. If this happens, don't hesitate to let us know; we love feedback since it helps us improve and make our index even better!

The next step is to fix how our crawlers handle binary file links and prevent them from being improperly associated with certain domains. We are in the process of working through that issue right now. We’re doing everything we can to resolve this bug as we know it is alarming to see these phantom inbound links.Thanks for your patience!Carin

Hey! Same message to you If you want me to look into your account, feel free to email me at carin@seomoz.org!

Just provide some info on what URL you are looking for, what tab you're requesting from and if you have any filters applied and we'll look into why these certain ones are having issues.

Thanks for your patience!!!

Carin

Hey! I need some more info from you and I can look into it - feel free to email me at carin@seomoz.org.

You're not being difficult - I understand the frustration

If you can provide your account information in an email to me, as well as any information regarding which tab you're requesting from and what URL you are querying for and if you were applying any filters.

Thanks!

Carin

Hey! Can you give me some more information so I can dig into this? Looks like the last batch of 100 reports is crunching through now, but I'd still like to find out why you're not getting yours.

Can you send me some info on the request like: the site searched, any filters used and from which tab.

Thanks!

Carin

Hey guys!

I wanted to jump in here and give you all the lastest on the CSV download issue. We were finally able to clear through the backlog of CSV reports about 7 pm PST last night, however, there were about 3,000 jobs that were still in a finalizing status and just hanging. We were able to work a quick fix to get these last remaining reports out today. The fix was such a great idea, we've decided to make it a permanent feature in Open Site Explorer!

Since the new launch of OSE, we've had reports of users requesting reports that end up hanging during high peak times. The fix we added today will help in those scenarios by re-queueing the report if it is hanging for a long period of time.

I'm hoping this helped get the last of the missing reports out, but please let me know if you guys are still seeing pending or hanging requests.

Thanks!

Carin

Hey John,We have made some updates to our crawling so it will go deeper into sites, and this is the first launch including the new metrics. We've discovered a bug, however, in the updated crawler. The first issue is the crawler is counting a binary file as a link, but the larger issue, is that the crawler doesn’t really know how to handle these types of files. This bug is causing some links to be improperly associated with certain domains. This is probably why you’re seeing your competitor have these .edu links - they're probably incorrectly associated with their site.
There are two steps to addressing this issue: changing how the crawler sees these file types and then fixing how the crawler handles these file types. We have made improvements to our algorithm so that we will be able to handle the majority of these files correctly, however, this update will need about a month to propagate. The fix for this issue probably won’t be seen for two more updates, meaning late September. Our improvements should catch most of the issues, but there still could be a few cases we haven't addressed. If this happens, don't hesitate to let us know; we love feedback since it helps us improve and make our index even better!

The next step is to fix how our crawlers handle binary file links and prevent them from being improperly associated with certain domains. We are in the process of working through that issue right now. We’re doing everything we can to resolve this bug as we know it is alarming to see these “questionable” links associated to either your site or your competitors sites.

I hope this helps answer your questions around these .edu links, but let me know if you have any more questions!

Thanks,
Carin

Hey Zack,

Thanks so much for understanding! We are doing everything we can to get the bug resolved. Binary files are the downloadable files you see as links - .pdf, .exe, .img, etc.

I'm really sorry, but we don't have a URL to the old OSE. I saw Steven's response as a workaround - is that possible or are there too many file types to filter out?

Our crawlers that provide the metrics to OSE are always crawling, but will take about a month for our fix to propagate through to all the pages we crawl. Once we have removed these links from our crawlers, then we'll have to process the metrics. This is why it's looking like late September for the fix to show up.

I really appreciate your patience and understanding, we're doing everything we can to fix it!!

Thanks,

Carin

Hey Zack, I saw the ticket you filed was answered by Aaron, but I just wanted to follow up with you as well. We have made some really exciting changes to the crawler, but, unfortunately, there is a pretty obvious bug as well...

The reason for the “questionable” links coming from the Internet Wild West is due to the crawler reaching much deeper into sites where there are more download (i.e. binary) links. The first issue is the crawler is counting a binary file as a link, but the larger issue, is that the crawler doesn’t really know how to handle these types of files. This bug is causing some links to be improperly associated with certain domains. This is probably what you're seeing with all the crazy links from China and Russia which don't actually link to the site you're researching.

There are two steps to addressing this issue: changing how the crawler sees these file types and then fixing how the crawler handles these file types. We have made improvements to our algorithm so that we will be handle the majority of these files correctly, however, this update will need about a month to propagate. The fix for this issue probably won’t be seen for two more updates, meaning late September. Our improvements should catch most of the issues, but there still could be a few cases we haven't addressed. If this happens, don't hesitate to let us know; we love feedback since it helps us improve and make our index even better!

The next step is to fix how our crawlers handle binary file links and prevent them from being improperly associated with certain domains. We are in the process of working through that issue right now. We’re doing everything we can to resolve this bug as we know it is alarming to see these “questionable” links associated with your sites.I hope this helps and thanks so much for being patient :)Thanks,Carin

Hey guys,

The issue you are seeing is due to the new OSE update. We have done some updating with our crawler and this index represents the newest version - sadly, with a few bugs...We are looking into this issue and hope to have it resolved as soon as possible!

The newest version of our crawler is built to be fresher, but it is also going much deeper into high MozRank pages. This bug has probably always existed, but has never been obvious since we weren't crawling as deep into domains where there are more download links. We are currently looking into fixing this so these won't be counted as inbound links.

I'm so sorry for the inconvenience - once we get this new version of the crawler dialed and smoothed out, it will be providing you guys a much fresher and higher quality index!

There is another thread regarding this topic, so check it out if you want more information on what is going on with this index.

Thanks,

Carin

Hey guys!

Keri is right - we have done some updating with our crawler and this index represents the newest version - unfortunately with a few hiccups. People seem to be seeing two issues with this new index - link counts and domain authorities are going up or down considerably and there is an increase of "questionable" inbound links.

Both issues are due to the same root cause: our new crawler is built to be fresher, but it is going deeper into domains, and, unfortunately not visiting as many domains. Domains with a high MozRank are getting crawled deeper, but domains with middle to lower MozRanks are not getting crawled.

Our top priority now is to get the domain diversity back up to or better than that of our last update as was originally designed. It's fixable and we will be focusing all efforts on this.

Previous crawling worked by selecting a list of the top MozRank URLs (around 10B) and then crawling one page from each of them. Now we are crawling links as we discover them, and crawling high MozRank sites daily, weekly or monthly. The advantage of the new crawlers is we are crawling all the time and so we will have fresher data. As links are added, we are much more likely to discover these deeper links. The new crawl had 59B urls, a lot more than the previous 42B, however, more of these links are from the same domain.

The reason for the "questionable" links is due to the fact that the crawler is reaching deeper into the domains where there are more download links. We are currently looking into fixing this so these won't be counted as links. We'll let you know as soon as that issue is resolved!

We are really sorry for the inconvenience. Once we have this new crawler dialed it will provide much fresher and higher quality data!!

Thanks,

Carin