Problems with OSE downloads

[randfish]

Hey gang - sorry for my delay getting in here. We had a crazy day.

Here's what happened and why OSE report downloads have been slow:

• Last week, we had some volume spikes. The new OSE V3 offers a lot more download functionality, and everyone's been using that functionality a lot (awesome, but also a big load)
• We had some technical operational issues that slowed things up on OSE for about 24 hours, creating a huge backlog of reports
• We tried to simultaneously scale the capacity and segment the downloaded requests so they wouldn't interfere with load on the rest of the platform
• Today, we finally cleared the queue fully and reports should be back to fairly speedy

I'm really sorry about this. The tech behind scaling is challenging and our engineering team worked their butts off, but I know there's no excuse when you need a report or to do some work. We're going to work very hard over the next few weeks to try and make sure we get better with uptime/responsiveness/speed/etc.

Thanks for the patience - and keep your eyes out for more improvements/upgrades in data quality, size and speed coming in the next few months.

Cheers,

Rand

[blocker0408]

Thanks Rand, I am sure I speak for the others with this problem when I say many thanks for personally looking at this. As always, excellent service.

Just to be clear, does this mean that the pending reports clogging up my downloads will be cancelled or automatically generated ? Not received any of them yet.

Warm regards

[randfish]

If you're not getting them, you should probably try re-filing. Reports should be taking <6 hours at the very most. We were seeing many that were <1 hour.

I'll ask the team to hop in this thread, too, in case I've gotten any details conveyed incorrectly.

[blocker0408]

Just checked, some have completed, others still pending that I ordered about 48 hours ago.

What time is it there ? Do you sleep ? you're crazier than me !

Regards

[Adwise_Internetmarketing]

Thanks for the response. The issues are worth the improvements. Looking at it long term, a lot has become better Some reports are in, others still pending. I can wait a couple days. Thanks again.

[randfish]

1:46am - I'll sleep when OSE is doing its job for you!

[carinoverturf]

Hey guys!

I wanted to jump in here and give you all the lastest on the CSV download issue. We were finally able to clear through the backlog of CSV reports about 7 pm PST last night, however, there were about 3,000 jobs that were still in a finalizing status and just hanging. We were able to work a quick fix to get these last remaining reports out today. The fix was such a great idea, we've decided to make it a permanent feature in Open Site Explorer!

Since the new launch of OSE, we've had reports of users requesting reports that end up hanging during high peak times. The fix we added today will help in those scenarios by re-queueing the report if it is hanging for a long period of time.

I'm hoping this helped get the last of the missing reports out, but please let me know if you guys are still seeing pending or hanging requests.

Thanks!

Carin

[blocker0408]

Just checked, still no change with the pending reports at my end.

Thanks

[carinoverturf]

Hey! Can you give me some more information so I can dig into this? Looks like the last batch of 100 reports is crunching through now, but I'd still like to find out why you're not getting yours.

Can you send me some info on the request like: the site searched, any filters used and from which tab.

Thanks!

Carin

[Adwise_Internetmarketing]

I can confirm all except one which were on finalizing are done now. Thanks! the other has the same issue now as addressed below.

There are still like 11 reports starting from scratch every time I visit the csv export page without coming further than around 30%. Others with exactly the same issue are finished though. So it's a bit random why it works for the one and not the other.

I'm not doing anything related to link building today so I've got all patience to wait till it's resolved

[blocker0408]

Hi Carin

Not being difficult but pushed for time, can you not simply access my account and have a look yourself ?

Thanks & Regards

[blocker0408]

In exactly the same situation, may be something to do with requesting the same report a few times because it didn't come through first time ? Looks like it's close to being fixed though by the Moz team.

[Adwise_Internetmarketing]

I'm quite sure I only requested most of the ones pending ones, not a second time. Only a view of them to try out. Like they said they are working on it.

[carinoverturf]

Hey! I need some more info from you and I can look into it - feel free to email me at carin@seomoz.org.

You're not being difficult - I understand the frustration

If you can provide your account information in an email to me, as well as any information regarding which tab you're requesting from and what URL you are querying for and if you were applying any filters.

Thanks!

Carin

[carinoverturf]

Hey! Same message to you If you want me to look into your account, feel free to email me at carin@seomoz.org!

Just provide some info on what URL you are looking for, what tab you're requesting from and if you have any filters applied and we'll look into why these certain ones are having issues.

Thanks for your patience!!!

Carin

[ColumK]

I am still getting problems downloading reports. They have been going for 5 hours now. Has the log increased again? Could you not use some of that $18m to buy a few more servers?

[carinoverturf]

Hey there!

Haha - we ran into a problem on Monday night with one of the machines falling over causing a huge backlog to pile up. We were able to get things back on track yesterday and churn through the backed up reports, but with the index launch yesterday, we're seeing a bit of a backlog again this morning.

We are getting a monster machine up right now to speed through this! Once things calm down you should see these come through. It looks like you have about 8 pending - I'll keep an eye on them to make sure they go through!

Thanks,
Carin

[ColumK]

Thanks Carin! Appreciate it

[carinoverturf]

No problem! I'm so sorry for the inconvenience!

I just pushed the remaining pending reports through, so I think you should be set, but if you continue to run into any problems, just let me know!