902 Error and Page Size Limit

[Paul_FL]

Hello,

I am getting a 902 error when attempting to crawl one of my websites that was recently upgraded to a modern platform to be mobile friendly, https, etc.

After doing some research it appears this is related to the page size. On Moz's 902 error description it states: "Pages larger than 2MB will not be crawled. For best practices, keep your page sizes to be 75k or less."

It appears all pages on my site are over 2MB because Rogbot is no longer doing any crawling and not reporting issues besides the 902. This is terrible for us because we purchased MOZ to track and crawl this site specifically.

There are many articles which show the average page size on the web is well over 2MB now: http://www.wired.com/2016/04/average-webpage-now-size-original-doom/

Due to that I would imagine other users have come up against this as well and I'm wondering how they handled it. I hope Moz is planning to increase the size limit on Rogbot as it seems we are on a course towards sites becoming larger and larger.

Any insight or help is much appreciated!

[tawnycase]

Hi there! Tawny from Moz's Help Team here.

I'm so sorry that you're running into persistent 902 errors from your site crawl! Unfortunately, this is a technical limitation of our tools. We're in the process of building a bigger, badder, better crawler, but it isn't quite ready yet.

You're certainly not the first person to write in about this issue, so I'll be sure to take your feedback to our product team! The ability to crawl larger pages is going to be more and more important as time moves forward.

Thanks for writing in to let us know you were having issues! If you have follow-up questions or need any other help with the tools, feel free to write in to help@moz.com and we'll get things sorted out for you.