Welcome to the Q&A Forum

Someone from Moz got back to me and said the best option is to add a directive to robots.txt to tell Roger to slow down the crawl.

Has anyone ever tried to stop rogerbot from crawling their site? My tech team thinks that Rogerbot is crashing our site and causing a 504 error. Does anyone have experience with this or trying to stop the crawl? Thanks!

Has anyone else had issues with Roger crawling your site in the last few weeks? It shows only 2 pages crawled. I was able to crawl the site using Screaming Frog with no problem and we are not specifically blocking Roger via robots.txt or any other method. Has anyone encountered this issue? Any suggestions?

I checked traffic trends in Adwords and Google Trends, and there was some decrease around spring break, but it wasn't sustained and we didn't see a similar dip last year. Traffic is starting to come back, but still trailing last month significantly. We did add some language to our title tags that targets more of a niche/qualified audience on some of our pages. I haven't seen a decline in rankings or avg rank in GWT, but now I'm wondering if this is somehow related. Anyone have this type of experience before?

My site is experiencing a decrease in organic traffic WOW for the last two weeks and for the first time all year is showing a decrease compared to last year's traffic for the same weeks. At first I thought this was a seasonal pattern due to spring break (we are mostly b to b), but the dip has sustained for another week. The only changes made during this time period were a few on-page updates and some title tag updates to a specific group of pages. However, the decrease is sitewide including branded clicks and impressions. I haven't noticed any changes in rankings. Impressions and clicks are down per Search Console, but CTR and Avg Rank haven't changed. Is it possible that we've been penalized or hit by an algo shift? What's the best way to know for sure?

VLGLUTt