Welcome to the Q&A Forum

Hey Garrett!

While the robots.txt file does load fine in a browser, our crawler is receiving a timeout when trying to access it.  I tried to curl from our user-agent and received the same issue.

https://www.screencast.com/t/PDXVg4yf5Y

You may want to reach out to your hosting admin to see if they have changed a setting or added any server level bot security.  You may also request that they whitelist our user-agent "rogerbot".  If you need further assistance on this, you can always shoot us an email to help@moz.com for further assistance.

Hey Kirk!  We built our crawler to obey robots.txt crawl-delay directives. In the future, if this is ever an issue, you can use the crawl delay to slow Rogerbot down to a more reasonable speed. However, we don't recommend adding a crawl delay larger than 10 or Rogerbot might not be able to finish the crawl of your site.

Just add a crawl delay directive to your robots.txt file like this:

User-agent: rogerbot
Crawl-delay: 10

Here's a good article that explains more about this technique: https://moz.com/learn/seo/robotstxt.  I hope this helps, feel free to reach out if you have any other questions!

Hey Steve!

Those are older deprecated responses, they will return a value of '0'.  We left them available for calling so they don't break anyones existing script but they are not valid metrics.  You can see the full list of supported URL metrics here:

https://moz.com/help/links-api/making-calls/url-metrics

Hope that helps, let us know if you have further questions.

Hey!  Dave from the Help Team here,

we are currently running some tests internally to try and determine what is the optimal length now for Meta Descriptions.  Once we have some conclusive results we will update our tools.  We should have a blog post coming out soon about it, so stay tuned!

Hey Roman!

We have seen instances where capitalization of keywords can return different serp results, so we do collect rankings based off if a keyword is capitalized or not.  Hope that helps to clarify, let me know if you need anything else.

Hey, Dave here from the Help Team.

All URL metrics can be found here: https://moz.com/help/links-api/making-calls/url-metrics

There seems like a lot of variations but that is because you can call data on a root domain, subdomain and page level, so there are varying factors to that need to be determined before we can answer the above questions.

Do you want total backlinks to a specific page?  root domain?

The description column of that page is pretty good at specifying what each bit flag is calling, feel free to reach out to help@moz.com if you are stuck on something.

thanks!

Hey!  Dave here from the Help Team,

There are a couple different things you can do to mark items that you have done.  One of the new features we have implemented into Site Crawl is the ability to mark items as "Fixed".  This tool can be handy if you know that you have fixed issues with your site but are still waiting for your next update.  Another trick you might want to do is download your "all crawled pages" CSV and then create a "notes" column.  It wont live in the Moz dashboard but at least you would have a good record!  Hopefully those options help you out!

Hey Tamar!  Dave here from the support team.

Please keep in mind that the Beta Crawler is still in the Beta phase.  We have seen some instances of the new crawler not finding H1 tags, and we are working on getting that fixed ASAP so please bare with us and thanks for your patience!

-Dave

Hey, Dave here from the help team!

On Page Grader is uses DNS checks to validate and it looks like all of your nameservers are not valid.  You may want to start looking there.  I also responded to your help ticket, if you have any follow-up questions, feel free to hit me back there!

https://www.screencast.com/t/jmKQ5BFuN8

I also responded to your help ticket, if you have any follow-up questions, feel free to hit me back there!

Hey, Dave here from the Help Team!

This is really up to you and how you want to organize your campaigns, however I think you are right about not overly complicating it.  I would suggest that you track your entire domain in one campaign and categorize your keywords with labels.  If you decide to later track specific sections of you site, you can, but its probably good to have one campaign with an overall view of your entire site.  Hopefully that helps!

Hi Aimee, Dave here from the help team

I took a look at your account and I see a campaign called JHM that only has two pages crawled, so I am assuming that is the site in question.  The two pages are the domain and then the redirect to the WWW version.  When I load the last page we crawled and then look at the source code, I noticed that the site is entirely javascript and does not contain any other HTML links.  Since our crawler follows HTML link structure, we are hitting this page and then stopping when we don't find anything else to crawl.  Hopefully that helps to clarify, let me know if you have any other questions about this or feel free to reach out to us at help@moz.com.

Hey Maureen, thanks for reaching out!

So I tried to curl your new domain (http://ryemeadgroup.co.uk) against our link index user-agent and it appears we are blocked from that site.  I get a 403 forbidden when trying to access it so we would not be able to index it without being unblocked.  Typically this is resolved by your hosting admin, they should be able to whitelist our user-agent "dotbot".  I hope that helps to point you in the right direction, if you need further technical assistance, please reach out to help@moz.com!

Hey Joel!

We switched to Bing Custom Search

Hey Jason, sorry for the inconvenience!

Rank tracker was never really intended for daily keyword rankings, which is the reason you had to do the tedious manually updating and even then there was some caching involved.  I will certainly pass along your frustration with our product team.  Let us know if you we can do anything to help make this transition easier!

hey, Dave from the Help Team here.

The 430 error seems to be a result of shopify blocking our bot from accessing those pages temporarily. We have seen instances where this clears up after the second crawl, so keep your eye out for your weekly campaign update email in the meantime.

The good news is, that your human visitors will still be able to access your pages to do their shopping, phew!

Thanks so much for letting us know. We'll track this issue and look into a fix. I'm sorry I don't have better news for you at this time.

Hey!  Dave here from the Help Team.

It looks like we are getting a 503 status code when trying to access your robots.txt file.  You will want to check your server logs for the response from User-agent: rogerbot/1.2 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+aardwolf@moz.com) and make sure you are not blocking us from accessing your site!  Hope that helps

Hey, Dave from the Help Team here.

You could track the entire site with all your keywords listed and label them per variation, or you could setup separate campaigns restricted to the subfolders of the variant languages and track the multilingual keywords in each perspective campaign.  As long as the keywords are latin characters, you shouldn't have an issue.  Hopefully that helps!

Hi There, Dave here from the Moz help team!

Unfortunately at this time our crawler isn't compatible with Server Name Indication (SNI), which appears to be in use on your site. In general, SNI is a totally acceptable security configuration, but our crawler simply isn't equipped to be able to handle these settings.

https://www.screencast.com/t/lddqo8gVOt2

The good news is that our team has been working hard on a new crawler that will support this configuration and this crawler is currently in its Beta phase. If you're interested in being a part of the beta testing, the link to the sign up page is here: http://goo.gl/forms/LCvL9Ix8JDHfbAvr1. Once you fill out that form, you will automatically be added into the Beta during the next round.

We will be rolling the new crawler out to all of our users once it's out of it's Beta phase, but I don't have an exact timeframe of when that will be at this time as it's still being worked on.

In the meantime, I apologize for the inconvenience and let me know if you have any questions or need any additional clarification.

Hey Cesare, Dave here from the Help Team.

We are working on increasing better functionality around international URLs/Keywords for Keyword By Site, but as for right now its mainly U.S. focused.  Sorry for that inconvenience that causes!

Hey Lee, Dave from the Help Team here.

Site Crawl will detect any error that we report on, for any page that is accessible to the crawler.  As you develop and change your website, new crawl issues might get detected as well.  Hopefully that helps to clarify, if you need us to look into specific examples, feel free to email help@moz.com.