Welcome to the Q&A Forum

Hello! Tawny from Moz's Help team here.

Unfortunately, it's a bit more complicated than just adding in a new location to our database.

The locales are tracked as a defined region with borders established by Google. It's hard to say exactly how Google is defining these areas, but it is likely related to ZIP codes and IP addresses.

I found this article while I was researching — it doesn't have exact answers to this question either, but had some valuable insight into Google's local search results: http://searchengineland.com/everything-need-know-pigeon-algorithm-211771. Section 7 of this resource would be the most relevant to how those locations are defined.

The locales we include in Moz Pro are specific to the Google search engine only, and our list is based on a finite postal code database, so you may not see every locale you would like to track in the list. It is helpful to be very specific when entering the locale, and it can help to include a specific neighborhood, state, or county name when entering the locale (e.g.: Melbourne, Victoria vs. Melbourne, Australia).

That means you might see more specific areas than the Dallas-Fort Worth metropolitan area, but probably won't see locales as broad as an entire county.

​If you still have questions we can help with, feel free to drop us a line at help@moz.com and we'll do our best to sort everything out.

Hey there!

Sorry to hear that a couple of your Campaigns are running into Robots.txt crawl issues! Maybe I can help figure out why those sites can't be crawled.

It looks like the robots.txt file for the first site you mentioned is returning a 403 Forbidden response to our user-agent, Rogerbot: https://www.screencast.com/t/TFFaWtikf Anything other than a 301 redirect or a 200 OK status wills top us from being able to access the robots.txt file, which will result in a crawl failure. Please make sure your robots.txt file is fully accessible to rogerbot.

For the second site, ptone.nl, we're getting a 406 Not Allowed error: https://www.screencast.com/t/np1AEJ3oDiy2 Again, we'll need to be able to access those robots.txt files in order to crawl your site.

And that last site is also returning a 403 Forbidden response when we try to access your robots.txt file for that site: https://www.screencast.com/t/CEpKpLq3z

I'd recommend checking in with your web developer or whoever manages your site for you. They'll probably know how to make the necessary changes to make sure that your robots.txt files are accessible to our crawler.

Hope this helps! If you've still got questions, feel free to drop us a line at help@moz.com and we'll do everything we can to answer them all.

Hey there! Tawny from Moz's Help Team here!

A meta refresh is caused by some meta tag elements in your site's source code. You can read up on meta refreshes over here: https://en.wikipedia.org/wiki/Meta_refresh#Examples
There are even some handy examples of what the code might look like!

It also looks like Donna Duncan has beat me to the punch here — that's excellent insight into how you might go about fixing the issue!

I hope that helps, but if you still have questions, feel free to reach out to us at help@moz.com and we'll do our best to answer them all.

Plans to upgrade the Crawl Test aren't on the road map as far as I know — our engineering resources are devoted to other areas of the tools at the moment, and the Crawl Test is pretty low priority compared to some of the sweet new things the engineers are working on at the moment. I'm afraid I don't know when or if the Crawl Test will be updated to handle SNI settings. Sorry I can't give you a solid timeframe!

I just found the ticket where we were working together on this, so no need to write in again. I'll just respond to you there.

If you write in to us at help@moz.com and ask for me, I can take your questions that way. It's much easier to work through our customer support system than through the Q&A forum, so that's the best way to go!

Thanks, Emina!

Hey there! Tawny from Moz's Help team here.

Oh boy! There's a lot of specifics in this question! It will be way easier and much quicker to get to the bottom of this issue if we can take a look at the same things you're seeing, like the specific Moz Pro Campaign(s) where you're seeing these fluctuations.

That's pretty tough to do in the Q&A forum, so if you wouldn't mind, please write in to us at help@moz.com with all your questions and we'll do everything in our power to answer them for ya.

Thanks!

Hey there! Tawny from Moz's Help Team here!

The timeframe in Open Site Explorer is a bit tricky to pin down, as it takes us about a month to gather and collect index data. You can see what date our most recent index update was on the main page of Open Site Explorer, below the search bar. The index spans roughly 180 days of data.

Hope that helps! If you still have questions, feel free to drop us a line at help@moz.com and we'll do our best to sort everything out for you.

Hey there!

We'd love to help you out here, but it will be easiest if we can see your Campaign and the data it's returning. Please write in to help@moz.com with the details of what's happening, and we'll see if we can figure out what's going on with those callto: links!

I'm not seeing that URL coming up with Duplicate Title or Duplicate Content issues — when I search by that URL I see no Content issues at that URL. I do see that URL in the All Crawled Pages section, but I can't find it bringing up Content issues in the app.

That said, I took a look at your robots.txt file, and I think this could be a result of having an Allow command before the rest of the Disallow commands. I think possibly if you put that Allow command at the end of the block of Disallow commands, rogerbot would see the disallow for /page/ and stop crawling those URLs.

If you're still running into trouble, I would suggest writing in to us at help@moz.com so we can take a closer look at the Campaign and what could be going on there.

It's not that the data is false, just that it's outdated. The last time our crawler crawled your site it found those callto links were returning a 404, but if they're no longer doing so, you don't need to worry about them. That's why I suggested a robots.txt disallow for those pages — if you know those pages are fine and you're not worried about them, you can just tell our tools to skip crawling those pages entirely so we don't report those issues anymore.

The rest of the data in the report should be as accurate as possible, given that the freshest data in our index is about 60 days old or more.

Hey there! Tawny from Moz's Help team here.

I think the easiest way to resolve issues from those Call links would be to block our user-agent, dotbot (or Rogerbot for Moz Pro Campaigns) from accessing those URL paths. You could set up a Disallow directive for dotbot in your robots.txt file for the site. It would look something like this:

User-agent: dotbot
Disallow: /callto

That should prevent us from crawling those telephone number pages, and we'll stop surfacing those pages as 404ing pages.

I hope that helps! If you still have questions, feel free to shoot us a note over at help@moz.com and we'll do our best to sort everything out with you.

Drat. In that case, please send us a message at help@moz.com with the details of what's going on, and we'll do our best to sort things out for you. There are a couple more troubleshooting steps we can try from there.

So sorry for the inconvenience!

Hey VujaDay! Looks like our staff have endorsed an answer we think answers the question! If your question is answered to your satisfaction, go ahead and mark the question as Answered. If you've still got questions, feel free to drop us a line at help@moz.com and we'll do our best to sort everything out.

Hi there! Tawny from Moz's Help Team here!

I think I can help you figure out what's going on with your robots.txt file. First things first: we're not starting at the robots.txt URL you list. Our crawler always starts from your Campaign URL and goes from there, and it can't start at an HTTPS URL, so it starts at the HTTP version and crawls from there. So, the robots.txt file we're having trouble accessing is khadination.shop/robots.txt.

I ran a couple of tests, and it looks like this robots.txt file might be inaccessible from AWS (Amazon Web Services). When I tried to curl your robots.txt file from AWS I got a 302 temporary redirect error (https://www.screencast.com/t/jy4MkDZQNbQ), and when I ran it through hurl.it, which also runs on AWS, it returned an internal server error (https://www.screencast.com/t/mawknIyaMn).

One more thing — it looks like you have a wildcard character ( * ) for the user-agent as the first line in this robots.txt file. Best practices indicate that you should put all your specific user-agent disallow commands before a wildcard user-agent; otherwise those specific crawlers will stop reading your robots.txt file after the wildcard user-agent line, since they'll assume that those rules apply to them.

I think if you fix up those things, we should be able to access your robots.txt and crawl your site!

If you still have questions or run into more trouble, shoot us a note at help@moz.com and we'll do everything we can to help you sort everything out.

Hey there!

It looks like someone has already helped you out here — the only invoice I see on your account has already been refunded. It typically takes a few business days to see the funds returned to your account, so keep an eye out!

If you still have questions, feel free to reach out to us at help@moz.com and we'll do our best to sort everything out.

Hey there! Tawny from Moz's Help Team here. I think I can help explain why you're seeing fewer links with our tools than with others.

Open Site Explorer and the Link Analysis page of Moz Pro Campaigns are both tied to our Mozscape index, which tends to update roughly once a month.

Just a few points on how we compile our index:

• We grab the most recent index.
• We take the top 10 billion URLs with the highest MozRank (with a fixed limit on some of the larger domains).
• We start crawling from the top down until we've crawled ~130 billion URLs

The idea here is that we're focusing on the highest-quality links we can find, coming from the most prominent pages of authoritative sites. So, while you may not see every link for a site within our index, we're aiming to report the most valuable ones available!

Most new sites and links will be indexed by our spiders and available in Mozscape and Open Site Explorer within 60 days, but some take even longer for many reasons - including the crawl-ability of sites, the number of inbound links to them, and the depth of pages in subdirectories. This tends to bias our index in favor of newer links. Linking data is only stored in the index for around 180 days. After that point, unless the crawler has a compelling reason to return to your site and the sites linking to yours, recrawl and reindex them, those links to your site can fall back out of our index again. That doesn't mean they're not out there on the web affecting your SEO, just that our tools don't see them anymore.

You can see our most recently updated schedule here as well as some more technical metrics on our Mozscape API Updates page. You can also see when the last and next updates happened on the Open Site Explorer (OSE) homepage at any time.

Since Moz focuses on quality of links over quantity, we are always focused on the most relevant links to display to our users. It's possible that Moz's index will leave out some of the lower-quality (non-link juice providing) links out of our index because of this. So, that might explain why you may see some discrepancies with what other tools may be showing.

You can read more about how we build our index in our guide here.

I know this is a ton of information, so if you have any questions or if I didn't make anything clear enough, please don't hesitate to ask! You can always drop us a line at help@moz.com and we'll do our best to clear up any questions you might have.

Hey there! Tawny from Moz's Help Team here.

The only way to see what time our crawler is crawling your site would be to check for our user-agent in your server logs: rogerbot.

Unfortunately, there's no way to control the time of day at which we crawl your site. We just don't have a setting for that.
There's also no way to control the day of the week the crawl takes place, for similar reasons. Your site will be crawled around the same time each week, with no way of adjusting when that takes place. 
We currently don't have a way to slow the crawl rate, either. Normally you'd be able to use a crawl delay in your robots.txt file, but our new crawler isn't following those crawl delays at the moment. In the future, that will be how you would slow down our crawler, but for now that option isn't up and running. Sorry about that!

I'm afraid I wasn't too much help here, Patrick. My apologies I couldn't be more useful to you. If you still have questions, feel free to shoot us a note at help@moz.com and we'll do our best to sort things out with you.

Hi there! Thanks for writing in! Tawny from Moz's Help Team here.

That's a great question! Unfortunately, you can't add more than 3 competitors per campaign, at the moment. However, you do have the option to create a new campaign for the same website using another set of different competitors, if you wanted to do that!

If you were looking for extra competitive link data, Open Site Explorer would give you the ability to compare up to 5 sites within the "Compare Link Metrics" section.

Sorry for the inconvenience, but I hope this clears things up. If you've still got questions for us, feel free to drop us a line at help@moz.com and we'll do our best to sort things through with you!

Hey there!

Tawny from Moz's Help Team here.

Adding a disallow directive for /tag/ won't help with the example URL you've provided — that URL doesn't have /tag/ in the URL pathway. To block us from seeing content like that URL you listed, you'd need a disallow directive for /place_tag/.

If you include that disallow directive, that should stop us from seeing duplicate content on pages with /place_tag/ in the URL.

Hope that helps! If you've still got questions, feel free to shoot us a note over at help@moz.com and we'll do our best to sort things out with you.