Subdomain 403 error

[WeAreDigital_BE]

Hi Everyone,

A crawler from our SEO tool detects a 403 error from a link from our main domain to a a couple of subdomains. However, these subdomains are perfect accessibly. What could be the problem? Is this error caused by the server, the crawlbot or something else?

I would love to hear your thoughts.
Jens

[Roman-Delcarmen]

These are all for client errors. That means the page wasn’t found and something is wrong with the request. Whatever is happening though, the issue is typically on the client side:

403: Forbidden, So In your case, the first place that you need to check is your .htaccess and your Robots.txt file and make sure that they are not blocking any crawler or at least the crawler of your tools.

For example, some Hosting providers block all the crawlers that are not Google or Bing to save resources. So is usual that Roger (Moz Crawler) has problems to crawl a page that is blocked on the server side. Usually, Moz, Ahrefs, Semrush has this kind of problem so in summary

• Make sure your .htaccess and your Robots.txt is not blocking your crawler
• Make sure your hosting is not blocking your crawler
• If all the above does not work try to modify the user-agent of your tool

Hope this info helps you with your problem

[Roman-Delcarmen]

Which tool are you using is this a custom tool or commercial tool such as Screamingfrog?

[WeAreDigital_BE]

Hi Roman,

Thanks for your answer!

It's a commercial tool.

I checked the robots.txt file and .htaccess, but didn't saw any problems.
As you say, the problem can just be caused by the user-agent.

If so, this will not affect my SEO efforts, right?

[Roman-Delcarmen]

no at all