Site crawl only shows homepage

[WeAreDigital_BE]

Hi everyone,

A client of us has a quite new website with a lot of URLs. (Google Search Console indicates around 5300.) However, when I execute a site crawl with screaming frog, or a crawl test in MOZ, it only shows me one URL, the homepage.

Does somebody have an idea why the other pages of the website are not showing up?

Thanks,
Jens

[eli.myers]

Hey Jens,

Thanks for reaching out to us!

Would you be able to head over to this link https://moz.com/help/contact and drop us a quick email. This will allow us to investigate further (and for that, we would need to access your Moz Pro account).

Looking forward to hearing from you!

Eli

UPDATE - The Campaign was set up for us to only crawl a specific subfolder as opposed to the whole domain.  All sorted now

[SEOchris1]

The crawlers may be blocked inside of the robots.txt file.  In Screaming Frog change the User Agent to Googlebot and recrawl the site.  If that doesn't work you'll have to check the server log files inside of the hosting account for the site.  Those may tell you what the crawl doesn't work.

[WeAreDigital_BE]

Hi SEOchris,

Thanks for your answer. I checked the robots.txt file, changed the User Agent to Googlebot in Screaming Frog, but non of these gave new insights. For now, we don't have access yet to the server log files, but when we have, hopefully they will tell us more.