Crawler doesn't discover the links in the main nav

[bbop33]

Hi Moz Community,

We have a headless ecom (Magento) client that I'm trying to crawl the site.  During the crawl, the tool (Screaming Frog) cannot discover the sub-category URLs in the main navigation when I start crawling via homepage.

Similarly, when I start crawling with one of the sub-category page, it doesn't crawl any of the product URLs on the sub-category page itself.

When I inspect product and sub-cat URLs through Search Console, they seem as indexed and if I view how Googlebot rendered the sub-category page, I can see the product URLs on the sub-cat page too.

If you have any idea what's the issue with Screaming Frog and would like to help me out, I'd be so grateful!

Thanks in advance

[katemorris]

Ah, I might know your problem. What is your site? We had this issue at my last company, had to do with crawling using JS. If you will send me the site, I can take a look.

[bbop33]

Hi Kate,

Thank you for your time to respond! Is there any way that I can contact you directly?

By the way, I've tried crawling after changing configurations as 'rendering Javascript' however, it still didn't discover product URLs.

Thank you!

[katemorris]

Find me on twitter @katemorris and follow me. Tell me your name and I'll follow you and we can DM.

[bbop33]

Hi Kate,

Thank you! I followed you on Twitter, my user name is @curetuvana