Why isn't the Moz crawler getting all of my item pages?

[Tylerj]

I am stumped and Moz is being terrible to work with.

This site has about 40k pages

39,800 of them are item pages roughly.

Moz is only finding about 2400 of my pages. So they are missing most but not all of my item pages.

I do not know which item pages they are missing.

The fact that they are finding about 2k but not the rest leads me to believe the crawler is struggling with pagination.

The site is built on Magento 2 and uses the Amasty Layered Navigation extension.

Does anyone have any ideas?

[SEOBrunch]

Hi Tyler, are you setting your pagination pages as nofollow? That might be causing the issue. Any chance you can post the URL and we can try crawling the site to see if we hit any issues with a crawlers like screaming frog?

[Tylerj]

Meghan from Moz pointed out I had Disallow: /*? in the robots.txt and that appears to have been the problem. It is the default Magento 2 setting, so not sure why they had that in there.