Log File Analyzer Only Showing Spoofed Bots and No Verified Bots

[geodigitalmarketing]

Question for you guys: After analyzing some crawl data in Search Console in the sitemap section, I noticed that Google consistently isn't indexing about 3/4 of the client sites I work on that all use the same content management system. I began to wonder if maybe Google (and others) have a hard time crawling certain parts of the sites consistently, as finding a pattern here could lead me to investigate whether there's a CMS problem.

To research this, I started using a log file analyzer (Screaming Frog's version) for some of those clients. After loading the files, I noticed that none of the crawl activity logged by the servers is considered verified. I input one month's worth of log files, but when I switch the program to show only verified bots, all data disappears. Is it possible for a site not to have any search engines crawling it for a whole month? Given my experience, that seems unlikely, particularly since we've been submitting crawl requests. I know that doesn't guarantee a crawl, but it seems odd that it's never happening for any search engines across the board.

Context that might be helpful:

• I did check technical settings, and the sites are crawlable.
• The sites do appear in search but seem to be losing organic search traffic.

Thanks for any help you can provide!

[screamingfrog]

Hey David,

I thought I'd jump in here, as it's our tool

We have more information on bot verification here, including a troubleshooting section with common issues for genuine events being marked as spoofed -

https://www.screamingfrog.co.uk/log-file-analyser/user-guide/configuration/#verify-bots

You can also reach us via our support here - https://www.screamingfrog.co.uk/log-file-analyser/support/

Cheers.

Dan