Google Crawling Issues! How Can I Get Google to Crawl My Website Regularly?

[farhanm]

Hi Everyone!

My website is not being crawled regularly by Google - there are weeks when it's regular but for the past month or so it does not get crawled for seven to eight days. There are some specific pages, that I want to get ranked but they of late are not being crawled AT ALL unless I use the 'Fetch As Google' tool! That's not normal, right? I have checked and re-checked the on-page metrics for these pages (and the website as a whole, backlinking is a regular and ongoing process as well! Sitemap is in place too! Resubmitted it once too! This issue is detrimental to website traffic and rankings! Would really appreciate insights from you guys! Thanks a lot!

[UmarKhan]

Hello Farhan,

Yes, it's not normal. Did you change your web hosting in this period? or have you faced any kind of penalty in recent past? Usually this happens when you just recovered.

You also need to look out your Search Console for crawling problems, are there any crawling or server issues you're facing? Sometimes, your .htaccess creates crawling problems, make sure to check again. Also, see how fasts your pages are loading, if the speed is low it also cause the crawling delayed.

Hope this helps!

Umar

[AndersS]

Hi!

Are all pages available for GoogleBot, or are they perhaps blocked by robots.txt or a meta tag?

Do you have a URL for the website so we can take a look?

Anders

[libero_net]

Hi

How regular are you updating content on the site?

We've found that regular updating and tweaks to content on pages to improve the pages content & user's experience helps the page indexing & frequency.

Looking at analytics and seeing what customers are searching for on your site can help you then improve the content on pages, by going back amending and improving page content we've found Google returns more often to re-index.

Dan

[farhanm]

Hi Umar! There’s been no change in web hosting nor a penalty! We regularly monitor the Search Console and the speed is good too! Thanks a lot for your time though!

[farhanm]

Hi Anders! They are all available to the Googlebot! I’m afraid, it won’t be possible for me to share the URL on a public forum! Please check your inbox! Thank you so much for your insights and concern!

[farhanm]

Hi Dan! Thanks a lot for your views! This might be it! Content on the website is not updated very regularly owing to the nature of the site. Perhaps updating the blog more regularly or making tweaks in the meta script could help as per your suggestions?

[libero_net]

Hi Farhan

Yes sure, we've definitely found this helps bring the crawlers back to the site more regular!

Hope it helps

Dan