The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Search Engine Trends
    4. Bot Crawling issues

    Bot Crawling issues

    Search Engine Trends
    5 3 78
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • SubhaVaastu
      SubhaVaastu last edited by

      Dear all, Is this cache:www.subhavaastu.com is not working now with Google. Why it is not showing when the site is crawled. What were the new algorithms which Google is adopted? in my searches of all of my site internal links of www.SubhaVaastu.com, I observed only 404 instead of Google visiting time and date. After observing this 404 of each of my site links, I understand Google stopped crawling my site.

      Some more examples are shown below with other websites:

      cache:www.vastuwebsite.com ("NOT" showing when Google visited this site)

      cache:www.vastuconsultantusa.com ("NOT" showing when Google visited this site)

      cache:www.shubhavaastu.com ("NOT" showing when Google visited this site)

      cache:www.subhavastu.com (Showing when Google visited this site)

      To my surprise, I noticed that Google crawled latest links in my site, which I added a new link (https://www.subhavaastu.com/remove-negativity.html) just 10 days back, this new link was clearly crawled by Google. I typed "remove negativity subhavaastu", I saw the results with this new page in the SERP, but on the same way when I typed "cache:www.subhavaastu.com/remove-negativity.html", it is showing again 404.

      what is happening with Google, is Google is following any new algorithms now. Is google changed any new concept? or, is my site is penalized in any case, I think it may not be, because if my site is penalized, then Google should not visit the new links and should not show the results with my site. Coming with my site, its pure from viruses, no malicious codes. Indeed it's an article based site, which has a good reputation. This domain is taken in the year 2003. We never spam anywhere we never did any wrong methods.

      If my site is penalized, then is it manual penalized or bot penalized. I thoroughly checked webmaster tools and google console, I never found any notices or any note from Google.

      Require experts analyzation on this doubt. Thanks in Advance.

      1 Reply Last reply Reply Quote 0
      • Vijay-Gaur
        Vijay-Gaur last edited by

        It can be due to a lot of SEO/technical issues, Here are a couple of observations/suggestions:

        • Your site works with https:// and with http:// both and also with www. and without www. , this might be an issue for duplicate content. Please make sure that you have only one version with or without www. or https://
        • Check your website for malware and viruses just as a precaution.
        SubhaVaastu 1 Reply Last reply Reply Quote 2
        • seoman10
          seoman10 last edited by

          You are asking quite a lot of questions here, basically if your site has been around for a reasonable amount of time and your page is accessible and can be crawled by search engines you should be able to Google your domain name (including the TLD) like "mydomain.com", if that doesn't show up you either have a serious technical error or you have had a penalty.

          As for telling whether it is bot or manual Google search console should give you a bit more information.  Personally as far as I can see a penalty is a penalty and needs to be overcome one way or another, whether it is automatic or manual is a bit beside the point!

          SubhaVaastu 1 Reply Last reply Reply Quote 2
          • SubhaVaastu
            SubhaVaastu @Vijay-Gaur last edited by

            Dear Vijay, Thank You.

            You are an expert in this field, I respect your thought, but I am thinking there should be some other reason, that might cause of this problem. I am MOZ PRO subscriber, if there is, canonical issues, I will get such notifications. Long back, there are some canonical issues, we fix them.

            Site is totally free from malware and viruses, I just now also observed Google webmaster tools and checked other ways, there is nothing viruses found in my site.

            Can it be possible to find any other issues which caused this problem.

            Many thanks.

            1 Reply Last reply Reply Quote 0
            • SubhaVaastu
              SubhaVaastu @seoman10 last edited by

              Dear SeoMAN,

              Thanks for your answer. I just tried at google, its shows normal, I just checked google search console and webmaster tools, there is nothing serious shows. Can it be possible to try to find out what was the main cause.

              Is I taken the right image, is I searched exactly what you are asking about?

              Many thanks.

              7J2WLXd

              1 Reply Last reply Reply Quote 0
              • 1 / 1
              • First post
                Last post
              • Training - How Google Crawls & Indexes Websites
                BeckyKey
                BeckyKey
                0
                5
                82

              • Log File Analyzer Only Showing Spoofed Bots and No Verified Bots
                screamingfrog
                screamingfrog
                0
                2
                161

              • Is googlebot the slowest bot?
                RyanPurkey
                RyanPurkey
                0
                4
                67

              • Anyone else notice a global Google cache issue?
                mdarvish
                mdarvish
                0
                4
                277

              • Should we crawl user profiles?
                OlegKorneitchouk
                OlegKorneitchouk
                0
                2
                268

              • Issue now resolved
                Dr-Pete
                Dr-Pete
                0
                13
                872

              • Issue: Having to Fight Product Marketing to Use Keywords
                SmartBear
                SmartBear
                0
                3
                556

              • Is URL appearance defined by crawling or by XML sitemap
                Jordanrg
                Jordanrg
                0
                3
                786

              Get started with Moz Pro!

              Unlock the power of advanced SEO tools and data-driven insights.

              Start my free trial
              Products
              • Moz Pro
              • Moz Local
              • Moz API
              • Moz Data
              • STAT
              • Product Updates
              Moz Solutions
              • SMB Solutions
              • Agency Solutions
              • Enterprise Solutions
              • Digital Marketers
              Free SEO Tools
              • Domain Authority Checker
              • Link Explorer
              • Keyword Explorer
              • Competitive Research
              • Brand Authority Checker
              • Local Citation Checker
              • MozBar Extension
              • MozCast
              Resources
              • Blog
              • SEO Learning Center
              • Help Hub
              • Beginner's Guide to SEO
              • How-to Guides
              • Moz Academy
              • API Docs
              About Moz
              • About
              • Team
              • Careers
              • Contact
              Why Moz
              • Case Studies
              • Testimonials
              Get Involved
              • Become an Affiliate
              • MozCon
              • Webinars
              • Practical Marketer Series
              • MozPod
              Connect with us

              Contact the Help team

              Join our newsletter
              Moz logo
              © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
              • Accessibility
              • Terms of Use
              • Privacy