The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. Significant Google crawl errors

    Significant Google crawl errors

    Intermediate & Advanced SEO
    3 2 105
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • GregWalt
      GregWalt last edited by

      We've got a site that continuously like clockwork encounters server errors with when Google crawls it.  Since the end of last year it will go a week fine, then it will have two straight weeks of 70%-100% error rate when Google tries to crawl it.  During this time you can still put the URL in and go to the site, but spider simulators return a 404 error.  Just this morning we had another error message, I did a fetch and resubmit, and magically now it's back.  We changed servers on it in Jan to Go Daddy because the previous server (Tronics) kept getting hacked.  IIt's built in html so I'm wondering if it's something in the code maybe?

      http://www.campteam.com/

      1 Reply Last reply Reply Quote 1
      • Mike_Davis
        Mike_Davis last edited by

        I see that your site is dealing fine with 404 errors.  Hrmm.  Could you copy and paste the crawl error URLs you are getting from webmaster tools?  Thanks!

        BTW I noticed that you have a duplicate content issue in that you haven't removed the www from your URL.  You should add the following code to your .htaccess file.

        <code class="htaccess" title="in your .htaccess file">RewriteEngine On
        RewriteCond %{HTTP_HOST} !^my-domain\.com$ [NC]
        RewriteRule ^(.*)$ http://my-domain.com/$1 [R=301,L]</code> 
        
        GregWalt 1 Reply Last reply Reply Quote 1
        • GregWalt
          GregWalt @Mike_Davis last edited by

          This is the URL error list in Webmaster Tools

          | Forms/Camp.pdf | 404 | 7/9/13 |
          | | 2 | sportsinsurance.php | 404 | 5/2/13 |
          | | 3 | Forms/Waiver.pdf | 404 | 7/2/13 |
          | | 4 | metro/index.htm | 404 | 6/21/13 |
          | | 5 | Forms/Camp_Tournament_Application.pdf | 404 | 7/9/13 |
          | | 6 | Forms/Spectator.pdf | 404 | 7/9/13 |
          | | 7 | Forms/Boxing.pdf | 404 | 5/6/13 |
          | | 8 | sports-camp-insurance.html | 404 | 6/16/13 |
          | | 9 | forms/T.C.S._ | 404 | 7/3/13 |
          | | 10 | Camp | 404 | 6/14/13 |
          | | 11 | Forms/Sports.pdf | 404 | 4/21/13 |
          | | 12 | pages/clients.html | 404 | 4/15/13 |

          |   |

          http://www.campteam.com/: Googlebot can't access your site****July 10, 2013

          Over the last 24 hours, Googlebot encountered 13 errors while attempting to connect to your site. Your site's overall connection failure rate is 72.2%.

          I've got 23 of these messages going back to 11/12

          It tells me that no Robots.txt Fetch issues were encountered, or DNS issues.  All errors are related to server connectivity according to Google.

          1 Reply Last reply Reply Quote 0
          • 1 / 1
          • First post
            Last post
          • Google Webmaster tools -Fixing over 20,000+ crawl errors
            LoganRay
            LoganRay
            0
            2
            68

          • Why is my site not getting crawled by google?
            DonnaDuncan
            DonnaDuncan
            0
            4
            162

          • Google Webmaster successfully fetched one of my webpages. Does that mean Google will crawl them or readable by bots?
            Gyorgy
            Gyorgy
            0
            2
            72

          • Somthing weird in my Google Webmaster Tools Crawl Errors...
            KristyO
            KristyO
            0
            3
            1.6k

          • Crawl errors in GWT!
            RuthBurrReedy
            RuthBurrReedy
            0
            4
            636

          • 202 error page set in robots.txt versus using crawl-able 404 error
            EricaMcGillivray
            EricaMcGillivray
            0
            3
            870

          • Is there any delay between crawling a page by google and displaying of the ratings in rich snippet of the results in google?
            NEWCRAFT
            NEWCRAFT
            0
            3
            545

          • Google crawl errors & serps [adult site]
            Ramp
            Ramp
            0
            6
            799

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy