The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Other Research Tools
    4. Crawl report shows that it gets 4xx errors for pages that work fine. Why?

    Crawl report shows that it gets 4xx errors for pages that work fine. Why?

    Other Research Tools
    4 2 209
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • TeamViviRealEstate
      TeamViviRealEstate last edited by

      On the crawl report it has all these "Critical Crawler Issues". They all say "4xx Error", yet when i click on the link from the crawler report, it goes to a perfectly functioning page, not a 404 page or anything. If i click in it actually says it's a 403 error. It's all for pages generated by the IDX solution for our real estate website.

      Is Moz broken or am i missing something?

      Here are a couple examples:

      <dl class="crawl-page-details-list">

      <dd class="crawl-page-details-list-emphasis">https://teamvivi.com/homes-for-sale-map-search/</dd>

      <dd class="crawl-page-details-list-emphasis">

      <dl class="crawl-page-details-list">

      <dd class="crawl-page-details-list-emphasis">https://teamvivi.com/email-alerts/</dd>

      </dl>

      </dd>

      </dl>

      1 Reply Last reply Reply Quote 0
      • WebElaine
        WebElaine last edited by

        A 403 error means the crawler isn't being allowed to access that page. You should check robots.txt and if nothing is blocking there, check with your host to see if they can help identify why Roger is being blocked from those particular pages.

        Related question: https://moz.com/community/q/403-error-code-but-page-shows-no-error

        On a side note, those pages took a really long time to load for me, so I'd also look into site speed. Visual Composer for one is adding all sorts of different stylesheets and bogging things down.

        TeamViviRealEstate 1 Reply Last reply Reply Quote 1
        • TeamViviRealEstate
          TeamViviRealEstate @WebElaine last edited by

          Ugh, that's not good news since my entire site is built with Visual Composer basically.... Interestingly though, those pages are not even using Visual Composer. Those are pages that are generated by the IDX solution (iHomeFinder by Optima Express).

          WebElaine 1 Reply Last reply Reply Quote 0
          • WebElaine
            WebElaine @TeamViviRealEstate last edited by

            There are definitely other optimizations you can do, and Visual Composer may not be the biggest slowdown. I just noticed when I viewed the source code that it was inserting a lot of separate CSS files and so it may be contributing some. Other optimizations like going through PageSpeed Insights' recommendations may get you farther. 🙂

            But any themes and plugins you use on a WP site, those resources are loaded across the whole site - even on pages they're not used on. So in a lot of cases, selecting a premade theme or having one built from scratch can make your site a lot faster than using page builders. Basically it's a tradeoff: do you want to pay someone to code a theme specifically for you and do the heavy lifting up front (or make design compromises and use a well-coded prebuilt theme), so the code that renders to visitors is lightweight; or do you want to just go wysiwyg on the admin side, so the code that renders to visitors is bogged down but you don't have to hire a developer?

            1 Reply Last reply Reply Quote 0
            • 1 / 1
            • First post
              Last post
            • Moz Crawl Report Increase in Errors?
              LisaHunt
              LisaHunt
              0
              3
              64

            • Site Crawl report show strange duplicate pages
              Neo4j
              Neo4j
              0
              4
              136

            • Rogerbot will not crawl my site! Site URL is https but keep getting and error that homepage (http) can not be accessed. I set up a second campaign to alter the target url to the newer https version but still getting the same error! What can I do?
              DirkC
              DirkC
              0
              3
              200

            • Crawl Test : Error attempting to request HTTPS page
              MattRoney
              MattRoney
              0
              4
              645

            • All of a sudden a number of my key pages are getting 403 errors with Moz?
              InvoqMarketing
              InvoqMarketing
              0
              3
              444

            • Cannot Crawl ... 612 : Page banned by error response for robots.txt.
              1sixty8
              1sixty8
              0
              6
              377

            • Moz Crawl Report showing non-existent Duplicate Errors since new reporting layout
              ChiarynMiranda
              ChiarynMiranda
              1
              2
              62

            • Moz crawl suddenly shows much less pages from what I really have
              DavidLee
              DavidLee
              1
              3
              255

            Get started with Moz Pro!

            Unlock the power of advanced SEO tools and data-driven insights.

            Start my free trial
            Products
            • Moz Pro
            • Moz Local
            • Moz API
            • Moz Data
            • STAT
            • Product Updates
            Moz Solutions
            • SMB Solutions
            • Agency Solutions
            • Enterprise Solutions
            • Digital Marketers
            Free SEO Tools
            • Domain Authority Checker
            • Link Explorer
            • Keyword Explorer
            • Competitive Research
            • Brand Authority Checker
            • Local Citation Checker
            • MozBar Extension
            • MozCast
            Resources
            • Blog
            • SEO Learning Center
            • Help Hub
            • Beginner's Guide to SEO
            • How-to Guides
            • Moz Academy
            • API Docs
            About Moz
            • About
            • Team
            • Careers
            • Contact
            Why Moz
            • Case Studies
            • Testimonials
            Get Involved
            • Become an Affiliate
            • MozCon
            • Webinars
            • Practical Marketer Series
            • MozPod
            Connect with us

            Contact the Help team

            Join our newsletter
            Moz logo
            © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
            • Accessibility
            • Terms of Use
            • Privacy