The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Other Research Tools
    4. Crawl report shows that it gets 4xx errors for pages that work fine. Why?

    Crawl report shows that it gets 4xx errors for pages that work fine. Why?

    Other Research Tools
    4 2 209
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • TeamViviRealEstate
      TeamViviRealEstate last edited by

      On the crawl report it has all these "Critical Crawler Issues". They all say "4xx Error", yet when i click on the link from the crawler report, it goes to a perfectly functioning page, not a 404 page or anything. If i click in it actually says it's a 403 error. It's all for pages generated by the IDX solution for our real estate website.

      Is Moz broken or am i missing something?

      Here are a couple examples:

      <dl class="crawl-page-details-list">

      <dd class="crawl-page-details-list-emphasis">https://teamvivi.com/homes-for-sale-map-search/</dd>

      <dd class="crawl-page-details-list-emphasis">

      <dl class="crawl-page-details-list">

      <dd class="crawl-page-details-list-emphasis">https://teamvivi.com/email-alerts/</dd>

      </dl>

      </dd>

      </dl>

      1 Reply Last reply Reply Quote 0
      • WebElaine
        WebElaine last edited by

        A 403 error means the crawler isn't being allowed to access that page. You should check robots.txt and if nothing is blocking there, check with your host to see if they can help identify why Roger is being blocked from those particular pages.

        Related question: https://moz.com/community/q/403-error-code-but-page-shows-no-error

        On a side note, those pages took a really long time to load for me, so I'd also look into site speed. Visual Composer for one is adding all sorts of different stylesheets and bogging things down.

        TeamViviRealEstate 1 Reply Last reply Reply Quote 1
        • TeamViviRealEstate
          TeamViviRealEstate @WebElaine last edited by

          Ugh, that's not good news since my entire site is built with Visual Composer basically.... Interestingly though, those pages are not even using Visual Composer. Those are pages that are generated by the IDX solution (iHomeFinder by Optima Express).

          WebElaine 1 Reply Last reply Reply Quote 0
          • WebElaine
            WebElaine @TeamViviRealEstate last edited by

            There are definitely other optimizations you can do, and Visual Composer may not be the biggest slowdown. I just noticed when I viewed the source code that it was inserting a lot of separate CSS files and so it may be contributing some. Other optimizations like going through PageSpeed Insights' recommendations may get you farther. 🙂

            But any themes and plugins you use on a WP site, those resources are loaded across the whole site - even on pages they're not used on. So in a lot of cases, selecting a premade theme or having one built from scratch can make your site a lot faster than using page builders. Basically it's a tradeoff: do you want to pay someone to code a theme specifically for you and do the heavy lifting up front (or make design compromises and use a well-coded prebuilt theme), so the code that renders to visitors is lightweight; or do you want to just go wysiwyg on the admin side, so the code that renders to visitors is bogged down but you don't have to hire a developer?

            1 Reply Last reply Reply Quote 0
            • 1 / 1
            • First post
              Last post
            • Crawl Test : Error attempting to request HTTPS page
              MattRoney
              MattRoney
              0
              4
              645

            • Cannot Crawl ... 612 : Page banned by error response for robots.txt.
              1sixty8
              1sixty8
              0
              6
              377

            • Is there a tool that works like crawl test that allows more than 3000 pages?
              Dr-Pete
              Dr-Pete
              0
              4
              642

            • I'm getting a Crawl error 605 Page Banned by robots.txt, X-Robots-Tag HTTP Header, or Meta Robots Tag
              DavidLee
              DavidLee
              1
              13
              939

            • Error 4XX showing by SEOmoz tool
              GPainter
              GPainter
              0
              4
              112

            • Can't get on page grader to work properly.
              DavidLee
              DavidLee
              0
              2
              83

            • When attempting to crawl my site, I'm getting the error: Oops! That URL doesn’t resolve, which means your report will be blank. Please fix the issue or change the URL. What's going on here?
              KeriMorgret
              KeriMorgret
              0
              4
              305

            • Moz crawl suddenly shows much less pages from what I really have
              DavidLee
              DavidLee
              1
              3
              255

            Get started with Moz Pro!

            Unlock the power of advanced SEO tools and data-driven insights.

            Start my free trial
            Products
            • Moz Pro
            • Moz Local
            • Moz API
            • Moz Data
            • STAT
            • Product Updates
            Moz Solutions
            • SMB Solutions
            • Agency Solutions
            • Enterprise Solutions
            • Digital Marketers
            Free SEO Tools
            • Domain Authority Checker
            • Link Explorer
            • Keyword Explorer
            • Competitive Research
            • Brand Authority Checker
            • Local Citation Checker
            • MozBar Extension
            • MozCast
            Resources
            • Blog
            • SEO Learning Center
            • Help Hub
            • Beginner's Guide to SEO
            • How-to Guides
            • Moz Academy
            • API Docs
            About Moz
            • About
            • Team
            • Careers
            • Contact
            Why Moz
            • Case Studies
            • Testimonials
            Get Involved
            • Become an Affiliate
            • MozCon
            • Webinars
            • Practical Marketer Series
            • MozPod
            Connect with us

            Contact the Help team

            Join our newsletter
            Moz logo
            © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
            • Accessibility
            • Terms of Use
            • Privacy