The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Getting Started
    4. Moz not able to crawl our site - any advice?

    Moz not able to crawl our site - any advice?

    Getting Started
    3 3 81
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • MyFamilClubLtd
      MyFamilClubLtd last edited by

      When I try and crawl our site through Moz it gives this message:

      Moz was unable to crawl your site on Aug 7, 2019. Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. Update these tags to allow your page and the rest of your site to be crawled. If this error is found on any page on your site, it prevents our crawler (and some search engines) from crawling the rest of your site. Typically errors like this should be investigated and fixed by the site webmaster.

      I have been through all the help and doesn't seem to be any issues. You can check the site and robots.txt here: https://myfamilyclub.co.uk/robots.txt.

      Anyone got any advice on where I could go to get this sorted?

      1 Reply Last reply Reply Quote 1
      • eli.myers
        eli.myers last edited by

        Hi,

        I'm sorry to hear about this! Would you be able to write into help@moz.com as this will allows us to investigate into this via the most convenient method.

        Looking forward to hearing from you,

        Eli

        1 Reply Last reply Reply Quote 1
        • Salience_Search_Marketing
          Salience_Search_Marketing last edited by

          Hey,

          We crawled the site just fine with Screaming Frog. Perhaps this is a Moz issue?

          Looks like they are well aware of it though.

          Thanks!

          1 Reply Last reply Reply Quote 0
          • 1 / 1
          • First post
            Last post
          • Why does Moz only seem to be crawling a snap shot of the site I am working with?
            dsmith802020
            dsmith802020
            0
            3
            32

          • Can't Crawl Site - but deducting crawls.
            dave.kudera
            dave.kudera
            1
            3
            52

          • When I crawl my site On Moz it says it can't access the robots.txt file, but crawl is fine on SEM Rush - Anyone know any reason for this?
            meghanpahinui
            meghanpahinui
            0
            3
            60

          • Site Crawl - Crawls only homepage?
            Shotlife_Studio
            Shotlife_Studio
            0
            3
            107

          • Why is Moz unable to crawl my site?
            jbarinaga
            jbarinaga
            0
            5
            145

          • Moz was unable to crawl your site - robots.txt
            JRGRDZ
            JRGRDZ
            0
            5
            386

          • After fixing Crawl Errors, how long does it take to for Moz or Google to re-crawl a website?
            DougRoberts
            DougRoberts
            0
            2
            650

          • New to using MOZ. Familiar with Google Analytics. With MOZ is there a code snippet to include on my site?
            Ron_McCabe
            Ron_McCabe
            0
            2
            88

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy