The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. SEOMOZ crawler is still crawling a subdomain despite disallow

    SEOMOZ crawler is still crawling a subdomain despite disallow

    Intermediate & Advanced SEO
    4 3 375
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • TheNorthernOffice79
      TheNorthernOffice79 last edited by

      This is for our client with a subdomain. We only want to analyze their main website as this is the one we want to SEO. The subdomain is not optimized so we know it's bound to have lots of errors.

      We added the disallow code when we started and it was working fine. We only saw the errors for the main domain and we were able to fix them. However, just a month ago, the errors and warnings spiked up and the errors we saw were for the subdomain.

      As far as our web guys are concerned. the disallow code is still there and was not touched.    User-agent: rogerbot     Disallow: /

      We would like to know if there's anything we might have unintentionally changed or something we need to do so that the SEOMOZ crawler will stop going through the subdomain.

      Any help is greatly appreciated!

      1 Reply Last reply Reply Quote 0
      • Andropenis_Australia
        Andropenis_Australia last edited by

        I have heard of this before recently, I think possibly the moz crawler all or sometimes now just ignores the disallow because it is not a usual S.E crawler.

        Hopefully one of the staff can provide some insight in this for you.

        All the best.

        1 Reply Last reply Reply Quote 0
        • Peterli
          Peterli last edited by

          John,

          Thanks for writing in!  I would like to take a look at which project you guys were working with that this is happening.  I will go ahead and start a ticket so we can better answer your questions 🙂 You should hear from me soon!

          Best,

          Peter
          Moz Help Team.

          TheNorthernOffice79 1 Reply Last reply Reply Quote 1
          • TheNorthernOffice79
            TheNorthernOffice79 @Peterli last edited by

            Thanks Peter for your assistance.

            Hope to hear from the SEOMOZ team soon with regards to this issue.

            1 Reply Last reply Reply Quote 0
            • 1 / 1
            • First post
              Last post
            • Robots.txt Disallowed Pages and Still Indexed
              Igor.Go
              Igor.Go
              0
              3
              2.9k

            • Pages with rel "next"/"prev" still crawling as duplicate?
              KeriMorgret
              KeriMorgret
              0
              3
              655

            • Seomoz crawl reporting thousands of 302 redirects?
              evolvingSEO
              evolvingSEO
              0
              4
              413

            • Why does SEOmoz bot see duplicate pages despite I am using the canonical tag?
              fablau
              fablau
              0
              9
              245

            • Is SEOmoz.org creating duplicate content with their CDN subdomain?
              JoelDay
              JoelDay
              1
              4
              2.2k

            • Website up for 3 months still not crawl
              David_ODonnell
              David_ODonnell
              0
              3
              361

            • How to Disallow Specific Folders and Sub Folders for Crawling?
              RyanKent
              RyanKent
              0
              14
              1.5k

            • Old pages still crawled by SE returning 404s. Better to put 301 or block with robots.txt ?
              RyanKent
              RyanKent
              0
              2
              721

            Get started with Moz Pro!

            Unlock the power of advanced SEO tools and data-driven insights.

            Start my free trial
            Products
            • Moz Pro
            • Moz Local
            • Moz API
            • Moz Data
            • STAT
            • Product Updates
            Moz Solutions
            • SMB Solutions
            • Agency Solutions
            • Enterprise Solutions
            • Digital Marketers
            Free SEO Tools
            • Domain Authority Checker
            • Link Explorer
            • Keyword Explorer
            • Competitive Research
            • Brand Authority Checker
            • Local Citation Checker
            • MozBar Extension
            • MozCast
            Resources
            • Blog
            • SEO Learning Center
            • Help Hub
            • Beginner's Guide to SEO
            • How-to Guides
            • Moz Academy
            • API Docs
            About Moz
            • About
            • Team
            • Careers
            • Contact
            Why Moz
            • Case Studies
            • Testimonials
            Get Involved
            • Become an Affiliate
            • MozCon
            • Webinars
            • Practical Marketer Series
            • MozPod
            Connect with us

            Contact the Help team

            Join our newsletter
            Moz logo
            © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
            • Accessibility
            • Terms of Use
            • Privacy