The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Getting Started
    4. Crawling issue

    Crawling issue

    Getting Started
    2 2 52
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Mat_C
      Mat_C last edited by

      Hi,

      I have to set up a campaign for a webshop. This webshop is a subdomain itself.

      First question: The two subfolders I need to track are /nl_BE and /fr_BE. What is the best way to handle this? Shall I set up two different campaigns for each subfolder, or shall I just make one campaign and add tags to keywords?

      **Second question: **it seems like Moz can't crawl enough pages. There are no disallows in the robots.txt. Should I try putting the following at the top into my robots.txt?

      User-agent: rogerbot
      Disallow:

      Or is it because I want to crawl only a subdomain that it doesn't work?

      Thanks

      1 Reply Last reply Reply Quote 0
      • eli.myers
        eli.myers last edited by

        Hey,

        Thanks for reaching out to us!

        You can create a Campaign solely for a subdomain or subfolder by selecting the +Advanced setting in the Campaign set-up; just click the check box there and it will limit our Campaign audit to the pages on that specific subdomain or subfolder. From there, you can see in your Campaign Setting if you've set it up for just that chunk of your site, or for the entire root domain.

        I've got a guide to this process that I think may help.

        With regard to your second question, feel free to reach out to help@moz.com so that we can take a closer look 🙂

        Looking forward to hearing from you,

        Eli

        1 Reply Last reply Reply Quote 0
        • 1 / 1
        • First post
          Last post
        • Moz DA Issue
          BartonInteractive
          BartonInteractive
          0
          3
          33

        • When I crawl my site On Moz it says it can't access the robots.txt file, but crawl is fine on SEM Rush - Anyone know any reason for this?
          meghanpahinui
          meghanpahinui
          0
          3
          60

        • Why do ignored crawl issues still count as issues?
          meghanpahinui
          meghanpahinui
          0
          2
          82

        • Duplicate titles issue
          davebuts
          davebuts
          0
          4
          229

        • Crawl test
          DavidLee
          DavidLee
          0
          2
          178

        • Crawl Page Issues
          TheeDigital
          TheeDigital
          0
          7
          135

        • After fixing Crawl Errors, how long does it take to for Moz or Google to re-crawl a website?
          DougRoberts
          DougRoberts
          0
          2
          650

        • Crawl Diagnostics
          JulieWhite
          JulieWhite
          0
          6
          202

        Get started with Moz Pro!

        Unlock the power of advanced SEO tools and data-driven insights.

        Start my free trial
        Products
        • Moz Pro
        • Moz Local
        • Moz API
        • Moz Data
        • STAT
        • Product Updates
        Moz Solutions
        • SMB Solutions
        • Agency Solutions
        • Enterprise Solutions
        • Digital Marketers
        Free SEO Tools
        • Domain Authority Checker
        • Link Explorer
        • Keyword Explorer
        • Competitive Research
        • Brand Authority Checker
        • Local Citation Checker
        • MozBar Extension
        • MozCast
        Resources
        • Blog
        • SEO Learning Center
        • Help Hub
        • Beginner's Guide to SEO
        • How-to Guides
        • Moz Academy
        • API Docs
        About Moz
        • About
        • Team
        • Careers
        • Contact
        Why Moz
        • Case Studies
        • Testimonials
        Get Involved
        • Become an Affiliate
        • MozCon
        • Webinars
        • Practical Marketer Series
        • MozPod
        Connect with us

        Contact the Help team

        Join our newsletter
        Moz logo
        © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
        • Accessibility
        • Terms of Use
        • Privacy