The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Link Explorer
    4. Is there some way to tell the Moz crawler not to crawl URL's with particular dynamic tags such as "?redirect-to:http//" ?

    Is there some way to tell the Moz crawler not to crawl URL's with particular dynamic tags such as "?redirect-to:http//" ?

    Link Explorer
    2 2 106
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • RegistrarCorp
      RegistrarCorp last edited by

      We are encountering an issue where the crawler is finding a ton of pages from our wordpress login url that has this dynamic tag in it to kinds of different blog entries.  It's madness.  I can't figure out what is causing these URLs to generate to be crawled in the first place!  Does this sound familiar to anyone out there, any constructive suggestions?  Robots text or maybe meta robots tags that would resolve this crawl issue?

      1 Reply Last reply Reply Quote 0
      • ecommercebc
        ecommercebc last edited by

        Hey,

        I'm not sure if this is resolved for you, but as you suggest you can do something with robots.txt. Specifically, you could use wildcards to capture these URLs and tell Rogerbot (Moz's crawler) to ignore them. Here's a great Stackoverflow query to get you started and details on how block Rogerbot, you can take a look here.

        1 Reply Last reply Reply Quote 0
        • 1 / 1
        • First post
          Last post
        • How to tell moz about the domain redirects?
          samantha.chapman
          samantha.chapman
          0
          2
          38

        • Moz crawling http rather than https site
          dave.kudera
          dave.kudera
          0
          3
          91

        • Moz tool to estimate a website's organic traffic?
          jocameron
          jocameron
          0
          2
          903

        • Why doesn't Moz crawler follow robots.txt?
          Tylerj
          Tylerj
          0
          10
          879

        • "Https://" version of my home page has PA of 39, while the "https://www" version has a PA of 43?
          Dezzign
          Dezzign
          0
          2
          87

        • What's with Moz continuously pushing back the Index Update date?
          DirkC
          DirkC
          1
          2
          131

        • Moz can't crawl domain due to IP Geo redirect loop
          TomVolpe
          TomVolpe
          0
          2
          411

        • What if all the URL's in my website reported by MoZ do not exist on my website? They existed on an old one that has now been rebuilt?
          Bonx
          Bonx
          0
          3
          91

        Get started with Moz Pro!

        Unlock the power of advanced SEO tools and data-driven insights.

        Start my free trial
        Products
        • Moz Pro
        • Moz Local
        • Moz API
        • Moz Data
        • STAT
        • Product Updates
        Moz Solutions
        • SMB Solutions
        • Agency Solutions
        • Enterprise Solutions
        • Digital Marketers
        Free SEO Tools
        • Domain Authority Checker
        • Link Explorer
        • Keyword Explorer
        • Competitive Research
        • Brand Authority Checker
        • Local Citation Checker
        • MozBar Extension
        • MozCast
        Resources
        • Blog
        • SEO Learning Center
        • Help Hub
        • Beginner's Guide to SEO
        • How-to Guides
        • Moz Academy
        • API Docs
        About Moz
        • About
        • Team
        • Careers
        • Contact
        Why Moz
        • Case Studies
        • Testimonials
        Get Involved
        • Become an Affiliate
        • MozCon
        • Webinars
        • Practical Marketer Series
        • MozPod
        Connect with us

        Contact the Help team

        Join our newsletter
        Moz logo
        © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
        • Accessibility
        • Terms of Use
        • Privacy