The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. On-Page / Site Optimization
    4. Need suggestion: Should the user profile link be disallowed in robots.txt

    Need suggestion: Should the user profile link be disallowed in robots.txt

    On-Page / Site Optimization
    6 3 343
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • AnkurJ
      AnkurJ last edited by

      I maintain a myBB based forum here. The user profile links look something like this http://www.learnqtp.com/forums/User-Ankur

      Now in my GWT, I can see many 404 errors for user profile links. This is primarily because we have tight control over spam and auto-profiles generated by bots. Either our moderators or our spam control software delete such spammy member profiles on a periodic basis but by then Google indexes those profiles.

      I am wondering, would it be a good idea to disallow User profiles links using robots.txt?

      Something like

      Disallow: /forums/User-*

      1 Reply Last reply Reply Quote 0
      • IM_Learner
        IM_Learner last edited by

        Use meta="noindex" tag for the user profile pages. It is more effective.

        Regards

        AnkurJ 1 Reply Last reply Reply Quote 0
        • AnkurJ
          AnkurJ @IM_Learner last edited by

          Let me re-phrase my question a bit. Would it be a good idea to get the user profile links noindexed in the first place? or should we keep them indexed in Google?

          KristinaKledzik 1 Reply Last reply Reply Quote 0
          • IM_Learner
            IM_Learner last edited by

            If you are talking about user profile pages, it is your call.

            If you are talking about links that users post in profile pages, use as signatures or use in threads, make sure to mark them nofollow.

            Regards

            AnkurJ 1 Reply Last reply Reply Quote -1
            • AnkurJ
              AnkurJ @IM_Learner last edited by

              I am only talking about user profile pages not the links inside them. I have already taken care of them.

              Request others to please come up with suggestions.

              1 Reply Last reply Reply Quote 0
              • KristinaKledzik
                KristinaKledzik @AnkurJ last edited by

                Maybe you should noindex user profile pages until they've been active on the site long enough that a) you're sure they're not a spam bot and b) they're actually active users, with some interesting information to share on their user page?

                Like Shaliendra said, this is your call. It's really about your website goals, and what those user pages do for the site. Are you getting much organic traffic from user pages? If you aren't, it's probably easiest to just noindex all user pages. But if some profiles are bringing in organic traffic, I'd recommend that you add a noindex tag immediately, but remove it once someone has commented a few times.

                Does that help?

                1 Reply Last reply Reply Quote 1
                • 1 / 1
                • First post
                  Last post
                • Robot.txt file issue on wordpress site.
                  AlanMosley
                  AlanMosley
                  0
                  8
                  229

                • Question about robots.txt
                  spencerhjustice
                  spencerhjustice
                  0
                  3
                  125

                • Should I be disallowing my forum in the robots.txt file?
                  waleedkhalid
                  waleedkhalid
                  0
                  5
                  287

                • Need Suggestion for Canonical Page
                  Anup_More
                  Anup_More
                  0
                  3
                  75

                • Site Maps / Robots.txt etc
                  LockCity
                  LockCity
                  0
                  3
                  120

                • Disallow a spammed sub-page from robots.txt
                  zigojacko
                  zigojacko
                  0
                  3
                  475

                • Do I need canonical link on target page?
                  Dr-Pete
                  Dr-Pete
                  0
                  7
                  717

                • Robots.txt file
                  Greenman
                  Greenman
                  0
                  3
                  652

                Get started with Moz Pro!

                Unlock the power of advanced SEO tools and data-driven insights.

                Start my free trial
                Products
                • Moz Pro
                • Moz Local
                • Moz API
                • Moz Data
                • STAT
                • Product Updates
                Moz Solutions
                • SMB Solutions
                • Agency Solutions
                • Enterprise Solutions
                • Digital Marketers
                Free SEO Tools
                • Domain Authority Checker
                • Link Explorer
                • Keyword Explorer
                • Competitive Research
                • Brand Authority Checker
                • Local Citation Checker
                • MozBar Extension
                • MozCast
                Resources
                • Blog
                • SEO Learning Center
                • Help Hub
                • Beginner's Guide to SEO
                • How-to Guides
                • Moz Academy
                • API Docs
                About Moz
                • About
                • Team
                • Careers
                • Contact
                Why Moz
                • Case Studies
                • Testimonials
                Get Involved
                • Become an Affiliate
                • MozCon
                • Webinars
                • Practical Marketer Series
                • MozPod
                Connect with us

                Contact the Help team

                Join our newsletter
                Moz logo
                © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                • Accessibility
                • Terms of Use
                • Privacy