The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. About robots.txt for resolve Duplicate content

    About robots.txt for resolve Duplicate content

    Intermediate & Advanced SEO
    4 3 449
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • magician
      magician last edited by

      I have a trouble with Duplicate content and title, i try to many way to resolve them but because of the web code so i am still in problem. I decide to use robots.txt to block contents that are duplicate.

      The first Question: How do i use command in robots.txt to block all of URL like this:

      http://vietnamfoodtour.com/foodcourses/Cooking-School/
      http://vietnamfoodtour.com/foodcourses/Cooking-Class/ .......

      User-agent: * Disallow: /foodcourses ( Is that right? )

      And the parameter URL: h
      ttp://vietnamfoodtour.com/?mod=vietnamfood&page=2
      http://vietnamfoodtour.com/?mod=vietnamfood&page=3
      http://vietnamfoodtour.com/?mod=vietnamfood&page=4

      User-agent: * Disallow: /?mod=vietnamfood ( Is that right? i have folder contain module, could i use: disallow:/module/*)

      The 2nd question is: Which is the priority " robots.txt" or " meta robot"? If i use robots.txt to block URL, but in that URL my meta robot is "index, follow"

      1 Reply Last reply Reply Quote 0
      • SoftzSolutions
        SoftzSolutions last edited by

        Yeah, you are right, the Disallow tag is correct. However, if you are not feeling confident enough, you can try it in Google Webmaster Tools. Go to “Health” > “Blocked URLs”. Then make changes in the robots.txt file and test the pages.

        _And there is no such thing as priority. If you blocked a page via robots and use "index, follow" as Meta tag, it will get blocked and also if you do not block the page via robots.txt but use "noindex, nofollow", it will still get blocked. _

        1 Reply Last reply Reply Quote 0
        • gmk1567
          gmk1567 last edited by

          I would suggest to use "noindex" for completely dropping the unwanted URLs from the index of search engines.

          Check Google guidelines at: http://support.google.com/webmasters/bin/answer.py?hl=en&answer=93708

          magician 1 Reply Last reply Reply Quote 0
          • magician
            magician @gmk1567 last edited by

            thanks both of you!

            Because of i can configure the meta robot tag so i must use robots.txt :((

            I wil check in Google webmaster tool!

            1 Reply Last reply Reply Quote 0
            • 1 / 1
            • First post
              Last post
            • Duplicate content - how to diagnose duplicate content from another domain before publishing pages?
              Chemometec
              Chemometec
              0
              7
              141

            • "noindex, follow" or "robots.txt" for thin content pages
              khi5
              khi5
              0
              6
              1.5k

            • Avoiding Duplicate Content with Used Car Listings Database: Robots.txt vs Noindex vs Hash URLs (Help!)
              Everett
              Everett
              0
              10
              2.0k

            • Blocking poor quality content areas with robots.txt
              KaneJamison
              KaneJamison
              0
              4
              128

            • Could you use a robots.txt file to disalow a duplicate content page from being crawled?
              KyleChamp
              KyleChamp
              0
              11
              1.3k

            • 10,000 New Pages of New Content - Should I Block in Robots.txt?
              EGOL
              EGOL
              0
              2
              709

            • Should I robots block site directories with primarily duplicate content?
              mdoegel
              mdoegel
              0
              2
              912

            • Robots.txt: Link Juice vs. Crawl Budget vs. Content 'Depth'
              rishil
              rishil
              0
              5
              1.5k

            Get started with Moz Pro!

            Unlock the power of advanced SEO tools and data-driven insights.

            Start my free trial
            Products
            • Moz Pro
            • Moz Local
            • Moz API
            • Moz Data
            • STAT
            • Product Updates
            Moz Solutions
            • SMB Solutions
            • Agency Solutions
            • Enterprise Solutions
            • Digital Marketers
            Free SEO Tools
            • Domain Authority Checker
            • Link Explorer
            • Keyword Explorer
            • Competitive Research
            • Brand Authority Checker
            • Local Citation Checker
            • MozBar Extension
            • MozCast
            Resources
            • Blog
            • SEO Learning Center
            • Help Hub
            • Beginner's Guide to SEO
            • How-to Guides
            • Moz Academy
            • API Docs
            About Moz
            • About
            • Team
            • Careers
            • Contact
            Why Moz
            • Case Studies
            • Testimonials
            Get Involved
            • Become an Affiliate
            • MozCon
            • Webinars
            • Practical Marketer Series
            • MozPod
            Connect with us

            Contact the Help team

            Join our newsletter
            Moz logo
            © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
            • Accessibility
            • Terms of Use
            • Privacy