The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. Do i have my robots.txt file set up properly

    Do i have my robots.txt file set up properly

    Technical SEO Issues
    4 2 319
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • ClaireH-184886
      ClaireH-184886 last edited by

      Hi, just doing some seo on my site and i am not sure if i have my robots file set correctly. i use joomla and my website is www.in2town.co.uk.

      here is my robots file, does this look correct to you

      User-agent: *
      Disallow: /administrator/
      Disallow: /cache/
      Disallow: /components/
      Disallow: /includes/
      Disallow: /installation/
      Disallow: /language/
      Disallow: /libraries/
      Disallow: /media/
      Disallow: /modules/
      Disallow: /plugins/
      Disallow: /templates/
      Disallow: /tmp/
      Disallow: /xmlrpc/

      many thanks

      1 Reply Last reply Reply Quote 1
      • SpiderWebDesigns
        SpiderWebDesigns last edited by

        Yes, this does look good. However, usually the robots.txt will define a location of a sitemap. Not absolutely needed, but good to know.

        Here is an example of one of our client's wordpress sites.

        User-agent: *
        Disallow: /wp-admin
        Disallow: /another-post
        Disallow: /dolor-and-the-sit-amet/
        Disallow: /hello-world-2-2/
        Disallow: /second-page-post/
        Disallow: /hello-world-2-3/
        Disallow: /tag/
        Disallow: /events/
        Disallow: /wp-content/
        Sitemap: http://backcountrysnow.com/sitemap.xml.gz
        
        
        ClaireH-184886 2 Replies Last reply Reply Quote 1
        • ClaireH-184886
          ClaireH-184886 @SpiderWebDesigns last edited by

          thanks for this. been having for a long time trouble with a site map. the reason is, i use joomla 1.5 and i am not sure the best way to have it set or which is the best tool to use.

          my articles change all the time and not sure how many of the articles i should have in the site map or to have just the sections.

          on an old site i had all the articles, well up to 2,000 and that gain me a lot of traffic but with the new site i took that down

          1 Reply Last reply Reply Quote 1
          • ClaireH-184886
            ClaireH-184886 @SpiderWebDesigns last edited by

            thanks for this, i will add a sitemap now

            1 Reply Last reply Reply Quote 0
            • 1 / 1
            • First post
              Last post
            • Blocking subdomains with Robots.txt file
              PaulM01
              PaulM01
              0
              3
              641

            • One robots.txt file for multiple sites?
              RyanPurkey
              RyanPurkey
              0
              4
              1.1k

            • Sub Domains and Robot.txt files...
              Vizergy
              Vizergy
              0
              3
              160

            • Is there any value in having a blank robots.txt file?
              LiamVictor
              LiamVictor
              0
              6
              3.1k

            • Meta Robots Noindex and Robots.txt File
              Devanur-Rafi
              Devanur-Rafi
              0
              2
              125

            • Question about construction of our sitemap URL in robots.txt file
              danatanseo
              danatanseo
              0
              16
              1.6k

            • Is my robots.txt file working?
              Resultify
              Resultify
              0
              5
              1.1k

            • Does Bing ignore robots txt files?
              Nightwing
              Nightwing
              0
              3
              2.8k

            Get started with Moz Pro!

            Unlock the power of advanced SEO tools and data-driven insights.

            Start my free trial
            Products
            • Moz Pro
            • Moz Local
            • Moz API
            • Moz Data
            • STAT
            • Product Updates
            Moz Solutions
            • SMB Solutions
            • Agency Solutions
            • Enterprise Solutions
            • Digital Marketers
            Free SEO Tools
            • Domain Authority Checker
            • Link Explorer
            • Keyword Explorer
            • Competitive Research
            • Brand Authority Checker
            • Local Citation Checker
            • MozBar Extension
            • MozCast
            Resources
            • Blog
            • SEO Learning Center
            • Help Hub
            • Beginner's Guide to SEO
            • How-to Guides
            • Moz Academy
            • API Docs
            About Moz
            • About
            • Team
            • Careers
            • Contact
            Why Moz
            • Case Studies
            • Testimonials
            Get Involved
            • Become an Affiliate
            • MozCon
            • Webinars
            • Practical Marketer Series
            • MozPod
            Connect with us

            Contact the Help team

            Join our newsletter
            Moz logo
            © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
            • Accessibility
            • Terms of Use
            • Privacy