The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. Can't find mistake in robots.txt

    Can't find mistake in robots.txt

    Technical SEO Issues
    3 3 338
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • accessKellyOCG
      accessKellyOCG last edited by

      Hi all,

      we recently filled our robots.txt file to prevent some directories from crawling.

      Looks like:

      User-agent: *
      Disallow: /Views/
      Disallow: /login/
      Disallow: /routing/
      Disallow: /Profiler/
      Disallow: /LILLYPROFILER/
      Disallow: /EventRweKompaktProfiler/
      Disallow: /AccessIntProfiler/
      Disallow: /KellyIntProfiler/
      Disallow: /lilly/
      

      now, as Google Webmaster Tools hasn't updated our robots.txt yet,

      I checked our robots.txt in some ckeckers.
      

      They tell me that the User agent: * contains an error.

      **Example:**
      

      **Line 1: Syntax error! Expected <field>:</field> <value></value>

      1: User-agent: ***

      **`I checked other robots.txt written the same way --> they work,`** 
      

      accordign to the checkers...

      **`Where the .... is the mistake???`**
      ```
      1 Reply Last reply Reply Quote 0
      • SebastianCowie
        SebastianCowie last edited by

        Hi!

        The robots.txt is fine. Some checkers return wildcards as an error, as not all crawlers supported "*". I wouldn't worry about it 🙂

        1 Reply Last reply Reply Quote 0
        • Debdulal
          Debdulal last edited by

          _Hi, _

          _Just wondering .. Did you save the txt file in ANSI format? Sometimes, people mistakenly save it different format and this is where the problem creeps in. _

          1 Reply Last reply Reply Quote 2
          • 1 / 1
          • First post
            Last post
          • Crawl solutions for landing pages that don't contain a robots.txt file?
            Nomader
            Nomader
            1
            10
            353

          • Site hacked, but can't find the code
            LinkWheelOldSchool
            LinkWheelOldSchool
            0
            4
            134

          • Another client copies everything to blogspot. Is that what keeps her site from ranking? Or what? Appears to be a penalty somewhere but can't find it.
            katandmouse
            katandmouse
            0
            8
            209

          • Google insists robots.txt is blocking... but it isn't.
            Christy-Correll
            Christy-Correll
            0
            4
            523

          • Error: Missing Meta Description Tag on pages I can't find in order to correct
            gfiedel
            gfiedel
            0
            10
            466

          • 404 error - but I can't find any broken links on the referrer pages
            IainReloadMedia
            IainReloadMedia
            0
            12
            2.0k

          • I have 404 errors but can't find where these links are?
            JackMurphy
            JackMurphy
            0
            2
            280

          • How do I use the Robots.txt "disallow" command properly for folders I don't want indexed?
            portalseo
            portalseo
            0
            5
            1.9k

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy