The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. Site not being Indexed that fast anymore, Is something wrong with this Robots.txt

    Site not being Indexed that fast anymore, Is something wrong with this Robots.txt

    Technical SEO Issues
    4 4 828
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • ideas123
      ideas123 last edited by

      My wordpress site's robots.txt used to be this:

      User-agent: *

      Disallow: Sitemap: http://www.domainame.com/sitemap.xml.gz I also have all in one SEO installed and other than posts, tags are also index,follow on my site.

      My new posts used to appear on google in seconds after publishing. I changed the robots.txt to following and now post indexing takes hours.

      Is there something wrong with this robots.txt? User-agent: *

      Disallow: /cgi-bin

      Disallow: /wp-admin

      Disallow: /wp-includes

      Disallow: /wp-content/plugins

      Disallow: /wp-content/cache

      Disallow: /wp-content/themes

      Disallow: /wp-login.php

      Disallow: /wp-login.php

      Disallow: /trackback

      Disallow: /feed

      Disallow: /comments

      Disallow: /author

      Disallow: /category

      Disallow: */trackback

      Disallow: */feed

      Disallow: */comments

      Disallow: /login/

      Disallow: /wget/

      Disallow: /httpd/

      Disallow: /*.php$

      Disallow: /?

      Disallow: /*.js$

      Disallow: /*.inc$

      Disallow: /*.css$

      Disallow: /*.gz$

      Disallow: /*.wmv$

      Disallow: /*.cgi$

      Disallow: /*.xhtml$

      Disallow: /?

      Disallow: /*?Allow: /wp-content/uploads

      User-agent: TechnoratiBot/8.1

      Disallow:

      ia_archiverUser-agent: ia_archiver

      Disallow: /

      disable duggmirror

      User-agent: duggmirror

      Disallow: /

      allow google image bot to search all imagesUser-agent: Googlebot-Image

      Disallow: /wp-includes/

      Allow: /*

      # allow adsense bot on entire siteUser-agent: Mediapartners-Google*

      Disallow:

      Allow: /*

      Sitemap: http://www.domainname.com/sitemap.xml.gz

      1 Reply Last reply Reply Quote 0
      • irvingw
        irvingw last edited by

        Why don't you revert back to the original robots.txt and determine for certain that the problem is with this file?

        1 Reply Last reply Reply Quote 0
        • Ursacescu
          Ursacescu last edited by

          Hello,

          Robots.txt, allow or disallow access to certain files or folders. He can not delay or slow down access. I do not think the problem is the robots.txt

          Radu

          1 Reply Last reply Reply Quote 0
          • oznappies
            oznappies last edited by

            I am not sure why you are setting disallow of file types.  Google would not index wmv or js etc anyway as it cannot parse that type of file for data. If you want to coax  google into indexing your site submit a sitemap in webmaster tools. You could also set NoFollow on the anchors for the pages you want to exclude and keep robots.txt cleaner by just including top  level subdirectories such as admin etc. There just seems to be a lot of directories in there that do not relate to actual pages, and google is only concerned with renderable pages.

            1 Reply Last reply Reply Quote 0
            • 1 / 1
            • First post
              Last post
            • Should you use robots.txt for pages within your site which do not have high quality content or are not contributing a great deal so when Google crawls your site the best performing content has a higher chance of being indexed?
              Jacksons_Fencing
              Jacksons_Fencing
              0
              5
              44

            • What's wrong with this robots.txt
              BlueprintMarketing
              BlueprintMarketing
              1
              19
              493

            • Google indexing despite robots.txt block
              john4math
              john4math
              0
              13
              654

            • Google is indexing blocked content in robots.txt
              bjs2010
              bjs2010
              0
              5
              147

            • I am trying to block robots from indexing parts of my site..
              Smurkcreative
              Smurkcreative
              0
              3
              374

            • Robots.txt blocking site or not?
              RyanKent
              RyanKent
              0
              2
              445

            • Mobile site: robots.txt best practices
              AlanMosley
              AlanMosley
              0
              2
              542

            • Need Help With Robots.txt on Magento eCommerce Site
              Francisco_Meza
              Francisco_Meza
              0
              4
              5.3k

            Get started with Moz Pro!

            Unlock the power of advanced SEO tools and data-driven insights.

            Start my free trial
            Products
            • Moz Pro
            • Moz Local
            • Moz API
            • Moz Data
            • STAT
            • Product Updates
            Moz Solutions
            • SMB Solutions
            • Agency Solutions
            • Enterprise Solutions
            • Digital Marketers
            Free SEO Tools
            • Domain Authority Checker
            • Link Explorer
            • Keyword Explorer
            • Competitive Research
            • Brand Authority Checker
            • Local Citation Checker
            • MozBar Extension
            • MozCast
            Resources
            • Blog
            • SEO Learning Center
            • Help Hub
            • Beginner's Guide to SEO
            • How-to Guides
            • Moz Academy
            • API Docs
            About Moz
            • About
            • Team
            • Careers
            • Contact
            Why Moz
            • Case Studies
            • Testimonials
            Get Involved
            • Become an Affiliate
            • MozCon
            • Webinars
            • Practical Marketer Series
            • MozPod
            Connect with us

            Contact the Help team

            Join our newsletter
            Moz logo
            © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
            • Accessibility
            • Terms of Use
            • Privacy