The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Search Engine Trends
    4. What is the appropriate Robot.txt to unblock if Google cannot get all the resources from my homepage?

    What is the appropriate Robot.txt to unblock if Google cannot get all the resources from my homepage?

    Search Engine Trends
    2 2 5.7k
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • gamesotd
      gamesotd last edited by

      Hello everyone. I did some research as to why my website has decreased in the Google search rankings recently. After reading this Yoast article I believe it's because the robot.txt files I have set up on my wordpress website. The following is a screen shot of the results of a fetch & render query of my webpage.Googlebot couldn't get all resources for this page. Here's a list:

      URL Type Reason
      http://fonts.googleapis.com/css?family=Open+Sans:400,600,700,800%7CPT+Sans:400,400italic,700,700italic%7COswald:400,300,700&subset=latin,latin-ext Style Sheet Denied by robots.txt
      http://www.kmollinslaw.com/wp-content/plugins/slick-contact-forms/css/admin.css?ver=3.9.1 Style Sheet Denied by robots.txt
      http://www.kmollinslaw.com/wp-content/plugins/contact-form-plugin/css/style.css?ver=3.9.1 Style Sheet Denied by robots.txt
      http://www.kmollinslaw.com/wp-content/plugins/hupso-share-buttons-for-twitter-facebook-google/style.css?ver=3.9.1 Style Sheet Denied by robots.txt
      http://www.kmollinslaw.com/wp-content/plugins/latest-post-accordian-slider/css/lpaccordion.css?ver=3.9.1 Style Sheet Denied by robots.txt
      http://www.kmollinslaw.com/wp-content/plugins/latest-post-accordian-slider/css/style.css?ver=3.9.1 Style Sheet Denied by robots.txt
      http://www.kmollinslaw.com/wp-content/plugins/revslider/rs-plugin/css/settings.css?rev=4.1.1&ver=3.9.1 Style Sheet Denied by robots.txt
      http://www.kmollinslaw.com/wp-content/plugins/revslider/rs-plugin/css/dynamic-captions.css?rev=4.1.1&ver=3.9.1 Style Sheet Denied by robots.txt
      http://www.kmollinslaw.com/wp-content/plugins/revslider/rs-plugin/css/static-captions.css?rev=4.1.1&ver=3.9.1 Style Sheet Denied by robots.txt
      http://www.kmollinslaw.com/wp-content/plugins/wp-email-capture/inc/css/wp-email-capture-styles.css?ver=1.0 Style Sheet Denied by robots.txt
      http://www.kmollinslaw.com/wp-content/themes/infographer/style.css?ver=3.9.1 Style Sheet Denied by robots.txt
      http://www.kmollinslaw.com/wp-content/themes/infographer/css/stylesheet.min.css?ver=3.9.1 Style Sheet Denied by robots.txt
      http://www.kmollinslaw.com/wp-content/themes/infographer/css/style_dynamic.php?ver=3.9.1 Style Sheet Denied by robots.txt
      http://www.kmollinslaw.com/wp-content/themes/infographer/css/custom_css.php?ver=3.9.1 Style Sheet Denied by robots.txt
      http://www.kmollinslaw.com/wp-content/plugins/convertable-contact-form-builder-analytics-and-lead-management-dashboard/assets/css/convertable.css?ver=3.9.1 Style Sheet Denied by robots.txt
      http://www.kmollinslaw.com/wp-content/plugins/google-maps-widget/css/gmw.css?ver=1.66 Style Sheet Denied by robots.txt
      http://www.kmollinslaw.com/wp-content/plugins/acurax-social-media-widget/style.css?ver=3.9.1 Style Sheet Denied by robots.txt
      http://www.kmollinslaw.com/wp-includes/js/swfobject.js?ver=2.2-20120417 Script Denied by robots.txt

      My current robot.txt settings are as follows;

      User-agent: *
      Disallow: /wp-admin/
      Disallow: /wp-includes/
      Disallow: */xmlrpc.php
      Disallow: */wp-*.php
      Disallow: */trackback/
      Disallow: *?wptheme=
      Disallow: *?comments=
      Disallow: *?replytocom
      Disallow: */comment-page-
      Disallow: *?s=
      Disallow: */wp-content/
      Allow: */wp-content/uploads/
      ```What to I need to allow/disallow to allow Google spiders to properly read my website?
      1 Reply Last reply Reply Quote 0
      • OlegKorneitchouk
        OlegKorneitchouk last edited by

        Remove the following line:

        Disallow: */wp-content/
        
        And add:
        Allow: /wp-includes/js/
        
        1 Reply Last reply Reply Quote 1
        • 1 / 1
        • First post
          Last post
        • Need only tens of pages to be indexed out of hundreds: Robots.txt is Okay for Google to proceed with?
          GastonRiera
          GastonRiera
          0
          2
          24

        • Is there a we to get Google to index our site quicker?
          JaneCopland
          JaneCopland
          0
          9
          157

        • How to you get a company details to appear on the right side of google searches? For example is you type dr martens in google , the search page shows links and then a snippet of the company on the right. Thank you.
          crackingmedia
          crackingmedia
          0
          3
          9.8k

        • Google cant read my robots.txt from past 10 days
          toxicpls
          toxicpls
          0
          5
          152

        • Google Unable to Access Robots.txt
          Modi
          Modi
          0
          4
          205

        • Getting Listed in Google Places
          MiriamEllis
          MiriamEllis
          0
          6
          396

        • Why has my homepage been replaced in Google by my Facebook page?
          BlueprintMarketing
          BlueprintMarketing
          0
          3
          912

        • Is it hurting my seo ranking if robots.txt is forbidden?
          Sebes
          Sebes
          0
          6
          1.3k

        Get started with Moz Pro!

        Unlock the power of advanced SEO tools and data-driven insights.

        Start my free trial
        Products
        • Moz Pro
        • Moz Local
        • Moz API
        • Moz Data
        • STAT
        • Product Updates
        Moz Solutions
        • SMB Solutions
        • Agency Solutions
        • Enterprise Solutions
        • Digital Marketers
        Free SEO Tools
        • Domain Authority Checker
        • Link Explorer
        • Keyword Explorer
        • Competitive Research
        • Brand Authority Checker
        • Local Citation Checker
        • MozBar Extension
        • MozCast
        Resources
        • Blog
        • SEO Learning Center
        • Help Hub
        • Beginner's Guide to SEO
        • How-to Guides
        • Moz Academy
        • API Docs
        About Moz
        • About
        • Team
        • Careers
        • Contact
        Why Moz
        • Case Studies
        • Testimonials
        Get Involved
        • Become an Affiliate
        • MozCon
        • Webinars
        • Practical Marketer Series
        • MozPod
        Connect with us

        Contact the Help team

        Join our newsletter
        Moz logo
        © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
        • Accessibility
        • Terms of Use
        • Privacy