The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. On-Page / Site Optimization
    4. How can i block the below URLs

    How can i block the below URLs

    On-Page / Site Optimization
    4 4 621
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • MasonBaker
      MasonBaker last edited by

      Google indexed plugins pages for my website. Please check below. How can stop them to be indexed on google.?

      http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/glass/

      http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/borderlesslight3d/

      http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/defaultskin/


      My robots.txt file is -

      User-agent: *
      Disallow: /wp-admin/
      
      1 Reply Last reply Reply Quote 0
      • GPainter
        GPainter last edited by

        Take a look over here - http://www.robotstxt.org/robotstxt.html and we can't forget the Moz version - http://moz.com/learn/seo/robotstxt

        Alternatively you can just add meta noindex on the pages with with the added bonus of letting link juice flow better as well as being a bit more stern to robots ( recommend the noindex tag!)

        1 Reply Last reply Reply Quote 1
        • TomRayner
          TomRayner last edited by

          It's also worth remembering that blocking a URL in a robots.txt file does not automatically mean that the URL will be deindexed.  The robots.txt file will prevent robots, such as the Googlebot, from seeing or accessing the file and that's all.  Now obviously, if a robot repeatedly tries to access a file and gets denied it will eventually stop trying to, which is what leads to the file being deindexed (the same principle applies to 404 and 410 errors).

          Therefore, if you want a quicker and more definite deindexing solution, you should use the explicit noindex robot command, as recommended above.  This will tell any visiting robot to not index it straight away, which will reduce the number of revisits and drop the page from the index faster.

          1 Reply Last reply Reply Quote 1
          • ColeLusby
            ColeLusby last edited by

            robots.txt:

            Disallow:

            /wp-admin/

            /wp-content/plugins/

            Then, deindex them as mentioned in the first response.

            1 Reply Last reply Reply Quote 1
            • 1 / 1
            • First post
              Last post
            • Duplicate URL errors when URL's are unique
              DirkC
              DirkC
              0
              2
              96

            • How can I redirect anything after the article url to main article?
              mediawyse
              mediawyse
              1
              4
              151

            • Can't get my preferred URL, how much does it matter?
              isaac663
              isaac663
              0
              4
              124

            • Can Sitemap Be Used to Manage Canonical URLs?
              ThompsonPaul
              ThompsonPaul
              0
              2
              167

            • Recommendation: Add a canonical URL tag referencing this URL to the header of the page.
              ThompsonPaul
              ThompsonPaul
              0
              7
              1.5k

            • Can bad text URLs hurt pages?
              loopyal
              loopyal
              0
              5
              439

            • Can you optimize for 2 keywords per URL?
              scanlin
              scanlin
              0
              3
              963

            • Can someone please help me identify where all these URLS to my homepage are coming from?
              KeriMorgret
              KeriMorgret
              0
              7
              1.1k

            Get started with Moz Pro!

            Unlock the power of advanced SEO tools and data-driven insights.

            Start my free trial
            Products
            • Moz Pro
            • Moz Local
            • Moz API
            • Moz Data
            • STAT
            • Product Updates
            Moz Solutions
            • SMB Solutions
            • Agency Solutions
            • Enterprise Solutions
            • Digital Marketers
            Free SEO Tools
            • Domain Authority Checker
            • Link Explorer
            • Keyword Explorer
            • Competitive Research
            • Brand Authority Checker
            • Local Citation Checker
            • MozBar Extension
            • MozCast
            Resources
            • Blog
            • SEO Learning Center
            • Help Hub
            • Beginner's Guide to SEO
            • How-to Guides
            • Moz Academy
            • API Docs
            About Moz
            • About
            • Team
            • Careers
            • Contact
            Why Moz
            • Case Studies
            • Testimonials
            Get Involved
            • Become an Affiliate
            • MozCon
            • Webinars
            • Practical Marketer Series
            • MozPod
            Connect with us

            Contact the Help team

            Join our newsletter
            Moz logo
            © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
            • Accessibility
            • Terms of Use
            • Privacy