The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. Whole site blocked by robots in webmaster tools

    Whole site blocked by robots in webmaster tools

    Intermediate & Advanced SEO
    2 2 109
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • rrogers
      rrogers last edited by

      My URL is: www.wheretobuybeauty.com.auThis new site has been re-crawled over last 2 weeks, and in webmaster tools index status the following is displayed:Indexed 50,000 pagesblocked by robots 69,000Search query 'site:wheretobuybeauty.com.au' returns 55,000 pagesHowever, all pages in the site do appear to be blocked and over the 2 weeks, the google search query site traffic declined from significant to zero (proving this is in fact the case ).This is a Linux php site and has the following: 55,000 URLs in sitemap.xml submitted successfully to webmaster toolsrobots.txt file existed but did not have any entries to allow or disallow URLs - today I have removed robots.txt file completely URL re-direction within Linux .htaccess file - there are many rows within this complex set of re-directions.  Developer has double checked this file and found that it is valid.I have read everything that google and other sources have on this topic and this does not help.  Also checked webmaster crawl errors, crawl stats, malware and there is no problem there related to this issue.Is this a duplicate content issue - this is a price comparison site where approx half the products have duplicate product descriptions - duplicated because they are obtained from the suppliers through an XML data file.  The suppliers have the descriptions from the files in their own sites.Help!!

      1 Reply Last reply Reply Quote 0
      • AdamThompson
        AdamThompson last edited by

        After reviewing your site, I can see that:

        • Your site is well indexed by Google.
        • Your site has very thin content.
        • Your site is mostly duplicate content.

        I think you'll need a content strategy to achieve and maintain significant rankings. You'll probably need to noindex your entire site except for a few pages that you add significant and valuable content to. (Unless you have the resources to write good content for your entire site.)

        1 Reply Last reply Reply Quote 0
        • 1 / 1
        • First post
          Last post
        • Links to my site still showing in Webmaster Tools from a non-existent site
          EGOL
          EGOL
          0
          3
          86

        • Robots.txt - Do I block Bots from crawling the non-www version if I use www.site.com ?
          PatrickDelehanty
          PatrickDelehanty
          0
          2
          479

        • Do Google webmaster tool and other backlinks analysis tool ignore the disavow data ?
          GPainter
          GPainter
          0
          6
          320

        • How to make Google index your site? (Blocked with robots.txt for a long time)
          SanjidaKazi
          SanjidaKazi
          0
          3
          128

        • When migrating website platforms but keeping the domain name how best do we add the new site to google webmaster tools? Best redirect practices?
          WNL
          WNL
          0
          6
          462

        • Robots.txt Blocked Most Site URLs Because of Canonical
          0
          1
          117

        • Disavow tool removed all our links from webmaster tools
          KeriMorgret
          KeriMorgret
          0
          11
          521

        • Should I robots block site directories with primarily duplicate content?
          mdoegel
          mdoegel
          0
          2
          912

        Get started with Moz Pro!

        Unlock the power of advanced SEO tools and data-driven insights.

        Start my free trial
        Products
        • Moz Pro
        • Moz Local
        • Moz API
        • Moz Data
        • STAT
        • Product Updates
        Moz Solutions
        • SMB Solutions
        • Agency Solutions
        • Enterprise Solutions
        • Digital Marketers
        Free SEO Tools
        • Domain Authority Checker
        • Link Explorer
        • Keyword Explorer
        • Competitive Research
        • Brand Authority Checker
        • Local Citation Checker
        • MozBar Extension
        • MozCast
        Resources
        • Blog
        • SEO Learning Center
        • Help Hub
        • Beginner's Guide to SEO
        • How-to Guides
        • Moz Academy
        • API Docs
        About Moz
        • About
        • Team
        • Careers
        • Contact
        Why Moz
        • Case Studies
        • Testimonials
        Get Involved
        • Become an Affiliate
        • MozCon
        • Webinars
        • Practical Marketer Series
        • MozPod
        Connect with us

        Contact the Help team

        Join our newsletter
        Moz logo
        © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
        • Accessibility
        • Terms of Use
        • Privacy