The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. White Hat / Black Hat SEO
    4. Robots.txt file in Shopify - Collection and Product Page Crawling Issue

    Robots.txt file in Shopify - Collection and Product Page Crawling Issue

    White Hat / Black Hat SEO
    4 2 3.5k
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • HuptechWebseo
      HuptechWebseo last edited by

      Hi, 
      I am working on one big eCommerce store which have more then 1000 Product. we just moved platform WP to Shopify getting noindex issue. 
      when i check robots.txt i found below code which is very confusing for me.
      
      **I am not getting meaning of below tags.**
      
      • Disallow: /collections/+
      • Disallow: /collections/%2B
      • Disallow: /collections/%2b
      • Disallow: /blogs/+
      • Disallow: /blogs/%2B
      • Disallow: /blogs/%2b
      I can understand that my robots.txt disallows SEs to crawling and indexing my all product pages. ( collection/*+* ) Is this the query which is affecting the indexing product pages?
      

      Please explain me how this robots.txt work in shopify and once my page crawl and index by google.com then what is use of Disallow:

      Thanks.

      1 Reply Last reply Reply Quote 0
      • slatronica
        slatronica last edited by

        While you may not want context indexed, it's still valuable to be crawled and access your most important content like products.

        If you are blocking your /collections pages, Google will not be able to see that page's meta robots set to noindex, causing an issue for you. You may consider allowing robots to crawl your /collections pages but noindex them if they are low value or duplicative.

        1 Reply Last reply Reply Quote 0
        • HuptechWebseo
          HuptechWebseo last edited by

          Thank you for replying,

          But, our main issue is that we have already crawled all collection pages but the product pages haven't crawled yet. Now we don't figure out that whether it's robots.txt issue or other crawling issue?

          For example: "www.abc.com/collection/" page is crawled but "www.abc.com/collection/product1/" page hasn't crawled.

          Please reply me some tips here.

          1 Reply Last reply Reply Quote 0
          • slatronica
            slatronica last edited by

            Make sure products are in your sitemap and it has been re-submitted. You can also submit your products to request indexing for them in Google Search Console.

            1 Reply Last reply Reply Quote 0
            • 1 / 1
            • First post
              Last post
            • Crawl anamoly issue on Search Console
              david-johns-sheetlabels
              david-johns-sheetlabels
              0
              2
              46

            • How does Google handle product detail page links hiden in a <noscript>tag?</noscript>
              0
              2
              174

            • Should we remove our "index" pages (alphabetical link list to all of the products on the site)?
              ChrisRoberts-MTI
              ChrisRoberts-MTI
              1
              3
              102

            • Does Google crawl and index dynamic pages?
              KevinBudzynski
              KevinBudzynski
              0
              3
              8.9k

            • Does Lazy Loading Create Indexing Issues of products?
              Everett
              Everett
              0
              8
              2.0k

            • Help with E-Commerce Product Pages
              YNWA
              YNWA
              0
              5
              167

            • Page not being indexed or crawled and no idea why!
              CSawatzky
              CSawatzky
              0
              7
              1.0k

            • Pages For Products That Don't Exist Yet?
              94501
              94501
              0
              5
              935

            Get started with Moz Pro!

            Unlock the power of advanced SEO tools and data-driven insights.

            Start my free trial
            Products
            • Moz Pro
            • Moz Local
            • Moz API
            • Moz Data
            • STAT
            • Product Updates
            Moz Solutions
            • SMB Solutions
            • Agency Solutions
            • Enterprise Solutions
            • Digital Marketers
            Free SEO Tools
            • Domain Authority Checker
            • Link Explorer
            • Keyword Explorer
            • Competitive Research
            • Brand Authority Checker
            • Local Citation Checker
            • MozBar Extension
            • MozCast
            Resources
            • Blog
            • SEO Learning Center
            • Help Hub
            • Beginner's Guide to SEO
            • How-to Guides
            • Moz Academy
            • API Docs
            About Moz
            • About
            • Team
            • Careers
            • Contact
            Why Moz
            • Case Studies
            • Testimonials
            Get Involved
            • Become an Affiliate
            • MozCon
            • Webinars
            • Practical Marketer Series
            • MozPod
            Connect with us

            Contact the Help team

            Join our newsletter
            Moz logo
            © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
            • Accessibility
            • Terms of Use
            • Privacy