The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. Help Crawl friendliness for large site

    Help Crawl friendliness for large site

    Technical SEO Issues
    4 2 144
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Gordian
      Gordian last edited by

      After watching Rand's video I am trying to think of the best way to make my large site more crawl friendly.

      Background

      I have a large site with over 100k product skus and so when you get to a particular page of products there are tons of different refinements and options that help you sort the products. Most of these are noindex followed, but I was wondering if I should be nofollowing the internal links as well in order to keep bots out of those pages and going to the pages that I want them to go too. Is this a good way to handle it?

      Also, does anyone have good recommendations of links to posts that deal with helping the crawl friendliness of a large site?

      Thanks!

      1 Reply Last reply Reply Quote 0
      • TakeshiYoung
        TakeshiYoung last edited by

        Nofollowing internal links is almost never a good idea. You're just wasting valuable link juice.

        Google actually just recently came out with a good guide for how to handle ecommerce navigation with lots of product options: http://googlewebmastercentral.blogspot.com/2014/02/faceted-navigation-best-and-5-of-worst.html

        Also, if you have a lot of categories in you store, try to show navigation that is only relevant to the section of the store the user is in. For example, if the user is in the Flowers section, don't show a ton of links for Cellphones.

        Gordian 1 Reply Last reply Reply Quote 2
        • Gordian
          Gordian @TakeshiYoung last edited by

          This post is deleted!
          TakeshiYoung 1 Reply Last reply Reply Quote 0
          • TakeshiYoung
            TakeshiYoung @Gordian last edited by

            Good point. If you don't want the filter pages crawled at all, it would be better to just block them via robots.txt. My preferred approach is to use query parameters for filters, and canonicaling the filtered pages back to the original, unfiltered page.

            Another approach is to use AJAX to dynamically filter the page. This takes more programming overhead, but won't result in tons of extra pages being crawled and potentially indexed.

            1 Reply Last reply Reply Quote 1
            • 1 / 1
            • First post
              Last post
            • Country and Language tags-Running an SEO audit on a site that definitely has more than one language, but nothing is pulling up. I don't quite understand href lang or how to go about it. HELP HELP!
              Mindgruver
              Mindgruver
              0
              2
              26

            • Please Help! Crawl & Site Errors - Will This Impact My SEO?
              KristinaKledzik
              KristinaKledzik
              0
              5
              78

            • Should you use robots.txt for pages within your site which do not have high quality content or are not contributing a great deal so when Google crawls your site the best performing content has a higher chance of being indexed?
              Jacksons_Fencing
              Jacksons_Fencing
              0
              5
              44

            • Help Setting Up 301 Redirects from Coldfusion Site to Wordpress Site.
              MarketHubb
              MarketHubb
              1
              3
              143

            • My site is not being regularly crawled?
              Everett
              Everett
              0
              5
              138

            • How Often is Site Crawled
              Matt-Williamson
              Matt-Williamson
              0
              5
              1.8k

            • Site Crawl
              Sean_Dawes
              Sean_Dawes
              0
              2
              414

            • Recently revamped site structure - now not even ranking for brand name, but lots of content - what happened? (Yup, the site has been crawled a few times since) Any ideas? Did I make a classic mistake? Any advise appreciated :)
              AlanBleiweiss
              AlanBleiweiss
              0
              9
              844

            Get started with Moz Pro!

            Unlock the power of advanced SEO tools and data-driven insights.

            Start my free trial
            Products
            • Moz Pro
            • Moz Local
            • Moz API
            • Moz Data
            • STAT
            • Product Updates
            Moz Solutions
            • SMB Solutions
            • Agency Solutions
            • Enterprise Solutions
            • Digital Marketers
            Free SEO Tools
            • Domain Authority Checker
            • Link Explorer
            • Keyword Explorer
            • Competitive Research
            • Brand Authority Checker
            • Local Citation Checker
            • MozBar Extension
            • MozCast
            Resources
            • Blog
            • SEO Learning Center
            • Help Hub
            • Beginner's Guide to SEO
            • How-to Guides
            • Moz Academy
            • API Docs
            About Moz
            • About
            • Team
            • Careers
            • Contact
            Why Moz
            • Case Studies
            • Testimonials
            Get Involved
            • Become an Affiliate
            • MozCon
            • Webinars
            • Practical Marketer Series
            • MozPod
            Connect with us

            Contact the Help team

            Join our newsletter
            Moz logo
            © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
            • Accessibility
            • Terms of Use
            • Privacy