The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Moz Tools
    4. Large site with content silo's - best practice for deep indexing silo content

    Large site with content silo's - best practice for deep indexing silo content

    Moz Tools
    1 1 159
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • DrewProZ
      DrewProZ last edited by

      Thanks in advance for any advice/links/discussion.  This honestly might be a scenario where we need to do some A/B testing.

      We have a massive (5 Million) content silo that is the basis for our long tail search strategy.  Organic search traffic hits our individual "product" pages and we've divided our silo with a parent category & then secondarily with a field (so we can cross link to other content silo's using the same parent/field categorizations).

      We don't anticipate, nor expect to have top level category pages receive organic traffic - most people are searching for the individual/specific product (long tail).  We're not trying to rank or get traffic for searches of all products in "category X" and others are competing and spending a lot in that area (head).

      The intent/purpose of the site structure/taxonomy is to more easily enable bots/crawlers to get deeper into our content silos.  We've built the page for humans, but included link structure/taxonomy to assist crawlers.

      So here's my question on best practices.  How to handle categories with 1,000+ pages/pagination.  With our most popular product categories, there might be 100,000's products in one category.  My top level hub page for a category looks like www.mysite/categoryA and the page build is showing 50 products and then pagination from 1-1000+.

      Currently we're using rel=next for pagination and for pages like www.mysite/categoryA?page=6 we make it reference itself as canonical (not the first/top page www.mysite/categoryA).

      Our goal is deep crawl/indexation of our silo.  I use ScreamingFrog and SEOMoz campaign crawl to sample (site takes a week+ to fully crawl) and with each of these tools it "looks" like crawlers have gotten a bit "bogged down" with large categories with tons of pagination.  For example rather than crawl multiple categories or fields to get to multiple product pages, some bots will hit all 1,000 (rel=next) pages of a single category. I don't want to waste crawl budget going through 1,000 pages of a single category, versus discovering/crawling more categories.

      I can't seem to find a consensus as to how to approach the issue.  I can't have a page that lists "all" - there's just too much, so we're going to need pagination.  I'm not worried about category pagination pages cannibalizing traffic as I don't expect any (should I make pages 2-1,000) noindex and canonically reference the main/first page in the category?).  Should I worry about crawlers going deep in pagination among 1 category versus getting to more top level categories?

      Thanks!

      1 Reply Last reply Reply Quote 1
      • 1 / 1
      • First post
        Last post
      • Can we discuss Moz's Forecast? It's been over 100 degrees for a month.
        TaylorRHawkins
        TaylorRHawkins
        1
        6
        89

      • Duplicate content flagged by Moz that's not actually duplicate content at all
        Caro-O
        Caro-O
        1
        4
        89

      • Difference between Open site Explorer's Root Domain and Basic SERP Report's Linking Root Domain?
        ErinMcCaul
        ErinMcCaul
        0
        2
        114

      • Now our site is ready what's next
        Andy.Drinkwater
        Andy.Drinkwater
        0
        2
        84

      • What is the best ranking checker solution for 100's of sites
        csfarnsworth
        csfarnsworth
        0
        8
        675

      • Where do these URL's come from?! (Indexation issues)
        AlanMosley
        AlanMosley
        0
        6
        1.0k

      • Best Practices for having Social Profiles indexed
        IoanSaid
        IoanSaid
        0
        4
        1.2k

      • Which would you chose? Link on PA56 with 88 OBL's and 80 IBL's or a link on a PA75 with 225 OBL's & 40 IBL (Same Domain)
        Dan-Petrovic
        Dan-Petrovic
        0
        5
        988

      Get started with Moz Pro!

      Unlock the power of advanced SEO tools and data-driven insights.

      Start my free trial
      Products
      • Moz Pro
      • Moz Local
      • Moz API
      • Moz Data
      • STAT
      • Product Updates
      Moz Solutions
      • SMB Solutions
      • Agency Solutions
      • Enterprise Solutions
      • Digital Marketers
      Free SEO Tools
      • Domain Authority Checker
      • Link Explorer
      • Keyword Explorer
      • Competitive Research
      • Brand Authority Checker
      • Local Citation Checker
      • MozBar Extension
      • MozCast
      Resources
      • Blog
      • SEO Learning Center
      • Help Hub
      • Beginner's Guide to SEO
      • How-to Guides
      • Moz Academy
      • API Docs
      About Moz
      • About
      • Team
      • Careers
      • Contact
      Why Moz
      • Case Studies
      • Testimonials
      Get Involved
      • Become an Affiliate
      • MozCon
      • Webinars
      • Practical Marketer Series
      • MozPod
      Connect with us

      Contact the Help team

      Join our newsletter
      Moz logo
      © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
      • Accessibility
      • Terms of Use
      • Privacy