The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. Ajax Crawling | Blocked URLs Spike

    Ajax Crawling | Blocked URLs Spike

    Technical SEO Issues
    2 2 163
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • RocketZando
      RocketZando last edited by

      http://www.zando.co.za/women/shoes/  (for example)

      Hello,

      I'm concerned that WMT is reporting a large spike in blocked URLs - now reporting more blocked URLs  than good URLs.

      Our product recommendations get generated via an Ajax call and these autogenerated, unique, URLs are rendered in the /recommendations/ folder which sits in the root of our site: http://www.zando.co.za/recommendations/

      I can't see how I can prevent Google from calling the Ajax - I can only assume that's what's happening.This is what the code typically looks like:

      1 Reply Last reply Reply Quote 0
      • smarties954
        smarties954 last edited by

        If the blocked urls are all the /recommendation/?c=catalog... pages, then looks like google is reading following the url in your ajax code, since the return is not a full page, then probably google discards them.

        Have you tried to disallow the /recommandation/ folder in your robot.txt?

        Also, why do you use ajax to call related products? To speed up the page load?

        1 Reply Last reply Reply Quote 0
        • 1 / 1
        • First post
          Last post
        • Why Google crawl parameter URLs?
          Rajesh.Prajapati
          Rajesh.Prajapati
          0
          2
          86

        • WEBMASTER console: increase in the number of URLs we were blocked from crawling due to authorization permission errors.
          odmsoft
          odmsoft
          0
          21
          835

        • Blocked URL parameters can still be crawled and indexed by google?
          anthonydnelson
          anthonydnelson
          0
          6
          1.7k

        • Decrease in index URLs, increase in URLs blocked by robots and drop in traffic
          pugh
          pugh
          0
          6
          2.4k

        • Crawl reveals hundreds of urls with multiple urls in the url string
          irvingw
          irvingw
          0
          5
          455

        • Should we block URL param in Webmaster tools after URL migration?
          gmk1567
          gmk1567
          0
          2
          594

        • Blocking AJAX Content from being crawled
          BryceHoward
          BryceHoward
          0
          2
          733

        • Blocking Google from Crawling Parameters
          wissamdandan
          wissamdandan
          0
          7
          1.1k

        Get started with Moz Pro!

        Unlock the power of advanced SEO tools and data-driven insights.

        Start my free trial
        Products
        • Moz Pro
        • Moz Local
        • Moz API
        • Moz Data
        • STAT
        • Product Updates
        Moz Solutions
        • SMB Solutions
        • Agency Solutions
        • Enterprise Solutions
        • Digital Marketers
        Free SEO Tools
        • Domain Authority Checker
        • Link Explorer
        • Keyword Explorer
        • Competitive Research
        • Brand Authority Checker
        • Local Citation Checker
        • MozBar Extension
        • MozCast
        Resources
        • Blog
        • SEO Learning Center
        • Help Hub
        • Beginner's Guide to SEO
        • How-to Guides
        • Moz Academy
        • API Docs
        About Moz
        • About
        • Team
        • Careers
        • Contact
        Why Moz
        • Case Studies
        • Testimonials
        Get Involved
        • Become an Affiliate
        • MozCon
        • Webinars
        • Practical Marketer Series
        • MozPod
        Connect with us

        Contact the Help team

        Join our newsletter
        Moz logo
        © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
        • Accessibility
        • Terms of Use
        • Privacy