Auto-loading content via AJAX - best practices

[paul.younghusband]

We have an ecommerce website and I'm looking at replacing the pagination on our category pages with functionality that auto-loads the products as the user scrolls. There are a number of big websites that do this - MyFonts and Kickstarter are two that spring to mind.

Obviously if we are loading the content in via AJAX then search engine spiders aren't going to be able to crawl our categories in the same way they can now. I'm wondering what the best way to get around this is.

Some ideas that spring to mind are:

• detect the user agent and if the visitor is a spider, show them the old-style pagination instead of the AJAX version

• make sure we submit an updated Google sitemap every day (I'm not sure if this a reasonable substitute for Google being able to properly crawl our site)

Are there any best practices surrounding this approach to pagination? Surely the bigger sites that do this must have had to deal with these issues?

Any advice would be much appreciated!

[stukerr]

Hi Paul,

Pagination is always a bit of a sticky area!

Firstly I certainly wouldn't do any user agent detection, you don't wanna get busted for cloaking when you aren't even up to anything that naughty.

A nice way i've seen this handled (for wordpress sites although the idea can work on any site) is with the wordpress infinite scroll plugin : http://wordpress.org/extend/plugins/infinite-scroll/

That basically leaves the site as it is for non-javascript web browsers (so with page 1, 2 3 etc.) but if you have js enabled (i.e. not a spider bot) it will keep scrolling the page. This functionality could I guess be changed to create a pagination effect.

Tie this is with some rel="prev" and rel="next" markeup (http://googlewebmastercentral.blogspot.co.uk/2011/09/pagination-with-relnext-and-relprev.html) and I think that is certainly one way to fix the problem.

Another way could be using similar markup for a 'View All' page : http://googlewebmastercentral.blogspot.co.uk/2011/09/view-all-in-search-results.html

Hope that helps!

Stuart