Why this page doesn't get indexed?

[soulnafein]

Hi, I've just taken over development and SEO for a site and we're having difficulty getting some key pages indexed on our site. They are two clicks away from the homepage, but still not getting indexed. They are recently created pages, with unique content on. The architecture looks like this:Homepage >> Car page >> Engine specific pageWhenever we add a new car, we link to its 'Car page' and it gets indexed very quickly. However the 'Engine pages' for that car don't get indexed, even after a couple of weeks. An example of one of these index pages are - http://www.carbuzz.co.uk/car-reviews/Volkswagen/Beetle-New/2.0-TSISo, things we've checked - 1. Yes, it's not blocked by robots.txt2. Yes, it's in the sitemap (http://www.carbuzz.co.uk/sitemap.xml)3. Yes, it's viewable to search spiders (e.g. the link is present in the html source)This page doesn't have a huge amount of unique content. We're a review aggregator, but it still does have some. Any suggestions as to why it isn't indexed?Thanks, David

[TobiasM]

Hi David,

When you first load the "Car Page"  http://www.carbuzz.co.uk/car-chooser only the first few cars show. THe rest are only accessible when you scroll down the page which then activates some AJAX (i think) which loads more cars i the listing.

I suspect this has something to do with is. The bots cant access the "more" listings.

You could use an HTML sitemap on the site to help them get indexed but it would be better if the pages were linked to by default from the listings page.

[soulnafein]

That's not the problem. The car in question got crawled correctly. It's the engine below it that didn't get crawled.

Anyway the car-chooser page is not the only way to get to cars.

[RyanKent]

Google has specifically stated they do not guarantee they will index all pages of a website. For Google to index a page, they have to decide the page offers value.

When I examine the page I note the following:

• the page's comments are all snippets from other pages which have the full comments. There is no unique content for the comments.

• the page's content is a total of 6 sentences, several of which are common to other pages on your site or are otherwise generic such as "Use the filter above to see reviews for the other engines."

• the page has no backlinks to it, and the linking page has no links either. There are no apparent off page factors to indicate this page is important.

If you were to earn links to the page, my bet is it would be indexed. If you were to add a decent amount of quality content on the page, it would probably be indexed as well.

[linztm]

Completely agree with Ryan, as an in-house SEO of a company with hundreds of thousands of pages I can guarantee you that not every page you have will be indexed via your sitemaps.  The best thing to do is just use the tips that Ryan stated above.  Link earning or content building, content building being the easier of the two.

[soulnafein]

Interesting in the previous (completely version of the site) that page used to get crawled fine, but it had more content (but exactly the same as the parent one).

I wonder whether I should accept some duplication, just bite the bullet and write lots of original content, maybe find a way to explain google (using microformats) that it's a page that aggregates reviews.

I still need to figure out whether is it more worthwhile focusing on long tail keywords that the engine page would allow (E.g. Audi A6 1.4 Bluemotion TDI reviews) or just merge it together with the parent one for a much more content rich (Audi A6 Reviews).

[RyanKent]

A factor could be another page was not indexed on the previous version of the site, which is presently indexed. On this site, the "other" page was indexed first, leading to this page not being indexed.

I noticed your site has other 2.0-TSI pages, and they were VW pages as well.

[TobiasM]

Hi David,

Apologies, in my haste I didnt take in it was the engine page. I hole heartedly agree with the other comments made here. I would also add that, although not the be all and end all, you may find that linking out to to other sites may help the page get indexed as well. If there are any additional, authoritative resources (that are not competition) it may be worth linking to a few of these. Adds value to the user as well.