Images on sub domain fed from CDN

[TammyWood]

I have a client that uses a CDN to fill images, from a sub domain ( images.domain.com). We've made sure that the sub domain itself is not blocked. We've added a robots.txt file, we're creating an image sitemap file & we've verified ownership of the domain within GWT.

Yet, any crawler that I use only see's the first page of the sub domain (which is .html) but none of the subsequent URL's which are all .jpeg.

Is there something simple I'm missing here?

[tretanto]

Did you reference the sitemap in the robots.txt file or did you set up it in GWT?

[TammyWood]

The sitemap is not completed yet. Server logs show Googlebot only indexing one page the .html page, not other pages.

[tretanto]

In order to put those image URLs for the crawler to be able to access them you should either:

• Link to the URLs of the images (does that .html page in the subdomain contain these URLs?)

or

• Use the images URLs as resources in the pages already been crawled. Unfortunately this could be tricky when dealing with CDNs since those resources are dynamic.

In either case, the sitemap will solve your problem.

[TammyWood]

We've correctly applied the CNAME directive from the CDN to reflect the subdomain. Yet, when Google or any other tool attempts to crawl it only shows ONE URL. Not the images that are residing on their own independent URL's.

[Everett]

Are you using your own subdomain or one somewhere else (e.g. akamai.com)? You should use your own subdomain, if possible.

Was this a change from a previous version that didn't use a CDN? If those images were/are hosted on your primary domain be sure to match the filenames and paths as closely as possible to what they were before.

If you're doing that you shouldn't have a problem once the sitemap is submitted.

For more information please check out this post:
http://www.goinflow.com/four-seo-best-practices-for-using-a-content-delivery-network-cdn/

How do you know that Google only attempts to crawl the primary domain URL (i.e. the .html page)? Are you checking log files?

Is the crawler you're using set to crawl external URLs? If not, that could be the issue. Technically a subdomain is a totally separate website so most tools don't crawl them by default.

[TammyWood]

Yes, we are doing everything correctly, aside from waiting for IT department to create a sitemap.

[FashionLux]

We have the same issue however we have image XML sitemaps on each country subdomain's XML Index which point to the image files on images.domain.com.

Example:

https://uk.domain.com/image-sitemap1.xml

https://us.domain.com/image-sitemap1.xml

These 2 files are the same.

We also don't have a homepage on images.domain.com and it currently responds with a 404.

Do you think we need to create a landing page on the homepage and host the image XML sitemap at https://images.domain.com/images-sitemap1.xml rather than in each sub-domain?

Thanks.

[alphonseha]

I am a little confused. The question was marked answered, but which one is the answer?

[Everett]

Alphonse it sounded like they were just waiting for the sitemap to launch. Other than that, I couldn't think of anything else to add because the sitemap should solve their issue. However, I have marked this as "Discussion" again.