How do I find out which pages are being indexed on my site and which are not?

[benjmoz]

Hi,

I doing my first technical audit on my site. I am learning how to do an audit as i go and am a lost. I know some page won't be indexed but how do I:

1. Check the site for all pages, both indexed and not indexed

2. Run a report to show indexed pages only (i am presuming i can do this via screaming Frog or webmaster tool)

3. I can do a comparison between the two list and work out which pages are not being indexed.

I'll then need to figure out way. I'll cross this bridge once i get to it

Thanks Ben

[WhiteboardCreations]

Ben,

You can check a couple things:

1. Have you submitted your XML site map to Google? If not, create one and get it submitted so you tell Google what pages you want indexed.
2. Submit your domain and all pages through Google Webmasters Tool as well (Login > left side bar > Crawl > Fetch as Google
3. Screaming Frog is an awesome software, so yes, if you have it, use it to scan your pages
4. Try and do a simple "site:domainname.com" search in Google to see what is being indexed from your domain

Cross reference it all and you will then have a better understanding. I do believe, your sitemap is crucial in telling Google exactly what pages you do and do not want indexed. They will follow that. You're on the right track and hope my input was helpful! - Patrick

[benjmoz]

Hi Patrick,

Thanks for replying.

Can you recommend any tools for creating the site map i've had a look around and the few i've found seem to all deliver different results? One has been submitted previously so i need to go through the process for myself so i can under these basics.

I've had a read up on robot txt so i understand what is happening there from an exclusion perspective and once i understand how the XML site works ill be able to do an audit as mentioned above.

Ben

[Paddy_Moogan]

Hi Ben,

I'd echo what Patrick has said and probably recommend his first suggestion the most. Google Webmaster Tools is a good way of checking indexation and if you have a large site with lots of categories, you can even break down the sitemaps by category so that you can see if certain areas are having problems.

Here is an old, but still relevant post on the topic:

http://www.branded3.com/blogs/using-multiple-sitemaps-to-analyse-indexation-on-large-sites/

In terms of creating the sitemap, Screaming Frog has an option under Advanced Export for creating an XML sitemap file for you which works very well. You just need to make sure you're only including pages that you want indexed in there.

Cheers.

Paddy