Help! Pages not being indexed

[bshanahan]

Hi Mozzers, I need your help.
Our website (www.barnettcapitaladvisors.com) stopped being indexed in search engines following a round of major changes to URLs and content. There were a number of dead links for a few days before 301 redirects were properly put in place. And now, only 3 pages show up in bing when I do the search "site:barnettcapitaladvisors.com". A bunch of pages show up in Google for that search, but they're not any of the pages we want to show up.

Our home page and most important services pages are nowhere in search results.

What's going on here?
Our sitemap is at http://www.barnettcapitaladvisors.com/sites/default/files/users/AndrewCarrillo/sitemap/sitemap.xml
Robots.txt is at: http://www.barnettcapitaladvisors.com/robots.txt

Thanks!

[Raydon]

have you checked your google/bing webmasters tool yet? that often will give you the best answer in these situations?

Do you use analytics? Have you checked that as well?

if any pages do show up when you site search your domain on google, then you are being indexed, if pages you made are too new, it may take some time for them to become available in google's SERPs

it all just depends, how long has it been?

I see only 2 links in bing results for your site, but for google we have 9 results, how many should it be? 14?

[bshanahan]

Yes, I've searched Google Webmaster Tools and see a big dropoff in impressions on 5/4/13. 
There are 14 pages in the sitemap that should be showing up. But, instead, what's getting indexed (9 pages in Google and 2 of the 3 pages in Bing) are pages that aren't in the sitemap -- ones that were created and then taken down or that we don't care to be indexed. 
Yes, some of the URLs are new (three weeks ago) and that may be why they haven't appeared yet, but what about the home page? That isn't a new URL and it isn't appearing. Also, one new URL (which is also one of the important services pages that we want to appear) is appearing in bing: www.barnettcapitaladvisors.com/wealth-management but the other ones are not.

I use analytics as well, but what particularly in Google Analytics will help me resolve this?

[magicrob]

Hi Brien

The X-Robots tag has set many of your pages to noindex,nofollow. X-Robots is delivered as part of the HTTP response header so you won't find it in your HTML code.

In addition to that, your canonical tags are a little confused: the homepage canonical tag is set to

http://barnettcapitaladvisors.com/miami-financial-advisor-south-florida-financial-planner

when it should be barnettcapitaladvisors.com

I found these things by using Screaming Frog site crawler.

In your Robots file you're not allowing Yandex or Baidu to crawl the site - are you aware of this?

Have your developer correct the X-Robots and canonical issues.

[Raydon]

Nothing really to "resolve" this.

those tools could simply provide you with cues and signs to figure out what went wrong and when... traffic and referral data on those pages could also indicate when they exactly stopped appearing in search and you can the correlate that with your dev actions if you do have some memory or log of those steps taken and backtrack and find out what was set wrong.

So you see absolutely no errors no messages nothing in GWT per this old post?

if you are using drupal, check this thread about it, it may well be your problem...

also see the very bottom of this page that may help you diagnose if you have a setting wrong in your robots file.

[bshanahan]

Thanks for the help! The Screaming Frog site crawler really makes it much easier to understand the robots.txt file. It looks like it was set to "disallow" robots.