Welcome to the Q&A Forum

You are redirecting Google bot to a 500 page. Check http://serverfault.com/questions/238664/block-bots-with-iis-7-5-and-8-0 - this page list a number of ways on how to block bots to access an IIS server. Then check your own configuration. My guess would be that you implemented one of these rules by accident.

Update: you don't block Bing - but you block Yahoo as well.

Dirk

The number of pages indexed shown next to your sitemap isn't always very reliable - and you almost never get to 100%. 50% is pretty low - but could be normal if you launched your site only recently (or added the sitemap only recently).

The easiest way to check if a specific page is indexed is to check if it's cached. Just to do a cache:www.domain.com/mypage in your browser (at least in Chrome) - which will show you the version that Google has cached. If it's cached - it's in the index. Idem if you do a site:mydomain.com search - if you see all the pages of your site they are in the index.

With less than 20 pages it shouldn't take very long for Google to index them. If you are having doubts if your site is accessible you can try crawling your site with Screaming Frog (free for <500url's - else 99 pounds/yr) - which will also show you if H1, titles, meta descriptions etc are ok. You can also start a campaign in Moz to check if there are any issues.

Dirk

It's not a big issue. Just mark as fixed - if they 404 & no links exist them they will disappear from the reporting (every site has 404 errors). Even if you don't do anything it's not a problem;

Dirk

A canonical url always applies to a "full page" = content accessible under a specific url so it's not possible to apply the canonical url to only a part of the page.

Duplication of content is not always a reason for "punishment" by Google - check https://support.google.com/webmasters/answer/66359?hl=en : "unless it appears that the intent of the duplicate content is to be deceptive and manipulate search engine results" that you will get an action from Google. In most cases - "we (=Google) do a good job of choosing a version of the content to show in our search results".

You could consider to make one main page for each type of product and then list the specifics for each model number on the same page but it could lead to lower traffic.

You could also try to make for each detailed page a piece of "original" content, but in cases like yours I guess this would be virtually impossible.

Don't think everyone would agree on this advice - but given that your pages are currently ranking quite well and they are answering a certain customer need I wouldn't do touch them. The risk you run seems limited.

Dirk

I don't fully agree with the answer above - having 404's is not a direct cause for demoting. Check https://googlewebmastercentral.blogspot.be/2011/05/do-404s-hurt-my-site.html . Another interesting video on expired e-commerce products is here: http://searchengineland.com/googles-matt-cutts-seo-advice-unavailable-e-commerce-products-186882. Redirecting to the homepage (or any other non related page) is certainly not a good thing - this would be considered a soft 404.

It is however best practice to redirect products that remain in the catalog to the newer versions of the page. Even for those that don't remain in catalog you might consider to redirect them to a closely related new product. You indicate that given the number of products this would be not feasible. Are there no common patterns in the url to redirect these pages in bulk? Is it possible to put the redirect in the page header?

Redirects (properly implemented = going to the corresponding content) are both good for users & Google. If you don't redirect it will take some time before the new products are indexed and replacing the old ones in the ranking - which will almost certainly lead to rank loss: the 404 results will quickly disappear from the results and it will take a lot longer for the new ones to take their positions, it they manage to get back to these positions at all.

If you happen to have links point to these old product pages - you will loose these as well if they go to a 404.

In case that redirects is indeed not a valid option - make sure that you have a custom 404 pages so that user can easily find what their looking for. It might reduce the damage a bit.

If it was my site  - I would try to find a solution to implement 301's.

Dirk

The site did exist before - check https://web.archive.org/web/20141117163048/http://www.(your domain)/ - so quite possible it had a manual action (if the type of content was as low quality then as it is now) erasing it from the index.

Just a small additional note - I see that your robots.txt is blocking access to both js & swf files - you guys certainly know how to annoy Google

Quote: "**Disallowing crawling of Javascript or CSS files in your site’s robots.txt directly harms how well our algorithms render and index your content and can result in suboptimal rankings." **(source: http://googlewebmastercentral.blogspot.be/2014/10/updating-our-technical-webmaster.html)

It depends a bit on what is causing the self-referrals; normally when you check analytics it gives an indication what the source of the problem might be.

You could check this article https://threeventures.com/how-to-fix-self-referrals-in-google-analytics/ or this one (http://gatipoftheday.com/use-the-landing-pages-report-to-isolate-self-referrals/) on how to fix the problem.

Hope this helps,

rgds,

Dirk

Dave,

While checking your site, it seems that the tracking code is not always loaded - so probably it's more a technical glitch after the maintenance. Not clear what is causing it - when you install the tag assistant plugin you will see yourself. Seems to happen on first visit.

When it's not loaded your source (homepage) looks like this:

Dirk

If you have the list of url's you want to check  in csv you could use Screaming Frog: put the string in a  custom filter & crawl the urls based on the list. Applying the filter will only include the url's that contain the string in the source code.

rgds,

Dirk

As the widget is on each page of your site, it is generating a lot of internal links to these articles. If these articles have value for search, it could help them to get better positions.

On the other side, widgets like these are very ofter positioned in the same column as advertising, newsletter subscriptionboxes, ...etc - so I am not sure if Google attributes the same value to these links as to the links which are really embedded/integrated in your content.

If you can replace it by higher value content I would do it - but you'll have to make sure that the new content has indeed a higher value for your visits (try some A/B testing and check vitals stats like bouncerate, avg time on site, pageviews/visit, clickrates,..)

Hope this helps,

Dirk

I agree with Ryan that it's more the engagement which is important than the pageviews.

If you have client side filtering & ordering - you could use event tracking in Analytics to get better idea of what visitors are actually doing on your page. Each time a user changes the view, you track an event in Analytics. When you have a high bounce rate on your site, this will also give you a better idea of the actual time spent on a page (remember that there is no measurement of visit duration when a user visits only 1 page and no events are tracked - see also: http://cutroni.com/blog/2012/02/29/understanding-google-analytics-time-calculations/).

Jake,

Personally, I am convinced that Flash is not the optimal technology to build websites, and I am not the only one have this opinion, but as you point out that's besides the question.

However, if you want to use Flash, you'll have to do it properly, both for SEO and for your users. This is not a question of following the sheep, or some biased opinion of a bunch of flash haters, it is a fact.

For your users: each time have opened the site - it took an extremely long time to load - and I ended op with a page showing only the navigation in the middle. This can also be seen on the load video I inserted from Webpagetest that you can check here.

For SEO:

• Flash can be indexed (https://support.google.com/webmasters/answer/72746?hl=en#1) - however you block robots to index any .sw* file.

• You serve different content to robots & human users - that's called literally the definition of cloaking, and is considered a violation of Google Webmaster Guidelines. See https://support.google.com/webmasters/answer/66355?hl=en : "Cloaking refers to the practice of presenting different content or URL's to human users and search engines". The url's indexed in Google are static HTML pages which in normal conditions are never seen by your visitors and they are different from the url's you serve to human users.

• as mentioned in my previous remark - you also block the indexing of .js files - where google clearly states that this can have an impact of your rankings.

If you would hate that somebody would think "Flash is bad" based on this post, I consider it a scary thought if future readers of this post would think that it's ok to completely ignore the Google Webmaster Guidelines just because you seem to get away with it (until now).

If you are convinced that this drop is caused by negative SEO, you are free to do so. Maybe there is a correlation between the sudden drop & the events you describe. The truth is out there...

But, there is plenty that you can do on your own site to improve your ranking and defend yourself against the effect of any negative SEO attempt.

rgds,

Dirk

Not sure if you could really call it a best practice - but in Belgium (3 different languages) the normal configuration is not to determine the default language automatically but rather to present a first time visitor a "choose language" page and store the choice in a cookie for future visits. This is mainly for direct visits.

People coming in via search engines use queries in one of the languages, so normally Google will direct them to pages in that language. Again, on first time visit, the implicit choice of language is stored in a cookie.

All pages contain a link to switch to the other language(s) - which also changes the choice stored in the cookie.

Disadvantage of this system is that you add an additional layer to the site (choice of language) - advantage is that error margin is zero.

Systems which are based ip, browser language, ...etc are not 100% reliable - which could lead to unwanted results (in Belgium quite a sensitive issue if you server a Dutch page to a French speaking person - idem for French & Dutch speaking).

Hope this helps,

Dirk

Both measures are not really reliable to know how many pages are actually in the index.

Googles definition on 'Total Indexed': Shows the total URLs available to appear in search results, along with other URLs Google might discover by other means. This number changes over time as you add and remove pages. The number of indexed URLs is almost always significantly smaller than the number of crawled URLs, because Total indexed excludes URLs identified as duplicatesor non-canonical, as less useful, or that contain a meta noindex tag. (source: https://support.google.com/webmasters/answer/2642366) 
=> basically this number contains all the pages Google has been crawling since the launch of your site, and is quite useless as absolute figure, it's the evolution over time which is more important

site:domain.com will show you pages that are eligible to show in the search results, but is not a really reliable measure either  - it's quite possible that more pages are indexed. On very big sites you will see this number change when you go to page 2,3,...Etc It is a good check to see if your site is indexed or not.

That said, the number of pages in the index is not a very important figure, it's rather the ranking of your site for your main keywords which is important, as it is the main driver of your traffic.

Sam,

I can only tell for the European market - but for the sites I manage there is a huge difference between the organic traffic estimated by Semrush & the real organic traffic measured in Google Analytics. I did a quick check for 3 domains, spread across Europe (traffic between 500K - 2 mio visits/month). Each time actual search traffic was about 3x higher than search traffic estimated by Semrush.

Dirk

You will find no tool on the market that will be able to show all the links to your site that can be found on the web.

If you would compare the links found by Semrush, Ahref, Moz,... you will probably links that are common for all tools, but also a lot of links that are only shown by one tool and not by the others. If you want to do a full analysis of your link profile you would probably need a subscription to all this tools but even then it will not be the same list as Google is using.

Apart from that, these indexes run on scheduled moments (next update of Moz is on April 30th). If these links were pretty recent - they will not yet have been indexed by Moz/Semrush.

Hope this helps,

Dirk

I agree with you that positioning is more about the content than having an exact match url. However, as Ryan indicates, buying the domain is low cost, so I would buy it anyway, if only to redirect it to the main domain.

This could potentially save you a lot of trouble if your boss (or higher) would type in the exact match domain only to see that it would go to one of your competitors. They usually don't know much about SEO, and for some reason a lot of them are convinced that having the exact match domain is the preferred tactic to secure a number one position...

Do a full crawl of your site with a tool like Screaming Frog. If you configure the spider to respect robots.txt & canonicals it will behave like google bot you will see if all your pages are indexed properly. The tool also gives you valuable info on things like H1, Title, Meta descriptions (and standard filters to check if they are missing, duplicates, too long/short,...). It shows your internal site structure (number of clicks needed to get to each page) and the size of your images. You can even use it to check if the analytics tag is present on each page using a custom filter.

It's not free if you use it in spider mode (the free version only spiders 500 URI's) - but if you have a sitemap, you could also do a crawl based on the list of your url's (which is free, regardless the number of url's). Limitation is that you don't get the structure and the internal links, but if it is a brand new site and not too big, that shouldn't really be a problem.

rgds

Dirk

Great to post this at the top of the Q&A - noticed the first questions appearing today - probably many more to follow (hopefully fellow Moz'ers will read this post before posting the question).