Problems in indexing a website built with Magento

[advmedialab]

Hi all

My name is Riccardo and i work for a web marketing agency. Recently we're having some problem in indexing this website www.farmaermann.it which is based on Magento.

In particular considering google web master tools the website sitemap is ok (without any error) and correctly uploaded. However only 72 of 1.772 URL have been indexed; we sent the sitemap on google webmaster tools 8 days ago. We checked the structure of the robots.txt consulting several Magento guides and it looks well structured also.
In addition to this we noticed that some pages in google researches have different titles and they do not match the page title defined in Magento backend.

To conclude we can not understand if this indexing problems are related to the website sitemap, robots.txt or something else.
Has anybody had the same kind of problems?

Thank you all for your time and consideration

Riccardo

[linda_h]

Hi Ricardo,

Your home page is indexed.

It is most likely your problems are because of the robots.txt. -> http://www.farmaermann.it/robots.txt

1. You set a crawl delay of 10 seconds for all bots, which is quite long.
User-agent: *
Crawl-delay: 10

2. Some of your pages are not allowed to be crawled, like this one in your menu: http://www.farmaermann.it/integratori.html and http://www.farmaermann.it/contraccettivi-e-gravidanza.html
Allow: /*?p=
Allow: /catalog/seo_sitemap/category/
Allow: /catalogsearch/result/

My advice is to modify your robots.txt: remove the crawl delay (and check whether your server can handle that) and make sure the pages in your menu can be crawled.

[Guest]

This post is deleted!

[advmedialab]

Hi Josh!

Thank you very much for your help!
So probably there is a delay in webmaster tools data. Unfortunately we didn't develop the site but we only work on its optimization so we are a little bit confused with these data.

[advmedialab]

Hi Linda!

Unfortunately we didn't develop the website but we have to work on its optimization. Probably you have right about the robots.txt because the sitemaps looks ok. I will try to remove the crawl delay. On the other hand which disallow rules should i remove or which modifies should i do in particular?

Thank you very much for your help!

Riccardo

[linda_h]

Hi Riccardo,

Since I do not know which pages exist on your site, I cannot be a 100% sure. You can remove this though from your robots.txt and see what happens (in Google Search Console & Bing Webmaster Tools).

Allow: /*?p=
Allow: /catalog/seo_sitemap/category/
Allow: /catalogsearch/result/

Good luck!

[evolvingSEO]

Hi Riccardo

Yes to confirm the site is indexed and crawlable. Checking the number of URLs from a sitemap that are indexed isn't the most reliable way to see if you content is indexed. You can do a site: search on your domain in Google like this as probably one of the most reliable ways. Also, you can try jus crawling the site with a tool like Screaming Frog SEO Spider - and if the tool can crawl everything, there may be just a delay on Google's end. But in your case now, all looks good!

-Dan

[evolvingSEO]

However, you should allow Google to crawl your JavaScript and CSS (which is now blocked). Here's some background info on that:

• https://moz.com/blog/why-all-seos-should-unblock-js-css
• http://googlewebmastercentral.blogspot.com/2014/10/updating-our-technical-webmaster.html

[advmedialab]

Thank you Linda!

We will try and we will see what happens.

Riccardo

[advmedialab]

Hi Dan!

Thank you very much for your help and suggestions. I will try to follow your guidelines also.

Riccardo