Google not indexing /showing my site in search results...

[valdarama]

Hi there,

I know there are answers all over the web to this type of question (and in Webmaster tools) however, I think I have a specific problem that I can't really find an answer to online.

site is: www.lizlinkleter.com

Firstly, the site has been live for over 2 weeks... I have done everything from adding analytics, to submitting a sitemap, to adding to webmaster tools, to fetching each individual page as googlebot and then submitting to index via  webmaster tools. I've checked my robot files and code elsewhere on the site and the site is not blocking search engines (as far as I can see)

There are no security issues in webmaster tools or MOZ. Google says it has indexed 31 pages in the 'Index Status' section, but on the site dashboard it says only 2 URLS are indexed.

When I do a site:www.lizlinketer.com search the only results I get are pages that are excluded in the robots file: /xmlrpc.php & /admin-ajax.php.

Now, here's where I think the issue stems from - I developed the site myself for my wife and I am new to doing this, so I developed it on the live URL (I now know this was silly) - I did block the content from search engines and have the site passworded, but I think Google must have crawled the site before I did this - the issue with this was that I had pulled in the Wordpress theme's dummy content to make the site easier to build - so lots of nasty dupe content.

The site took me a couple of months to construct (working on it on and off) and I eventually pushed it live and submitted to Analytics and webmaster tools (obviously it was all original content at this stage)... But this is where I made another mistake - I submitted an old site map that had quite a few old dummy content URLs in there... I corrected this almost immediately, but it probably did not look good to Google...

My guess is that Google is punishing me for having the dummy content on the site when it first went live - fair enough - I was stupid - but how can I get it to index the real site?!

My question is, with no tech issues to clear up (I can't resubmit site through webmaster tools) how can I get Google to take notice of the site and have it show up in search results?

Your help would be massively appreciated!

Regards,

Fraser

 

[MickEdwards]

In Google Webmaster Tools what happens when you use the Fetch function?  Is Google able to crawl and render the page/s?

[valdarama]

Hi Michael,

Thanks so much for getting in touch.

http://i.imgur.com/zTbnxcl.png?1 - this is what I see in webmaster tools after a fetch request - seems to be indexing (although only partially when I ask to render also).

http://i.imgur.com/rXwhVmy.png - this is the result of the 'partial' when I look at it more closely in Webmaster Tools.

Thanks very much!

Fraser

[DirkC]

Hi Fraser,

I doubt that it is the dummy content which is causing the troubles. You use the x-robots-tag to put noindex/nofollow on all your pages. Probably this a setting in the config of your Wordpress site. More info on the tag can be found here: https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag?hl=en

Apart from that :

• Your homepage is only visible when javascript is enabled - the same applies to your portfolio page.
• The images are extremely heavy to load - you should seriously consider to make them a lot lighter (more than 50% of your images > 100K (a lot of them are bigger than 500K)

rgds,

DIrk

[DonnaDuncan]

Here's the content of your robots.txt file.

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /trackback/
Disallow: /xmlrpc.php
Disallow: ?wptheme=

Sitemap: http://www.lizlinkleter.com/sitemap_index.xml

Robots files are very very touchy. The duplicate inclusion of "Disallow: /wp-admin/" could be throwing you off. I'd clean that up first.

[valdarama]

Hi Dirk & Donna,

Thanks so much for taking the time to respond - I appreciate it....

Dirk - you are right - the x robots tag in the .htaccess file must have been the issue - I'm an idiot! I assumed because there was nothing on idividual pages or the robots file it must be okay.

I will also look to clean up those images and take a look at the java script.

Donna - I will clean up the robots file.

Thanks guys - you've really helped me out.

Regards,

Fraser

[David-Kley]

Glad to see you got things worked out. Best practice is to always have a "Disallow: /" rule in place in the root location when building a site, or to build it on an IP address via cpanel. A long long time ago we had an issue like this when we hired a rookie web designer, and had to go through everything making sure it was set correctly. Htaccess, robots, sitemap, sitemap crawl frequency, ODP (open directory project) settings, EVERYTHING.

Hope everything works out for your new site! Also, since you are having large load times due to a heavy template style, you may want to check this out:  http://designshack.net/articles/css/18-css-compression-tools-and-techniques/. Compression is your friend