Welcome to the Q&A Forum

Hey EGOL

I have seen this in the past on my own site and on a few client sites in the past (which is not to say I have an answer here).

We were seeing completely random looking URLs that at first made me think the site had been somehow hacked or compromised but further investigation revealed that was not the case. We were just getting the strangest of links to pages that did not exist like xhyx.php?id=jamesbrown (that kind of thing).

We did nothing here and over time it seems to have resolved itself and these pages are not listed any longer. I tend to think of the webmaster tools data as diagnostics and it is telling me these pages don't exist so I can check for problems. Well, there is no problem, they don't exist and I am happy about that. Still, whether to mark them as fixed or not, I am unsure and would err towards not doing anything with them as they are not 'errors' as far as I am concerned. Likewise, I don't want to redirect them in most cases as I don't like the linking sites and have better things to do with my working day (I am not getting that time back - it's the digital equivalent or ironing clothes or some such laborious grind).

I had a look around again and whilst I can't find any specific answers regarding whether to mark them as fixed the following posts are of interest:

1. http://productforums.google.com/forum/#!topic/webmasters/3GTOLCE-8pk
2. https://productforums.google.com/forum/?hl=en#!category-topic/webmasters/webmaster-tools/rKI-38ohfbc

Particularly this quote from John Mueller at Google (webmaster tools guy I believe):

"In general, if a URL is really a 404, that's fine for us, and not something that would cause your site any problems in the long run. At any rate, you don't need to "fix" this problem (eg with a 301 redirect), if you're sure that the URL should really not exist. Having 404s listed in Webmaster Tools will generally not affect your site's crawling, indexing, or ranking; it's normal for websites to return 404 for URLs that don't exist."

So, my take is not to bother but would be interesting to ask the question in webmaster tools section of the Google product forums: https://productforums.google.com/forum/?hl=en#!categories/webmasters/webmaster-tools

Not an answer as such but hope that helps.

Marcus

Quite clearly, this is bonkers.

If you block access from search engine spiders how can they possibly index the content in good faith? You are hoping that they will not crawl the content to check what is there (or burn server resources) but they will still happily refer users of their search engine to these pages in good faith.

Additionally, it is highly unlikely that bots from the major engines are causing a measurable impact - Google for instance states they will only crawl one page every few seconds (1).

That said, there are a lot of parasites out there and crawlers that will eat up server time so there may also be some truth in what the host is saying. That said, there is still no excuse for this hatchet job of sorting things out.

The other angle here is that magento and ecommerce sites can often be a crawlers worst nightmare. As an example product comparison systems can often create thousands (I have seen millions) of crawlable URLs - now a sensible spider has a crawl budget and will give up but that's not saying all will. A simple crawl in screaming frog should give you an idea here (not that you will be able to do that) and in many cases where these problems exist this is enough to bring a server to it's knees.

In my mind you have a few things to do here

1. Convince the host that blocking all spiders is incorrect

Hopefully this thread and the references here should be more than enough to do this. Beyond that simply show them a fetch as Googlebot & the Crawl section in webmaster tools and you should be able to make your point quickly and easily.

2. Help the developer implement a more sensible list of what to block.

This article is a good start here:
http://searchenginewatch.com/article/2067357/Bye-bye-Crawler-Blocking-the-Parasites

Remember you can allow one (or more) robots and then disallow everything else:

User-agent: Google
Disallow:

User-agent: *
Disallow: /

Other options also exist such as limiting the speed at which a crawler can crawl - well, requesting that they limit the speed at which they crawl.

Also remember that any truly parasitic bot or crawler will likely ignore robots.txt anyway so you may need to implement some more advanced blocking at a firewall or server level.

3. Help the developer identify the cause of the resources problem

As hinted at above, if a crawl is causing problems there are likely issues somewhere. Whether this is as simple as straight up server resources or is more due to problems with the site and crawlable URLs needs to be determined but let me give you some pointers.

• SEO Audit - at least a crawl / indexation audit - lets see how many pages we can crawl? How does this stack up against the amount of products / categories? You may well find some easy wins here and sections of the site that can be blocked off or variables you tell Google not to crawl in webmaster tools. Nofollow directives and URLs can be your friend here as well so you tackle it on both fronts.
• Magento Optimisation - it is easy with a system like Magento to create pages that have a heavy burden on the database with hundreds of queries. If these options are not really used (only by crawlers) then they can be audited and removed / improved.
• Server Resources - Magento can be a hungry beast
• Dig into the http access logs to identify who and what is crawling and from where and come up with a list of what you need to block and how.

Summary

Ultimately, blocking all spiders is daft and there is a good chance it won't resolve the issue anyway - that is unless it screws over the clients search visibility so badly that they don't do any traffic! There are likely issues though be that with the site itself or something else so a good way to couch this to them is as their friend and helper - someone who will help them identify and resolve the issues. If it gets combative then it will only be harder to resolve.

Alternatively, you could move to another host. Part of me would suggest doing this anyway as no host should be able to hold you to ransom like this. This one daft move could have potentially ruined the clients visibility in what is a key time of the year for most online businesses. Imagine if they did not have an SEO on board? If they did not have an automated crawl to highlight these issues?

There is certainly a worthwhile exercise here as the site likely has some problems (or at least areas that can be improved upon) so optimisations can be made but, I would still consider jumping ship and moving to an SEO savvy host in the long term if bridges can't be built.

Hope that helps!
Marcus

References

1. https://support.google.com/webmasters/answer/182072?hl=en
2. http://searchenginewatch.com/article/2067357/Bye-bye-Crawler-Blocking-the-Parasites
3. http://www.robotstxt.org/

This is an art rather than an exact science but there are certainly things you can do to help decide which terms to go for.

1. Google Keyword Tool is for Adwords traffic & not organic search. This is not to say that it is not useful and the competitor information here can be used to gauge the quality of the keyword. That is, if there is a lot of paid search competition for this keyword, then it must convert  well so it is usually worth pursuing.

2. The keyword difficulty tool is useful as it looks at the metrics involved in ranking for the search term. There is a post on SEOMoz that explains the values returned and what is needed to do to rank for a term scoring X% - http://www.seomoz.org/blog/keyword-difficulty-tool-upgrade

3. Other tactics

I tend to do it like this:

• Set up a table with the keywords in for each page
• Get the advertiser competition as a quality score
• Get the seomoz difficulty score
• Get the exact match search volume from google adwords

Then I search for the keyword and check out the competition to look for two things

• Quality of results in terms of content & relevance
• Who ranks & how optimised are the first ten results

I am generally looking for cracks. Are there a lack of quality results? Great, it's an opportunity to make something better and get it rank - compare the organic & paid content as these guys are paying for the clicks so these pages are more likely to be finely tuned. Is there a lack of well optimised pages? Groovy, that's a crack and we can likely fight our way up the ranks.

Obviously,  there is so much too this. What is the standing of the site you are looking to optimise? Can you or the client create great content to help better fill the requirements of the search terms. Are there are opportunities with little competition, poorly optimised pages, bad landing page content etc.

By building a spreadsheet of keywords like  this, I then highlight them as I go. Red is no good, orange is possible or a future term, green is worth a crack.

Then... if they are difficult keywords, you can further assess the quality of the keywords and content you are planning to land them on with a brief adwords campaign & some conversion tracking in Google Analytics.

After all of that, you have some intelligence on which to base your search campaign and organic optimisation efforts.

Hope it helps!
Marcus

http://support.google.com/bin/answer.py?hl=en&answer=174717

Nothing to worry about

"1e100.net is a Google-owned domain name used to identify the servers in our network.

Following standard industry practice, we make sure each IP address has a corresponding hostname. In October 2009, we started using a single domain name to identify our servers across all Google products, rather than use different product domains such as youtube.com, blogger.com, and google.com. We did this for two reasons: first, to keep things simpler, and second, to proactively improve security by protecting against potential threats such as cross-site scripting attacks.

Most typical Internet users will never see 1e100.net, but we picked a Googley name for it just in case (1e100 is scientific notation for 1 googol)."

Hey, mobile is getting larger and I know that half of my web browsing (at least) is done on my smartphone and with android bringing smartphone features to the wider market then, it's just a matter of time before the world has the web in their pocket.

We tend to use Wordpress for most of our sites and if possible when we develop for clients. It is pretty SEO friendly out of the box and can easily be tweaked further if needs be. For mobile there are (free) plugins that allow you to serve a unique, simplified version of your site to mobile users and it takes about 5 minutes to download and install it.

For mobile device (apple/android/ipad etc) you can use WPTouch - it has a free version that does the job and then a pro version that provides  even more features. It has a load of AJAX and HTML5 goodness and is easily customised by non technical sorts through the control panel.

For those who are willing to dig into the code, it provides the same sort of foundation as a good theme and you can easily do more to brand it up and serve a spinky mobile version of your site with maybe a days worth of development. There are a bunch of others as well, onswipe is great for iPad so take a look around see what works best for you.

WPTouch Pro is the best one I have seen but there are similar plugins for Joomla, Drupal, Expression Engine etc so again, take a look and see what you can find.

Working on a bespoke site? Well, you could always take one of these solutions as a starting point, pick it apart and learn from it. Saves reinventing the wheel and some of these solutions are quite mature and robust now so again, there are options.

Hope it helps & give me a shout if any questions, I have been through the mill with this so happy to help. 
Marcus

Hey Michael

Chances are you will pick up some kind of filtering around these pages. The use of the word penalty may make it sound heavier than it should do and really we may just see these pages pulled down so they don't return for the targeted phrases. Inn terms of classification they are what is known as a doorway page and whilst there are multiple components of the algorithm that target this kind of content the famous one is Panda

Doorway pages - https://support.google.com/webmasters/answer/2721311?hl=en

Some notes from that page:

• Having multiple domain names or pages targeted at specific regions or cities that funnel users to one page
• Pages generated to funnel visitors into the actual usable or relevant portion of your site(s)
• Substantially similar pages that are closer to search results than a clearly defined, browseable hierarchy

I always find it useful to think about intent here and if you are based in Birmingham UK (so are we by the way so hello) and someone searches for 'web design birmingham' or 'web design hall green' then Google will tend to localise the results. This shows a pack of local results (3 pack now) and localised organic results where physical address weighs in. This makes it harder to rank these pages as you don't have the location signals but more importantly it illustrates the intent behind the search where folks are looking for a local business.

So, ranking these pages for locations like London without an address would be tough + they could send some low quality signals via Panda or other qualitative algorithm components. Then, even if they did rank, they may not convert the way you would like.

It is possible to work around this to some extent by creating truly authoritative pages that answer the need behind the query and why your company is a good fit. This involves getting inside your customers head and answering all the questions. I wrote about this in some detail here: http://searchengineland.com/local-seo-landing-pages-2-0-222583

Few options in my mind

1. Pick a few areas and create truly awesome pages and see how they do

2. Trial this with PPC and see how they do. If they pages don't convert at a rate that you can shovel traffic onto them then there may be better ways to spend your marketing time.

Hope that helps!

Marcus

You do get small variations in how the server wants the htaccess but give it a go and see if it works.

If you are happy it is working in the browser then there is a free bit of software called webbug that will let you see the HTTP headers to ensure you are getting the 301 permanent redirect returned.

Hope it helps.

Maybe I am missing something here but can you not use opensiteexplorer.org?

So, you want all links to say this page from internal and external sources:
http://www.seomoz.org/blog/how-much-does-google-make

A standard search (default settings in open site explorer) gives you just that:
http://www.opensiteexplorer.org/links?site=www.seomoz.org%2Fblog%2Fhow-much-does-google-make

You can then just dump all this info out to a spreadsheet and manipulate it as needed.

Hope that helps.
Marcus

Nope, not really.

What was the test?

Hey, you essentially have two URLs pointing to exactly the same page so you have an issue here. Exactly why prestashop is configured like that I don't know but it's certainly not going to help you long term and you need to get this sorted.

I guess you have a couple of options.

1. Reconfigure prestashop so it only maintains one unique URL for each page & 301 redirect the ones you remove to the new one to pick up any search referrals and transfer any link value

2. add a canonical URL to both versions of the page pointing to the main version of the page.

I just did a quick google and this seems to be an oft discussed topic on the prestashop forums so there is likely a tried and tested solution out there. As this is specific to that content management system I would suggest looking there for the solution as there is likely something you can do in the configuration of prestashop to sort this out.

You mentioned that you disabled the canonical plugin - why was this? Was it pointing to the correct versions of the pages (the categorised ones)? If so, putting that back in place may solve the issue and I would be happy to take a look under at the source code for you to confirm this.

Not exactly an answer,  but I think the answer  is likely simple and in the prestashop configuration of the disabled canonical module.

Let me know if I can help!
Marcus

Hey Michael

Yeah, it is possible to make landing pages work. These pages should really answer the questions the user provides and we have used this approach to supplement local visibility with many businesses - the real trick here is quality.

Ensure the page is high quality and avoid the temptation to just crank out hundreds of these with largely similar text.

Here is an example - you want business from London but are based in Birmingham. This page could outline how London is only an hour or so train journey from Birmingham and yet the price breaks you can offer over a London agency make using you a serious consideration. This speaks directly to that London customer and provides an angle for unique content. Then, bolster this with location specific reviews, testimonials, case studies etc etc to create a unique page and see how you get on.

For super competitive areas then you may need an additional push to get these pages to rank but you can always buy traffic and test and determine if the strategy is solid before doubling down on building some organic visibility.

At the end of the day it always comes down to quality - keep quality in mind at all times and all strategies have legs.

Hope that helps

Marcus

Hey Steve

Validation problems aside, if you want to update the homepage title you need to do this in one of two places. Assuming the all in one SEO package is working then you need to change it on the All In One SEO settings page which is under the settings menu.

Settings > All In One SEO

If that does not work, you can change the site title and tagline under the general settings page

Settings > General

If it still does not work then something is not quite right. Looking at your theme, it has been redesigned so it is possible the developers have hard coded the page title on the homepage. It would be a little unusual but I have seen worse.

Let me know if you are still struggling, happy to login to your WP admin and take a look for you.

Cheers
Marcus

I would guess the Moz blog and Q&A are a bespoke build so they can make it do exactly what they want.

If you are using WordPress this may be close to the kind of thing you are looking for:

http://wordpress.org/extend/plugins/cubepoints/

If this was me I would try to find something as close as possible and then dig into the code and hack it into exactly what you need (or get someone else to do the hacking if not your thing).

Hope that helps

Marcus

Hey,  I have just  ran that page through the on page optimisation tool and it comes back with no errors, perfect score as it happens. Think it may have just been a bug when you ran it through or some other small problem as it certainly seems fine now.

Marcus

Hey

First up, you have rel = next & prev on the paginated pages so that's good but I would also use the rel=canonical to the view all page as described here:

http://googlewebmastercentral.blogspot.co.uk/2011/09/view-all-in-search-results.html

The view all page in this category is not huge and loads nice and quickly so I cant see any reason not to 'help' google and give them the indication that this is where you want all rankings for those pages to be concentrated.

As always, experimentation is needed but I see things like this:

• You have a view all page and that is the desired page to display and Google prefers it all by itself

• You have a rel=next & rel=prev set up that is really for when you want to display individual component pages rather than the main page

• The search query you are referencing has no intent that makes it more specific to one of the paginated pages so the ideal landing page is the view all page

So, remove the rel=next & rel=prev and canonical it to the view all page and see how you get on. Allow it to reindex, record the results and make an decision based on that information.

As a disclaimer, this may not make any difference with the ranking as it seems they are not indexing your paginated pages AND if we do an info query on the main category page it shows details for the show all page. That said, this is the correct way to do it unless you would rather show the individual pages so I would still make the change.

I think when it comes down to it, Harveys just have like 5 x as many linking domains as you and you both have fairly natural looking anchor text (at the most cursory of views) so they are just outranking you here. I have not digged into the other results between you and them and a drop from 3 to 11 is a bit more than the usual flutters - is there anything else that has had a similar drop?

Hey, they are too long and don't really make any sense so... I think they need improving.

In an ideal world your URL should describe the content of the page as it will help win clicks and may give you an SEO boost through links that use the URL itself as the anchor.

Shorten them though, should be easy enough and certainly gives a usability boost.

Hey Blairr

Can you explain the problem a little more as I can't really see anything wrong with the site.

In your case, the 130 nav links is not the end of the world as you don't have a massive site (category, sub category, product etc). Also, I have done various searches for your title tag and the name of your church and you have great visibility in those results with a 1st  place homepage listing, a google map listing, site links and various other pages so nothing to worry about there.

I also did a search for a snippet of text from one of your deep testimonial pages "but I trust it will be encouraging to all. In March" - what was interesting was that it returned three pages, two of which were from another site called healingherald.org.

So, you may find that these pages are not going to rank so well as this content already exists on another site which is ranking above you for the content.

If you have ranking issues, I would investigate these first and depending on who owns the content either remove these dupe pages from the index or investigate using a canonical URL to the other site (or from the other site to you if it is your content).

Certainly, that is where I would start rather than the navigation as whilst there is a general 100 links a page rule, I would not obsess about  that in the overall hierarchy of this site.

Hope that helps!
Marcus

Hey Steve, that looks cool as well. Give me a shout when you get it set up, would like to take a look and see how it works out.

Cheers

Marcus

Hmm, not entirely sure, can you provide a link?

I have seen something a little odd recently with page titles. We are working on our new site,  was going to use Joomla and the Zoo CCK but unbelievably, whilst they have a really cool blog system built into Zoo, you can't set page titles so all article titles are the same and it uses the blog title.

But, we are seeing google take the article titles from the H1 and create it's own title tags from these to return the page in the results with a good, clickable title.

Obviously, we are going to change this and go back to Wordpress as it's a dealbreaker not being able to set page titles but the behaviour we have seen from Google in this instance is interesting.

So, sorry, waffling, are your page titles exactly as  you have put them? Is it trying to add anything? Do you have any special characters or anything that is causing problems?

I have just  done a bit of a random google and I can see lots of page titles getting cut off at around 58 characters in where the page title is longer than that. Lots of other full titles seem to be coming in at around 62 so there must be a slightly earlier breaking point now. Doing a bit of googling it looks like you can get snipped at anything over 64 so I would just work on getting them a little shorter.

Hope that helps!

Marcus

Yep, could be confusing, stick with the well worn conventions and use home.

If you are doing this for some supposed SEO benefit, honestly, don't bother, it won't help.