Welcome to the Q&A Forum

Hi Eduardo,

There is no reason why the language the site is developed in would have this affect since the page titles etc that the search engines read are in the final html produced, so if it looks right in the html it should look right to the crawlers. Same goes for the indexing of pages, although in that case there are more potential issues, but again none specific to ruby on rails. Care to give an example so we can have a look?

To rank well for a keyword + brand that is not your own is usually going to be tough. To be honest I doubt one paragraph is going to be enough. You should probably brainstorm about why people searching for this would find value in your page (for their search/intent). Once you have answered this question then you will be on the way to knowing what content you should add. Hope that makes sense

Hi Dana,

Thomas makes good points in regards nginx and lightspeed, but since making changes to server setup can sometimes be a big pain it is worth verifying if the page loads times have really been affected as you think.

The pagespeed insights numbers can be a bit deceiving in that they are telling you how many 'best practice' checkboxes you have ticked while not actually telling you how fast your pages are loading. You could have a pagespeed value of only 50 for example, but pages loading in 1.5 seconds (fast in other words). First thing to do if you havent already is to check the page speed times in analytics (content -> site speed) and see what numbers they are giving you and if they have gone up at all since you installed the apache modules. I cannot think of any technical reason for those modules to negatively affect page speed times, but it is worth checking. Also worth checking page load times through a tool that mimics higher server loads. I have used http://loadimpact.com/ which seems pretty good.

Once you have determined your real page load times you can then assess if there is a problem and the best way to approach it (which can depend a lot on what the root cause of the slow times is).

Hi Fabrizio,

If you put 50 in the 'bounce rate greater than box' instead of 0.5 then the table shown below the graph shows the data you want (only bounce rates over 50%, I think that is what you are after right?). I guess the graph cannot be show this filtered data although you can click the 'select a metric' link next to the visits dropdown on the top/left of the graph and add bounce rate to see the average bounce rate by visits/day if that helps with getting a baseline.

Hi,

The fact that you do not have mobile versions for all pages of the site makes your case a bit of an odd one when it comes to best practices.  I would recommend if at all possible to address this and make all pages available in mobile format also.

If for whatever reason this is not possible then I think you are better off using a meta robots tag NOINDEX to get your mobile results out of the serps completely and a rel canonical to the relevant desktop page to make extra sure there are no issues with duplicate content.

If you can get all pages into a mobile version then you will want to implement proper redirects based on user string as you are doing, and also implement rel alternate/canonical tags between the desktop and mobile pages of the site (which should make sure the right version of the site shows up in the relevant serps). For more details on the whole setup check:

http://moz.com/blog/how-to-optimize-a-mobile-site and
http://moz.com/blog/mobile-seo-process

Hi Rasmus,

Are you sure these clicks are being caused by crawlers? Usually this kind of tracking is triggered by javascript events and I don't think it would be common for crawlers to be triggering clicks like that (unless you have a different setup?).

Hi,

As Chris says I don't think there is a way to force a refresh on your campaign crawls, but that crawl test tool should be able to give you an indication if the relevant pages are still producing duplicate content issues or if the fix seems to be reducing them.

That being said, I don't think that robots.txt is the best way to approach duplicate content issues generally. Check out this guide for best practices. It is also worth noting that many times duplicate content issues can be solved by simply removing or adjusting the various differently formatted links that are producing them in the first place (though this depends a lot on which cms you are using and what the root cause of the duplicate content is).

Hi,

I think this will do what you are after: http://pro.moz.com/tools/crawl-test, limited to 3000 pages but should give an idea if the fix is working as you expect.

Hi Madhur,

Not much in terms of help, but this reminded me of a similar question here: http://moz.com/community/q/google-ranks-our-competitor-above-us-on-1000-s-of-branded-queires

Hi Melissa,

It is actually pretty common for wp based sites to spin out repeated 404 errors (or at least have the potential to do this). It could be a theme issue, a plugin issue or a setup issue or a combination of all 3 depending on the site.

First thing to do is figure out where the link is coming from. Download your moz error report in csv and filter by 404's. On the left you will have the 404 page and on the far right you will have the referring page. Go to the referring page and view the source and try to find that link. Keep an eye out for relative links from deep structure pages also (so you have page /category/page/5 and the link is a relative href="/" or href="/6" which can have the effect of spawning repeated 404s like /category/page/5/6 etc etc).

Once you have found the link, you should be able to identify the template part which is producing them and then act accordingly to either edit the template or adjust the settings/plugin so that they stop. If you can give us a real example of a 404 and its linking page we should be able to give you more specific info.

As I said the clue is likely in the message you get from the front end "Forbidden Access (flooding)"

If you search for this, the results all seem to mention joomla and that module. If you look through those results there are some mentions of the security features of this SEF module and how to turn them on/off. It is impossible to say if this is 100% the cause of your issue, but if your hosting company say everything is fine, and the message shown is specific to this joomla module, then it is a likely candidate. All things being equal, try turning off this security feature and see if the access denied errors in GWT go away.

Hi Tim,

Well both those urls you give for the 301 are returning a 404, but I don't think they are the cause of your original problem which is the access denied issue. For that I am pretty sure you need to be looking at that joomla SH404SEF module.

Hi Tim,

Your robots.txt looks ok from what I can tell. The 301s dont look odd (although what you have there is only one side of them right? I don't see the final page).

I think the clue is the message you get on the front end "Forbidden Access (flooding)". If you search for this phrase you start seeing references to the joomla SH404SEF module. See here for example: http://forum.joomla.org/viewtopic.php?p=1368937

I am not a joomla expert, but maybe it is a joomla issue instead of a server one, worth looking into.

Hi Tim,

It looks like you have some sort of server setup that is trying to block dos attacks or similar. If you put your site into screaming frog after the first few pages it starts returning 403 errors (access denied). If you then look at a page in a browser your see the message: Forbidden Access (flooding).

Is it likely that this is happening when google is trying to spider the site also?

Pushes competitors out of the SERPs.  Pisses them off.   Makes them spend time hopping around mad instead of waging war.

Love it!!

Hi,

Well it depends on the cms you are using, but most cms systems have the ability to spawn duplicate content through (for example) products in multiple categories with different urls, things like links to print versions of the page, wordpress can do it through archive pages, these kind of things. If you start looking at the referring pages in the right hand column you might well see a pattern emerging in regards how these duplicate pages are being linked to. This pattern is what I mean by systemic fault, not a fault as such but simply a part of the system's functionality that is creating this issue. The good news is these kinds of things are usually pretty easy to identify and fix either through altering the cms settings or else tinkering a bit in the code, and once you have done it, all related dup content issues are solved at the same time. Hope that makes sense!

Hi,

The duplicate report only shows that weeks results so you don't need to chase your tail

In the duplicate report csv download you should see the duped pages right above/below each other in the left hand column. One of these two pages is probably ranking, unusual for it to be both of them. Having the dup content potentially makes it harder for them to rank since you are in effect splitting authority between two identical pages so always good to get rid of duplicates.

If it is notifying of new duplicate content each week, it sounds like it might be a systemic fault. Check which pages are referring to the two different version (far right hand column in the csv) and that should help guide you to the point in the system where the duplicates are being created. Often times a pretty simple one time fix and you are good to go!

Hi Brandon,

I guess the part I dont understand is why you need to create a new campaign for the /widgets/ part specifically, the overall campaign for example.com will cover all sub folders and sub pages. So you can just set up the main campaign, create as many keywords as you like and label them for easy filtering into logical buckets (widgets, purple, brand, whatever suits). Then all pages in example.com will be tracked for these keywords and whichever page is highest ranked for each keyword will be shown in the main ranking page. If you have any other pages ranking for the same term but lower, they will be visible in the ranking history page. Hope I am not missing something obvious!

Hi Dan,

Well that will probably make it easier actually...

If you are using the Yoast plugin then it can produce breadrumbs which are pretty flexible and you can also manually add canonicals for individual pages. In that case I would think just set up the breadcrumbs as you like and if it makes sense choose a main category for any pages in multiple categories and canonical to the main one. Test to confirm, but I would think you might be able to do it without getting your hands into the code at all! The canonical tags themselves shouldn't effect the breadcrumbs in any way, so you should be good to go.

Hi Dan,

Just to confirm something that came to mind, is this for an ecom site with the potential to have products in multiple categories? If this is the case it is quite common to canonical the individual product pages to the root (or Everett Sizemore recommends to a standard /product/ or similar url which is also good for analytics filtering, check this video: http://moz.com/webinars/ecommerce-seo-fix-and-avoid-common-issues). If this is the case then depending on your cms it can be tricky to get the breadcrumbs to be created when people are directly hitting the single product page from a social share or other direct link.

It is possible though! I have had success with custom breadcrumb coding in Magento where if the single product page is directly accessed the breadcrumb will be created based on the products 'main category' and this has worked well in some situations. Again depends if you are talking ecommerce and which cms system you are using as to how tricky it will be.