Welcome to the Q&A Forum

It may help, but a lot of plugins burden the server more on the back-end than they do on the front end. Still, it all helps! In this case though, I wouldn't expect anything noticeable in that area (at all)

Right... I wrote WAY more below than I had anticipated writing. But I think I have demonstrated pretty conclusively that yes, site performance is your main hindrance. I may use some dramatic language below, it's just because I'm passionate about search and SEO :') so please don't be offended. That being said, yeah the situation is pretty bad

It (the site) actually is very slow and laggy, that could have a lot to do with it if site performance has decreased. Using my favoured page-speed tool, GTMetrix - you can easily see that the scores are pretty bad all around. Here's a screenshot

If you look at the waterfall chart it generates (needs a free account only, no payment details required) then you can see that the request "GET ryemeadgroup.co.uk" occurs three times and seems to take ages and ages to respond. Looking at the data as a once-over, I can't tell if that's just the request to get the whole page (so obviously it would be longest) or something else. If that is what it is, I don't get why it recurs thrice

You could optimise all your images. You could set GZip compression and do lots of things, but the fact is - the server environment is just horribly, horribly weak. I did a very small, few minutes long stress test which I had to cancel almost immediately. Even crawling the site at a few URLs per second stops it rendering and causes it to time-out! If I had left the test going it could have hurt the site or taken it offline, so before that happened I stopped the crawler in its tracks

By the way, this is a crawler designed for low-intensity SEO crawling, not actual stress-testing or DDoS simulation. From what I can see here, any user with a reasonable website connection (maybe BT infinity or one of the Virgin cable deals) could, if they wanted to - just take your site offline, just like that. Someone trying to do it maliciously would use much more aggressive tools and crawling techniques

The crawl delay has probably been set in your robots.txt to compensate for this. But what that means is, with the crawl delay on Google can't index your site and content fast enough. With the delay removed, they still won't be able to because even their basic, non-intensive crawling will take your site offline in seconds / minutes

Obviously when a new site goes live, even with the same files on the same server-environment, it slows down for a bit (while the cache builds back up again). Whilst that could be part of the problem, the main problem is that no matter what you do - that server environ is only fit for a hobbyist, not a fully-fledged business. Worse still, even if you did some amazing Digital PR and got loads of traffic, you wouldn't get it because it would knock the site offline. So even when you win, you'll lose anyway

Check out these terrible (worst I have ever seen) Google Page-Speed loading scores. I fired Google off to check your site, just after I had finished and killed my extremely moderate stress-test. Look at this screenshot. I know Google PSI asks for too much, but this is just dire

Let's check through Google's Mobile Website Speed Testing tool (which fires requests through 3g, just to be safe). This time I left a margin of time after the mild stress-test to see if scores got significantly better. Nope, the results are still really poor

Let's try Pingdom Tools. Here are the results. Again, really poor grade. Wouldn't be happy if my kid got a D on a school assignment, not happy with the grade here either. Beginning to see a pattern with all this?

I guess you might be in one of those situations where decision-makers are saying, before we put more money into the site (for a better server) we want to see more success. Well guess what? That's technically impossible. If you get more traffic, your site will go down - taking the Google Analytics tracking script with it. So all that traffic will be invisible to them, and they'll never have the data to decide that they need better. As such, it's a self-fulfilling circle here unless they'll just budge

What you're in danger of here, is taking an old mule on its last legs and 'optimising it'. Give it reinforced leg-braces, stuff it full of steroids. It all helps a bit, but ... you know, never be surprised when it entirely fails to beat an actual race-horse. It's not winning the grand national, the old girl  (the hosting environment) simply doesn't have it in her

So what's the problem? Not enough processing power (brain-power) for the server? Not enough RAM (memory)? Not enough bandwidth?

It could be any or all of the above. When someone requests data (web-pages) from your server, three basic things have to happen. The server has to 'think of' what the user wants, if the processor can't keep up - then no matter how good the bandwidth, it's like putting "2+2=4" on a huge blackboard and expecting it to look good, look sophisticated (it won't). Next you have local memory. Once the server thinks of what it has to 'assemble' for the user, those lego pieces have to be put down in (very) temporary 'storage' before they can be shipped to the user. If you have great processing power and bandwidth on your server, great - but if it's funneled through narrow local memory... It's like trying to fit the entire theory of general relatively on one corner of a post-it note. It's not happening

Finally you have your bandwidth. If everything else is great and your bandwidth sucks, then locally you generate complex pages really fast - but can't get them 'shipped' to the user in a timely fashion

I don't know what the exact problem(s) with you server are, but it sucks. You have to investigate and secure better spend, or it will never ever improve!

Quite often, page-speed changes only influence moderate gains in Google's SERPs. That's because once you reach a certain standard, most users will be satisfied and so will Google. But in your extreme situation, I can well mark that - you will be under nasty algorithmic SERP devaluations.

Dev changes and coding will only get you so far, at the end of the day your site needs a good home to live in. Currently, it doesn't have one. It doesn't live on a server that Google would take seriously for an online business (IMO)

Important P.S: There is one other alternative issue, other than what I have summarised. It may not be that the server is weak, it may be that the server is programmed to 'fake' weakness and 'play dead' when one source (a crawler, or a user) gets too aggressive. If so, that same fail-safe has been over applied and is affecting Google's results. Play dead to Google? Get dead results. To establish whether my initial thoughts are right - or whether this final 'PS' is correct, we'd need to talk in real-time (over chat) and establish a two-way stress test. I'd need to stress the site again a little, make it time out for me - then see if it's also timing out for you. If it's affecting just me, your problem is an over-applied defense mechanism. If it's affecting both of us... the server is garbage

Basically I don't think Moz records enriched URL names, because their database and / or CSV output is incapable of storing rich characters (e.g: in UTF-8 you have Latin and Greek characters, but with super basic txt sub-formats like ANSI you do not). As such Moz is forced to 'convert' the URLs to utilise URL encoding: https://www.w3schools.com/tags/ref_urlencode.asp - whereby a combination of 'allowed' symbols creates the special characters in your browser window (once a link is visited)

In reality it would be better if Moz would just update their tools to support special characters as they see a lot of common usage in URL strings these days, particularly in Western / Eastern Europe (and beyond)

Unless Moz were to update their tool I think you'd just have to 'manually ignore' those errors or recalculate the 'real' values yourself in a spreadsheet

+1 for "Make sure that they are not created in the first place" haha

The erroneous advice you got was that Google handle reconsideration requests within a few days. Usually it does take a few weeks and in extreme situations it can take months. Usually when it takes months, it's because you have repeat-offended over the same issue. If you get a penalty for link-spam and then do more link-spam, each time you submit to be reconsidered they leave it longer and longer

It's also down to Google's internal resources. The sad fact of the matter is that, although losing Google does heavy damage to your site... Losing your site, doesn't do heavy damage to Google. If there are other matters which Google are pulling focus to internally, it can take quite some time to 'be seen' as it were

It would be strange of Google to mis-apply a penalty of some kind. Usually if you have hacked content, either those pages get nerfed or your whole site gets nerfed. Having one page get nerfed which was not part of the assault, is extremely unusual

I know that one thing you can do, is to create a free account to query Google's 'safe browsing' API

• https://developers.google.com/safe-browsing/

My next step would be, to ascertain the URL of every page on your site that exists now or has existed within the past 12 months (just to be sure). You can get historic URLs out of the Wayback Machine, Google Analytics (by making a table that combines host-name and page / landing page - unfortunately they won't give you protocol... so hope that hasn't changed for you in the past year!) or Google Search Console. The live URLs, just crawl with Screaming Frog or similar.

Once you have a complete list, get a developer to build a rough script that will query all your URLs against Google's safe-browsing API. That would tell you if Google still sees a problem. It will tell you where Google sees the problem, if it does indeed see a problem in this area (and whether it sees the problem on live or dead pages)

When you have that to hand, you'll be in a much better position to know whether you still have an issue or whether you just haven't been seen by a Google rep yet. When they decline a reconsideration request, usually they do tell you

I think that the safe-browsing API, whilst free to access - is limited to 10k queries per day. Don't try and get clever and get around it, if you are already having Google problems (you don't want MORE!)

Another thing, it never hurts to link to an 'open' (link access required only) GoogleDoc (their version of word) within your reconsideration note. In the GoogleDoc you can much more fully explain, even with screenshots - what the heck is goin' on!

You need more information right now. Sorry I'm not giving you an instant solution, but I am telling you exactly what I'd do in your position

Usually that depends upon the significance of the edit and how much Google trusts the query-space (or whether it is a YMYL query-space). A lot of people say that gambling sites are YMYL sites

The real question is, do the rankings get removed and then come back again later? Or do they get removed and just stay gone period?

Robots.TXT Rules

If you have architecture like:

site.com/blog/post/page/1

Then use:

User-agent: rogerbot

Disallow: /page/

If you have architecture like:

site.com/blog/post?p=1

Then use:

User-agent: rogerbot

Disallow: /*?p=

If you have architecture like:

site.com/blog/post?page=1

Then use:

User-agent: rogerbot

Disallow: /*?page=

---

That should pretty much stop Rogerbot from crawling paginated content. It would certainly stop Googlebot, but I don't quite know if Rogerbot respects the "*" wildcard like Googlebot does. Give it a try, see what happens

Don't worry, in the robots.txt file only "*" is respected as a wildcard, so you won't have any problems with "?" and there won't be any need for an escape character

So in the SEO industry, we have tools which measure a site's estimated worth and traffic intake (just from search, it doesn't tend to reflect anything else)

One is SEMRush:

• https://d.pr/i/zvr8cY.png (screenshot)

The other main one, is Ahrefs:

• https://d.pr/i/0EbHaI.png (screenshot)

Neither of these tools picked up extremely significant movements within the past year. Neither of them seem to think that (in terms of SEO) the site was ever doing that well to begin with.

Basically these tools contain colossal indexes of Google keywords. They monitor these high to mid-value terms frequently, and see who is ranking. They leverage CTR (click-through-rate) data against ranking positions to estimate 'search visibility' (which is like a, ultra-rough traffic estimate - never to be taken as an absolute)

If these tools aren't showing that anything bad happened (or if they're not showing that performance was ever very good), then there are some possible reasons:

1. The tools happen not to contain most of your main keywords in their keyword indexes

2. Your visits were never coming from SEO in the first place, or you had broken tracking which was inflating those numbers

3. You had good tracking and your SEO company broke it (thus making it look falsely like there's been some massive drop)

4. Your visits were coming from SEO, but mostly not from Google. Other search engines exist

It's hard to know which of these is the main offender, or whether there's another reason (some 'unknown-unknown'). I'll take a brief look into your Analytics profile if you want. It could possibly shed some light in terms of what the heck is going on!

If you want to connect further, my email is on my profile page. I can't promise I'll find a solution, but these kinds of problems intrigue me

It's likely that  you don't have access to edit the coding on these weird plugin URLs. As such, normal techniques like using a Meta no-index tag in the HTML may be non-viable.

You could use the HTTP header (server level stuff) to help you out. I'd advise adding two strong directives to the afflicted URLs through the HTTP header so that Google gets the message:

1. Use the X-Robots deployment of the no-index directive on the affected URLs, at the HTTP header (not the HTML) level. That linked pages tells you about the normal HTML implementation, but also about the X-Robots implementation which is the one you need (scroll down a bit)

2. Serve status code 410 (gone) on the affected URLs

That should prompt Google to de-index those pages. Once they are de-indexed, you can use robots.txt to block Google from crawling such URLs in the future (which will stop the problem happening again!)

It's important to de-index the URLs before you do any robots.txt stuff. If Google can't crawl the affected URLs, it can't find the info (in the HTTP header) to know that it should de-index those pages

Once Google is blocked from both indexing and crawling these pages, they should begin to stop caching them too

Hope that helps

In general, I don't think that this is a great idea. Although Google does meter out crawl-allowance, Google also wants a realistic view of the pages which it is crawling. Your attempt at easing the burden of Google's crawl-bots may be seen as an attempt to 'fake' good page-speed metrics, for example (by letting Google load the web-page much faster than end users). This could cause some issues with your rankings if uncovered by a 'dumb' algorithm (which won't factor in your good intentions)

Your efforts may also be unrequired. Although Google 'can' fire and crawl JavaScript generated elements, it doesn't always do so and it doesn't do that for everyone. If you read my (main) response to this question, you'll get a much better idea of what I'm talking about here. As such, the majority of the time - you may be taking on 'potential' risk for no reward

Would it be possible to code things slightly differently? Currently you state that this is your approach:

"This means that we are actively adding javascript code which will load the Intercom javascript on each page, and render the button afterwards"

Could you not add the button through HTML / CSS, and bind a smaller script to the button which then loads the "Intercom javascript"? I am assuming here that the "Intercom javascript" is the large script which is slowing the page(s) down. Why not load that script, only on request (seems logical, but also admit I am no dev - sorry)? It just seems as though more things are being initiated and loaded up-front than are really required

Google want to know which technologies are deployed on your page if they choose to look, they also don't want people going around faking higher page-speed loading scores

If you really want to stop Google wasting time on that script, your basic options would be:

• Code the site to refuse to serve the script to the "googlebot" user agent
• Block the script in robots.txt so that it is never crawled (directive only)

The first option is a little thermonuclear and may mean you get accused of cloaking (unlikely), or at the least 'faking' higher page-speed scores (more likely). The second option is only a directive which Google can disregard, so the risks are lower. The down-side is that Google will pick up on the blocked resource, and may not elevate your page-loading speed. Even if they do, they may say "since we can't view this script or know what it does, we don't know what the implication for end-users is so we'll dampen the rankings a little as a risk assessment factor"

Myself, I would look for an implementation that doesn't slow the site down so much (for users or search-bots). I get that it may be tricky, obviously re-coding the JS from Intercom would probably break the chat entirely. Maybe though, you could think about when that script has to be loaded. Is it really needed, on page-load, all the time for everyone? Or do people only need that functionality, when they choose to interact? How can you slot the loading of the code into that narrow trench, and get the best of both worlds?

Sorry it's not a super simple answer, hope it helps

Search Console's reports (Webmaster Tools) do include SEO data (organic) yes

In this case you would do well to read this documentation:

https://moz.com/help/moz-procedures/crawlers/rogerbot

This forum sees a lot of general SEO queries and is a bit of a broader community, not just for Moz products (though we do see lots of Moz product questions as well!)

Can you just deploy Meta no-index through the HTTP header instead of through the HTML?

https://yoast.com/x-robots-tag-play/

Thanks for all the info. I got your email and replied in full

A page which sends another URL PageRank does lose some of its own PageRank in the process. If you have 8 pages under a homepage and they all recieve PageRank, and then you add more pages which are also linked to from the HomePage, then all of the URLs will get a relatively equal amount (but since more URLs are being linked to, each linked page gains slightly less PageRank than before!)

If you don't want those pages to rank on Google or be passed PageRank (they are purely landing pages for referral or ad traffic) then you can orphan them if you want. Shouldn't be a huge deal and would keep things isolated. Those new pages, once orphaned - won't rank very well though (on Google's normal, organic results)

I think it is an issue because, people browsing your site in other languages will have the wrong language title displayed in their browser tabs if they are multi-tab browsing! The title tag is still one of the important ones for SEO, nothing has really come along to replace it

A businesses' ambitions in terms of an international roll-out, are to break into new (foreign) international query-spaces and get extra traffic (especially from Google, or leading search engines in other nations like Yandex and Baidu). Google's ambitions (when adding your international pages to their index) are that their audience can break onto other areas of the web which (due to the language barrier) were previously closed to them. But they want your content to then be 'tailored' to their international audiences, traffic which Google has no obligation to send your way. Google wants good UX for their searchers, so that Google remains top-dog in the search world

The less tailored your international roll-out is, the more shallow it is (with more pieces missing), the less confident Google will be. They will be less confident that sending their users to you will result in positive search-sentiment

Every piece of the jigsaw which you are missing, counts against you. It makes your international roll-out look more like a quick Google-translate powered land-grab, and less like an authentic international roll-out

My question to you is, when you identify a bad signal - why carry on sending it to Google?

Search is a competitive environment. If there are thing you won't do, others will

0. "We are a Grocery co-operative retailer and have chain of stores owned by different people. We are building a new website, where we would geo-locate the closest store to the customer and direct them to a particular store (selected based on cookie and geo location). All our stores have a consistent range of products + Variation in 25% range. I have few questions" - make sure you exempt Googlebot's user-agent from your geo-based redirects otherwise the crawling of your site will end up in a big horrible mess

1. "How to build a site-map. Since it will be mandatory for a store to be selected and same flow for the bot and user, should have all products across all stores in the sitemap? we are allowing users to find any products across all stores if they search by product identifier. But, they will be able to see products available in a particular store if go through the hierarchical journey of the website." - any pages you want Google to index should be in your XML sitemap. Any pages you don't want Google ti index should not be in there (period). If a URL uses a canonical tag to point somewhere else (and thus marks itself as NON-canonical) it shouldn't be in the XML sitemap. If a URL is blocked via robots.txt or Meta no-index directives, it shouldn't be in the XML sitemap.  If a URL results in an error or redirect, it shouldn't be in your XML sitemap.The main thing to concern yourself with, is creating a 'seamless' view  of indexation for Google. It seems like you'll have to have the same products available in multiple stores. You will want them all indexed, but will have to work hard to differentiate them (different images, different copy, different Meta data) otherwise Google will probably pick one product from one store as 'canonical' and not index the rest, leading to unfair product purchasing (users only purchasing X product from Y store, never the others). In reality, setting out to build a site which such highly divergent duplication is never going to yield great results, you'll just have to be aware of that from the outset

2. "Will the bot crawl all pages across all the stores or since it will be geolocated to only one store, the content belonging to only one store will be indexed?" - No it won't. Every time Google crawls from a different data centre, they will think all your other pages are being redirected now and that part of the site is now closed. Exempt Googlebot's user-agent from your redirects or face Google's fiery wrath when they fail to index anything properly

3. "We are also allowing customers to search for older products which they might have bought few years and that are not part of out catalogue any more. these products will not appear on the online hierarchical journey but, customers will be able to search and find the products . Will this affect our SEO ranking?" - If the pages are orphaned except in the XML sitemap, their rankings will go down over time. It won't necessarily hurt the rest of your site, though. Sometimes crappy results are better than no results at all!

Hope that helps

I would personally be more willing to trust in your rank tracker as it's independent data and you know how it works. With GSC in particular, you can have things like a knowledge graph image being hot-linked (by Google) counting as a ranking when actually no one can visit your site through it. For ranking data GSC can be very 'iffy' indeed

What Robin says about 301s and 302s is pure truth and that could also be a significant contributing factor, especially if it's quite widespread. Chantelle remind me at some point via email to look into this and nail down the 'exact' URLs that are 302-ing. If there is a problem there, we can find it and address it

Omg I haven't seen AWStats in years good on you for knowing what it is and that it even exists!

Your traffic figures though, don't prove that the hits are coming 'from' Google. So Google Search Console could be right (maybe those visits aren't from Google)

Search Console only shows your Google clicks (NOT visits), nothing else. It's data from Google's end, it just means people aren't clicking on your search results although they are sometimes seeing them (maybe they are bland and uncompelling)

Even clicks don't = traffic. Errors can happen between the click and the site loading which mean that a click isn't necessarily traffic. Google's data in this area can sometimes be sampled, so you could be right that GSC is wrong - but I wouldn't say it's overly likely (especially since it's data direct from Google)

You can get traffic from Google without getting any users visiting from Google, e.g: if you are visited by Googlebot. Just because Google is indexing you, that doesn't mean users 'from' Google are visiting you

Analytics may have better data for you, if you have Google Analytics installed 'properly'

If you had two different source codes served via user-agent (web-user vs googlebot) then you'd be more at risk of this. I can't categorically state that there is no risk in what you are doing, as Google operates multiple mathematical algorithms to determine when  'cloaked' content is being used - and guess what? Sometimes they go wrong

That being said, I don't believe your risk of garnering a penalty is particularly high with this type of thing

These are the guidelines:

• https://support.google.com/webmasters/answer/66355?hl=en

You're in a really gray area because, you aren't serving different URLs - but you _could _be serving different content (albeit only slightly). I say 'could' rather than 'are' as it entirely depends upon whether Google (on any particular crawl) decides to enable rendered crawling or not

If Google uses rendered crawling, and they take the content from their headless-browser page-render (which they can do, but don't always choose to as it's a more intensive crawling technique) then your content is actually the same for users and search engines. If however they just do a base-source scrape (which they also do frequently) and they take the content from the source code (which doesn't contain the visual cut-off) then you are serving different content to users and search engines

Because you've got right down into a granular area where the rules may or may not apply conditionally, I wouldn't think the risk was very high. If you ever get any problems, your main roadblock will be explaining the detail of the problem on Google's Webmaster Forums here. Support can be very hit and miss

That is a very weird situation and I think that, adding some canonical tags may not meet with your expectations. If Google have strong views about what is / is not a canonical URL, they will usually override canonical tags anyway (which is usually when you get the error you have described). I think we'd really need to see some examples of actual URLs to help you