Split test content experiment - Why can't GA identify a winner here?

[OptiBacUK]

I have been running a content experiment for a short while now and GA has just ended it saying it cannot determine a winner.

Looking at the images (links below), without any form of analysis I can already see a pattern of greater success in Variation 1. It ended with a 93% probability of outperforming the original yet the content experiment ended with no winner. Does this mean the 95% confidence threshold I set should've been lowered?

Ultimately I'm going to choose this as my winner but why didn't GA push it as the winner? Is there something I am missing?

Image 1 - Showing e-commerce performance (objective of split test was transactions)

Image 2 - Showing conversions  (same split test, same objective, just different report)

Your thoughts and comments will be appreciated.

[TrentonGreener]

Hey Emeka,

Short Answer:

You're correct. Effectively what Google is saying here is that they don't have enough statistical confidence to definitely tell you that the variation is outperforming the original at a 95% confidence level, but they do at a 93.8% confidence level.
Quick Note: 95% is the lowest setting in GA Experiments.

Long Answer:

The math behind this statistical significance calculation is:

Full credit to vwo.com for their A/B Testing Significance Calculator & doing all the work here.

Link to Image One - This is simply the data of the Control vs Variation & the Conversion Rate & Standard Error Rates

• Conversion Rate is: Conversions/Sessions.
• Standard Error is: √(Conversion Rate*(1-Conversion Rate)/Visitors)

Link to Image Two - Confidence Levels, Z-score, & P-value
To find if something is truly significant at a specific confidence level, we need to calculate the Z-score then use that value to find the P-value and from there we can determine the confidence level.

• Z-score is: (Control Conversion Rate-Variation Conversion Rate)/√((Control Standard Error^2)+(Variation Standard Error^2))
• For the P-value, we need to calculate the normal distribution of the z-score with a mean of 0 and a standard deviation of 1. The easiest way to do this is to use an online tool, here's a link to your specific example.

Finally, we take the Confidence % expressed as a decimal (i.e. 0.90, 0.95, 0.99) and 1 minus these values (i.e. 0.1, 0.05, 0.01)
If the P-value is greater than the Confidence % or less than 1 minus the Confidence %, then it is significant, otherwise it is not. Let me explain that using our example:
at 95% confidence, our P-value needs to be <0.05 or >0.95. Since our P-value is .05799, it doesn't fit either of those requirements and such is not significant at that confidence level.

I know that's a lot of math, but this is why Google Experiments is saying that the result is not statistically significant.

Hope this helps! Let me know if you have any further questions on this!

Trenton