What is Confidence Level?

Confidence Level is the percentage of time that a statistical outcome would be sufficient if numerous random samples are taken.

Confidence level carries a great importance in interpreting A/B testing results.

Let's give an confidence level example to clarify the knowledge;

>>> An A/B testing with the two email variations is conducted, and gives the following results:

example-variation-table-ab-test-results

Conversion= Number of Clickthroughs / Number of Emails Sent

A: 0.02

B: 0.025

Standard Error= Square root of <Conversion Rate*(1-Conversion Rate)/Sample Size>

A: 0.00198

B: 0.0020

Significance= <Conversion Rate(Variation B) – Conversion Rate(Variation A)>/Square root of <(Standard ErrorStanard Error)(Variation A) + (Standard ErrorStanard Error) (Variation B)>

Z-Score: 1.77

Interpretation:

The probability corresponding to Z-Score of 1.77 is 0.96. This means the test is 96% confident that conversion derived in Variation B is truly higher than conversion in Variation A.

Generally in the case of email A/B testing, a confidence level of 95% or above is recommended.