This is some text inside of a div block.
This is some text inside of a div block.

What is Confidence Level?

Confidence Level is the percentage of time that a statistical outcome would be sufficient if numerous random samples are taken.

Confidence level carries a great importance in interpreting A/B testing results.

Let's give a confidence level example to clarify things for you:

>>> A/B testing with two email variations is conducted, and the following results appear:


Conversion= Number of Clickthroughs / Number of Emails Sent

A: 0.02

B: 0.025

Standard Error= Square root of <Conversion Rate*(1-Conversion Rate)/Sample Size>

A: 0.00198

B: 0.0020

Significance= <Conversion Rate(Variation B) – Conversion Rate(Variation A)>/Square root of <(Standard ErrorStanard Error)(Variation A) + (Standard ErrorStanard Error) (Variation B)>

Z-Score: 1.77


The probability corresponding to Z-Score of 1.77 is 0.96. This means the test is 96% confident that conversion derived in Variation B is truly higher than conversion in Variation A.

Generally in the case of email A/B testing, a confidence level of 95% or above is recommended.

Related Marketing Terms