Confidence Level is the percentage of time that a statistical outcome would be sufficient if numerous random samples are taken.
Confidence level carries a great importance in interpreting A/B testing results.
Let's give an example to clarify the knowledge;
>>> An A/B testing with the two email variations is conducted, and gives the following results:
Conversion= Number of Clickthroughs / Number of Emails Sent
Standard Error= Square root of <Conversion Rate*(1-Conversion Rate)/Sample Size>
Significance= <Conversion Rate(Variation B) – Conversion Rate(Variation A)>/Square root of <(Standard ErrorStanard Error)(Variation A) + (Standard ErrorStanard Error) (Variation B)>
The probability corresponding to Z-Score of 1.77 is 0.96. This means the test is 96% confident that conversion derived in Variation B is truly higher than conversion in Variation A.
Generally in the case of email A/B testing, a confidence level of 95% or above is recommended.