p-Value is the probability of observing a result (or one more extreme) when there is no actual difference (eg. no test effect). Differences between test and control can arise from random chance and statistical noise. Extreme differences are less likely and p-value is a measure of this probability and “extremeness” of a result under the null hypothesis (no effect).

• A p-value is used in hypothesis testing to reject the null hypothesis. A p-value below the significance threshold is evidence against a null hypothesis.
• The smaller the p-value, the less likely a result is (when there is no actual effect).
• To prevent bias and subjectivity, we commonly set a threshold (alpha) before the experiment. 0.05 is most commonly used.
• p-Value is not the probability that the control is better than the test, nor the probability that the null hypothesis is true.

Other helpful tips:

• p-Values are expressed as decimals although it may be easier to understand what they are if you convert them to a percentage. For example, a p value of 0.0254 is 2.54%. This means there is a 2.54% chance of obtaining this result (or one as extreme) if there isn’t any effect. That’s pretty tiny.