In A/B testing ad copy, what statistical measure determines if the improvement is unlikely to be random chance?

In A/B testing ad copy, the p-value is the statistical measure used to determine if an observed improvement is unlikely to be due to random chance. The p-value represents the probability of observing the obtained results (or more extreme results) if there is actually no real difference between the two ad copy versions. A small p-value (typically less than 0.05) indicates strong evidence against the null hypothesis (that there is no difference), suggesting that the observed improvement is statistically significant and not likely due to random variation. For example, if you A/B test two ad copies and find that one has a 10% higher conversion rate with a p-value of 0.03, it means there is only a 3% chance that the observed difference in conversion rates is due to random chance, providing strong evidence that the better-performing ad copy is genuinely more effective. A lower p-value provides higher confidence in the validity of the test results.