In A/B testing, what statistical metric determines whether the observed difference between two ad variations is likely due to chance or a real effect?

In A/B testing, the p-value is the statistical metric that determines whether the observed difference between two ad variations is likely due to chance or a real effect. The p-value represents the probability that the observed results could have occurred by random chance, assuming that there is no real difference between the variations being tested. A lower p-value indicates stronger evidence against the null hypothesis (that there is no difference), suggesting that the observed difference is statistically significant and not due to random variation. For example, if an A/B test shows that ad variation A has a higher click-through rate than ad variation B, and the p-value is 0.05, it means there is a 5% chance that the observed difference in click-through rates is due to random chance. Typically, a p-value of 0.05 or less is considered statistically significant, meaning that the difference is likely real.