When running A/B tests for Reddit ad creatives, what is the statistically significant sample size needed to confidently determine a winning variation at a 95% confidence level?

The required sample size depends on several factors, including the baseline conversion rate, the minimum detectable effect (the smallest improvement you want to be able to detect), and the desired statistical power (the probability of detecting a significant difference when one truly exists). Therefore, a fixed sample size cannot be universally stated. However, the calculation can be done systematically. First, the baseline conversion rate needs to be determined. Second, the minimum detectable effect or relative lift has to be decided, in terms of percentage. A lower MDE needs larger sample sizes. Third the confidence level is typically 95%, leaving a 5% chance of a false positive. Fourth the statistical power or probability that test will find statistical significance. High power means a decreased chance of a false negative. Once these four factors are calculated or decided upon, a sample size calculator can be used, such as one found online.