Question

In statistical testing, what specific phenomenon occurs when the number of observed samples in experiment groups deviates significantly from the planned split ratio, indicating a flaw in the randomization process?

Accepted Answer

The phenomenon is called sample ratio mismatch, or SRM. This occurs when the actual number of participants assigned to experimental groups differs significantly from the intended distribution, such as a planned 50/50 split resulting in a 55/45 outcome. This imbalance indicates a failure in the randomization process, which is the system responsible for assigning participants to groups by chance. Because the split is no longer random, the experimental groups are likely biased, meaning they differ in ways other than the variable being tested. This invalidates the statistical results because any difference in performance might be caused by this underlying bias rather than the experimental changes. To detect SRM, researchers use a chi-squared goodness-of-fit test, which compares the observed group sizes against the expected sizes to determine if the deviation is statistically significant rather than just random variation. If the p-value from this test is below the significance threshold, it confirms that the assignment process is compromised and the experiment results cannot be trusted.

Home → All Courses → Engineering and Technology Courses → Product Management Engineering → Flashcard

In statistical testing, what specific phenomenon occurs when the number of observed samples in experiment groups deviates significantly from the planned split ratio, indicating a flaw in the randomization process?