Question

During cross-validation, what is the exact term for the phenomenon where information from the test set inadvertently influences the training process, leading to overly optimistic performance estimates?

Accepted Answer

The exact term for this phenomenon is data leakage. Data leakage occurs when information from outside the training dataset is used to create the model, effectively allowing the model to peek at the answer key for the test data. During cross-validation, which is a technique used to evaluate how a model will perform on unseen data by partitioning a dataset into several folds for repeated training and testing, leakage happens if data processing steps are applied to the entire dataset before the split. For example, if you calculate the mean value of a feature across the whole dataset to fill in missing values before dividing the data into folds, the training set gains information about the distribution of the test set. Because the model has effectively been trained on a representation of the test set&#x27;s characteristics, it achieves artificially high accuracy during evaluation that does not reflect its ability to generalize to truly new data. To prevent this, all preprocessing steps such as normalization, scaling, or imputation must be calculated strictly using only the training folds and then applied to the test fold.

Home → All Courses → Science and Mathematics Courses → Data Science and Predictive Analytics → Flashcard

During cross-validation, what is the exact term for the phenomenon where information from the test set inadvertently influences the training process, leading to overly optimistic performance estimates?