How to spot data leakage in a machine learning test
A model can test well if it learned from information that would only be available after the prediction.
What to do
- For each input field record when the value is known, then remove fields populated after the decision point.
- Separate training and test records by customer or by time when those groups would otherwise overlap.
- Compare performance after the split. If it drops, check duplicate records and transformations built using data from the test set.
Reference: Google guide to dividing machine learning datasets.
Related MyNew service: AI & Machine Learning.