img/favicon.ico" />
MyNew Technologies

How to spot data leakage in a machine learning test

A model can test well if it learned from information that would only be available after the prediction.

What to do

  1. For each input field record when the value is known, then remove fields populated after the decision point.
  2. Separate training and test records by customer or by time when those groups would otherwise overlap.
  3. Compare performance after the split. If it drops, check duplicate records and transformations built using data from the test set.

Reference: Google guide to dividing machine learning datasets.

Related MyNew service: AI & Machine Learning.