How to benchmark a forecasting model against a simple baseline
A simple comparison shows whether a model improves on the rule people already rely on.
What to do
- Choose an error measure that reflects the business decision, then record the current rule, such as using last week's demand.
- Test both on the same unseen date range using only information available before each prediction.
- Compare overall results and inspect misses during busy periods. Record where the simpler method still works better.
Reference: Google guide to dividing machine learning datasets.
Related MyNew service: AI & Machine Learning.