How to investigate a model whose accuracy has dropped
Start with a known time window when model accuracy changed and make sure the comparison is fair.
What to do
- Check whether input fields, units, missing values, or user populations changed around that time.
- Review labeled examples in each affected group and compare them with the previous evaluation set.
- Ask the model owner to document the cause, next check and conditions for limiting usage or testing a replacement.
Reference: NIST AI Risk Management Framework.
Related MyNew service: AI & Machine Learning.