Data Drift
A change in the statistical properties of model input features between training time and deployment time, often a precursor to model performance degradation.
Data drift (also called covariate shift or input drift) occurs when the distribution of features used as model inputs changes after the model has been trained. It is the most common cause of gradual model performance degradation in production.
Causes of data drift:
- Seasonality — Regular, cyclical changes in customer behaviour (e.g., festive spending patterns in credit data)
- Economic regime changes — Interest rate rises, recessions, or inflationary periods alter borrower income and debt profiles
- Product changes — New loan products attract different customer segments than the model was trained on
- Data pipeline issues — Changes in upstream data systems alter how features are computed (a "technical" form of drift)
- Competitor actions — New competitors cherry-pick low-risk customers, shifting the risk profile of applicants
Measuring data drift:
- Population Stability Index (PSI) — Standard for credit industry; measures score distribution shift
- Kolmogorov-Smirnov (KS) test — Measures distance between two empirical distributions
- Kullback-Leibler divergence — Information-theoretic measure of distribution difference
- Wasserstein distance (Earth Mover's Distance) — Preferred for continuous distributions
Data drift vs. concept drift: Data drift is about inputs changing; concept drift is about the input-output relationship changing. Both degrade model performance, but concept drift is harder to detect because it does not require labels arriving to manifest. PSI-based monitoring catches data drift early; explanation monitoring catches concept drift.