The Observability Gap: Why Traditional MLOps Isn't Enough for AI Governance

In production AI systems, particularly in financial services, models can maintain 90%+ accuracy while their feature importance rankings completely flip. A credit model might shift from weighing income stability at 40% to just 15%, while compensating by over-indexing on zip code — a proxy for protected attributes. This is particularly dangerous in regulated industries where the *how* of a decision matters as much as the *what*. A loan approval with 95% confidence means nothing if the underlying logic violates fair lending laws. The root cause is distribution shift. As market conditions evolve — new customer segments emerge, economic patterns change, competitor behaviour shifts — the model adapts its internal weights to maintain performance. But these adaptations often encode biases or regulatory violations that aggregate metrics cannot detect. **Real-world example:** A major Indian bank's credit model maintained 94% accuracy for 8 months while silently penalising gig workers. Traditional monitoring showed "healthy" performance. Only segment-level PSI tracking revealed that the model's logic had drifted to favour salaried employees, creating disparate impact.