Shipping AI Features Without Shipping AI Regressions

The first version of an AI feature is usually the easy part. A well-written prompt against a capable model will look impressive in a demo. The hard part starts the week after launch, when someone asks to tweak the prompt and nobody can say with confidence what else that change affects.

We treat prompts the way we treat code: version-controlled, reviewed, and tested against a fixed evaluation set before any change ships. That evaluation set is built from real edge cases we've collected in production, not hypothetical examples written in a planning meeting.

The evaluation itself is rarely a single pass/fail number. We track categories separately — factual accuracy, tone, refusal behavior, latency — because a prompt change that improves accuracy but quietly makes responses twice as slow is still a regression for a real-time feature.

Human review still has a role, especially for anything customer-facing or high-stakes. We build a lightweight review queue into the product itself for the first few weeks after launch, so a person is checking a sample of real outputs, not just trusting the eval suite.

The teams that get burned by AI features are almost always the ones treating the prompt as a one-time creative decision instead of a piece of infrastructure that needs the same rigor as any other code path.