Prompts Are Deployments. Stop Shipping Them at 100%.

When an engineer changes a critical code path, no one ships it raw. There's a code review, a CI run, a feature flag, a canary rollout, telemetry on the new path, and an automatic rollback if error rate spikes. Every link in that chain exists because we learned, expensively, that "the tests passed and the code looks fine" is not the same as "this change behaves correctly in production."

Prompts get none of this treatment. The standard workflow looks like:

1. Edit the prompt string.
2. Run an eval suite locally.
3. See the new version score higher.
4. Deploy.
5. Every user gets the new prompt the moment the deploy finishes.

Step 5 is the problem, and it's the same problem we already solved for code a decade ago. We just haven't applied the solution to the artifact that most directly shapes our users' experience of an AI product.

I built a small Ruby gem called prompt_canary to fix this for myself. The interesting parts aren't Ruby-specific. They're about treating prompts the same way we treat code in production.

Why the eval suite is not enough

Three failure modes a typical eval can't catch, no matter how good it is:

Input distribution drift. Your golden set has 50 invoices. Production has 50,000. The new prompt scores better on your fifty. Then a new customer onboards whose invoices use a layout your golden set didn't represent, and the new prompt fails 40% of the time on that layout. The old one happened to handle it. You don't find out until Monday morning.

Silent model updates. You evaluated v2 against a model on October 1st. The provider quietly revised the model on October 15th: same model name, slightly different behavior. Your eval scored a model that no longer exists in the form you tested.

Latency and cost don't show up in quality evals. Your new prompt is better, but it's also 800 tokens longer. On a 50-example eval that's invisible. In production at 50,000 calls a day, your bill jumps 30% and p95 latency crosses your SLO.

The deeper issue: an eval tells you v2 is better on the eval. It does not tell you v2 is better in production. Canary deployment exists to bridge that gap for code. There's no reason prompts should be exempt.

Canary deployment, applied to prompts

The mechanics are familiar if you've ever used a feature flag or a deploy ramp. You declare multiple versions of a prompt. One is marked stable. New versions get a small slice of traffic, say 10%, or a specific user cohort like beta testers. Every call is recorded: which version served it, how long it took, how many tokens, whether it errored. Rules attached to each version describe what bad behavior looks like:

rollback_if :error_rate,  greater_than: 0.05, over: 100
rollback_if :latency_p95, greater_than: 2000, over: 100

A background monitor evaluates those rules on a schedule. If the new version's error rate exceeds 5% over the last 100 calls, or its p95 latency exceeds 2 seconds, the monitor demotes it. Traffic snaps back to the stable version within roughly a minute. No one gets paged. The user who would have hit the bad prompt at 3 a.m. doesn't.

This is the same pattern New Relic uses to roll back a backend deploy when error rate spikes on the canary cohort. When it works, no user ever sees the failure mode that caused the rollback, because only the canary slice saw it, and the canary slice got pulled before it grew.

Two design decisions worth flagging

Building prompt_canary forced two decisions that I think matter beyond Ruby.

Prompts stay in code. Every commercial prompt-management platform (Langfuse, Braintrust, PromptLayer, LangSmith) stores prompts in their database, edited through their UI, fetched by your application at runtime. That model has real strengths: a PM can iterate without a deploy, an audit trail is built in, non-engineers participate. It also has real costs: a runtime dependency on an external service for a critical-path call, vendor lock-in on your most product-critical artifact, and the loss of code-review on the artifact that most directly shapes user experience.

prompt_canary takes the opposite stance. Prompts are declared as classes in your application. They go through pull requests. They get reviewed. They get version-controlled alongside the code that calls them. The trade-off is honest: this is the wrong tool for a team where product managers iterate on prompts daily through a UI. It's the right tool for a team where prompts are owned by engineers and should be reviewed like any other production change.

Class declares intent; storage records reality. The DSL lets you write rollout percent: 10 in code. But the monitor can't edit your code when it demotes a misbehaving version. So the current rollout state lives in storage, as an override layer on top of the class declaration. The router consults both. The class says what you wanted; storage says what's actually live right now.

This means a demoted version stays demoted across deploys: you don't accidentally re-ship a known-bad version by redeploying. It also means staging and production can have different rollout state, which is correct but occasionally surprising. The CLI's status command exists specifically to make this visible: at any time, you can see what's declared in code, what's currently live, and why they differ.

What this is not

It's not an evaluation framework. Tools like Leva, Tribunal, and Promptfoo already do that well, and a gem trying to be both eval and rollout would be worse at both. prompt_canary integrates with them at the rollback-rule layer (rollback_if :eval_score, less_than: 0.75 is on the roadmap) rather than reimplementing them.

It's not A/B testing. The mechanics overlap (traffic splitting, per-variant telemetry) but the goals differ. A/B testing wants to learn which variant is better; the variants are equals. Canary deployment wants to ship a change you already believe is better, with a safety net if you're wrong; the variants are not equals. One is stable, one is on trial. The system is biased toward rolling back to stable. That bias is the whole point.

The bigger picture

The prompt management space matured a lot in 2025. The major SaaS platforms have made the case convincingly that prompts are production infrastructure, not text strings to be edited carelessly. That case is now broadly accepted; the open question is how you implement it.

For the high end of the market (large teams, cross-functional collaboration, dedicated prompt engineers) the SaaS path is probably right. For engineering teams adding AI features to existing applications, where prompts are part of the codebase and code review is non-negotiable, there's a gap. prompt_canary is a sketch of what filling that gap looks like in Ruby. The underlying ideas (declarative versioning, percentage-based rollout, metric-driven auto-rollback, an override layer that separates declared intent from live state) port to any language without modification.

The gem is on GitHub and RubyGems. It's early, it's opinionated, and it's been useful enough in my own apps that I'm sharing it. If you've shipped a prompt change at 2 a.m. and immediately regretted it, you already know the problem this is trying to solve.