"It Passed in Staging" Is Not Evidence

Everyone talks about running database changes safely. Almost no one answers the question that comes first.

There is a large, mature conversation about how to apply a database change without breaking production: approvals, maintenance windows, rollback scripts, blue-green cutovers. All of it assumes a prior question has already been answered, and in most shops it has not. Before you decide how to run a change, you have to know what the change will break. That is the harder question, and the industry's answer to it is thin.

The usual answer is to test the change in staging and hope staging looks like production. Sometimes it does. Often it does not. The schema has drifted, the data is sampled and hides the one row that violates a new constraint, and the foreign key, trigger, view, or downstream report that actually depends on the changed object was never in staging at all. A change that passes in staging can be valid SQL, apply cleanly, and still be wrong. Passing in staging is a hopeful signal. It is not evidence.

Why staging misleads

Staging is not production, and the ways it differs are exactly the ways a database change fails.

The schema has quietly drifted. A hotfix applied straight to production, an index added by hand under load, a column type changed in one place and not the other. Once staging and production disagree about the shape of an object, a change validated against staging is validated against the wrong schema.

The data is sampled, so data-dependent failures hide. Adding a NOT NULL constraint or a unique index passes cleanly against a tidy sample and fails against the one production row that violates it.

The forgotten dependency was never in staging at all. A view built on the column you are dropping. A trigger on the table you are rewriting. A nightly export or a dashboard owned by another team that reads the object directly. The change passes precisely because the thing it would break is not present to break.

The three kinds of pre-execution evidence

If staging is not evidence, what is? Useful evidence comes in three kinds, and they are not substitutes for one another. Each answers a question the other two cannot.

• Impact mapping reads the graph the database already holds (foreign keys, view definitions, triggers) and walks outward from the object you are changing. It is the only one that can find the dependency no human remembered.
• Consumer awareness crosses the database boundary to record which services, jobs, and dashboards read an object. Those readers leave no foreign key behind, so they have to be tracked deliberately.
• A rehearsed dry run provisions a faithful, disposable copy, runs the real change against it, diffs the result, and tears it down. It catches the failure that only real execution reveals: the lock, the constraint the real-shaped data violates, the change that is subtly not what the author intended.

Analysis narrows the risk. Rehearsal confirms it. You want both.

The honesty contract

A safety report is only worth attaching to a decision if it is honest about its own limits. A report that overstates its certainty is not a safety feature; it is a liability wearing one. Partial analysis should be labeled, never faked. The report should inform an approval, not block one, because a gate that halts changes on a low-confidence signal only trains people to route around it.

The difference between a controlled change and a hopeful one is evidence. Do not confuse "it passed in staging" with "we know what this will break." Map the impact, know the consumers, rehearse the run. Then approve on evidence, not on hope.

This is the thinking behind the blast-radius analysis in Simcha DB Studio. The full argument is in our white paper, The Blast Radius Problem.