Docs / Decisions

ADR 0029: Migrations work with the release still running, and a test enforces it

Context

The Helm chart migrates in a hook Job before the Deployments roll, so the previous release keeps serving on the new schema until its pods are replaced, and keeps serving indefinitely if the rollout stalls. A pod only turns unready when the schema is behind its binary, so the old pods stay in service. docs/operations.md asked for migrations the previous version can run against, but nothing checked it, and some shipped migrations did not hold to it:

GitHub, GitLab and Stripe treat this as a rule with tooling behind it (expand and contract, lock_timeout, concurrent indexes), not a hope.

Decision

  1. Expand, then contract. A migration may only make changes the previous release still works with: add a table, a nullable column or one with a default, an index, a constraint the previous release already honours. Removing or renaming a column, or making one stricter, happens in a later release, once no running release reads or writes it the old way:
    • first release: add the new shape, write both, read the new one;
    • next release: drop the old shape.
  2. Never block writes for long. Each migration starts with SET LOCAL lock_timeout = '5s';, so it fails and can be retried rather than queue every request behind it. (LOCAL: golang-migrate sends a file as one query, which Postgres runs as one transaction, on a connection the application's pool then reuses.) A new CHECK or FOREIGN KEY on an existing table is added NOT VALID (no scan, a brief lock), and checked with VALIDATE CONSTRAINT, which lets writes continue, in the next migration: in the same one it would scan under the lock the ADD took. An index on an existing table is built CONCURRENTLY, as the only statement in its migration, since CONCURRENTLY refuses to run in a transaction; such a file needs no lock_timeout. A foreign key can reference a unique index directly, so a new unique key needs no ADD CONSTRAINT ... UNIQUE, which would build its index under a lock.
  3. A contracting change says so. A statement that removes or tightens (DROP COLUMN, DROP TABLE, RENAME, SET NOT NULL, ALTER ... TYPE) is allowed only on a line after a comment beginning -- contract: that names the release that stopped using the old shape.
  4. A test enforces 2 and 3 on every migration from 000015 on (internal/store), with no database. Migrations before it shipped and are never edited.
  5. Down migrations stay as they are: written, never run automatically. Rolling back is running the previous release on the newer schema, which rule 1 makes safe. New ones use IF EXISTS, so a down that half ran can run again; 000025 and 000029 to 000036 shipped without it and are not edited.
  6. Data is changed by background tasks, not migrations (amended 2026-10-15). A migration is one transaction, so an UPDATE or DELETE over a table holds a lock on every row it touches until the whole file commits, blocking writes to those rows for as long as the scan takes. 000028 (masking stored errors) did that across a dozen tables; it shipped and is not edited. Rewriting existing rows is an opsched task working in short batches (id IN (SELECT ... LIMIT n), each its own statement), which new code runs alongside writing the new shape. A migration may touch rows only when it can be sure they are few, such as 000054 deleting the credentials an org deletion orphaned before 000053 gave them a foreign key.

Alternatives considered

Consequences

Edit this page on GitHub