Zero-Downtime Migrations

Expand and contract, dual writes and CDC, backfills, shadow reads, cutovers and re-sharding.

Real systems change while they run: a column gets renamed, a table is split, data moves from PostgreSQL to DynamoDB, a tenant moves to its own shard. Interviewers ask "how would you migrate this without downtime?" because it tests whether you understand that old and new code run at the same time, that data is being written during the move, and that every step must be reversible.

The core constraint

During a deploy, some servers run the old version and some the new. During a data move, writes keep arriving. So every intermediate state must work for both old and new code, and no write may be lost between the old and new location.

Schema changes: expand and contract

Never change a schema in a way that breaks the code currently running. Split every breaking change into backward-compatible steps:

Expand: add display_name -> Write both -> Backfill -> Read new -> Stop writing old -> Contract: drop full_name

Expand — add the new column (nullable, no default rewrite on large tables). Old code ignores it.
Dual-write — deploy code that writes both full_name and display_name.
Backfill — copy existing rows in small batches, throttled to protect the database.
Switch reads — deploy code that reads display_name (behind a feature flag, so you can switch back).
Stop writing the old column once nothing reads it.
Contract — drop the old column in a later release.

Each step is a separate deploy, and each can be rolled back on its own.

[!WARNING] Locking DDL on big tables Some schema changes rewrite or lock the whole table — adding a column with a default in older databases, changing a column type, adding an index without CONCURRENTLY. On a 500 GB table that's an outage. Use online schema-change tools (gh-ost, pt-online-schema-change, CREATE INDEX CONCURRENTLY) or the expand/contract pattern.

Moving data to a new store

Migrating from one database to another (or one shard to another) follows the same shape, at a larger scale:

CDC is often safer than application dual-writes. Writing to two stores from application code has the dual-write problem: one write can succeed while the other fails. Streaming changes from the old database's log to the new store guarantees the new store eventually sees every committed write.
Make backfill writes idempotent and version-aware, so a backfilled old value never overwrites a newer dual-written one (compare updated_at or a version number).

Backfills

Backfills — recomputing derived data, populating a new column, re-indexing search — share a playbook:

Process in small batches keyed by primary key ranges, with a checkpoint so the job can resume after failure.
Throttle to a fraction of database capacity, and pause automatically if replication lag or latency rises.
Make every step idempotent so re-running a batch is harmless.
For very large datasets, run them as batch jobs (Spark) reading a snapshot, then catch up from CDC.

Re-sharding and tenant moves

Moving a shard or a tenant is the same pattern with a routing switch:

Copy the data to the new shard while CDC streams ongoing changes.
When the new shard has caught up, briefly block writes for that key range (milliseconds to seconds), let CDC drain, then flip the routing table entry.
Unblock writes, now going to the new shard. Keep the old copy until you're confident.

See Sharding and Multi-Tenancy.

API and message format changes

APIs: add fields rather than changing them; version breaking changes (/v2) and run both until clients move. See API Design.
Events and messages: consumers must tolerate unknown fields; use schemas with compatibility rules (Avro or Protobuf with a schema registry) so producers can't deploy breaking changes.
Deploy order: deploy readers that understand the new format before writers start producing it.

[!INTERVIEW] Moving messages from PostgreSQL to Cassandra "We'll add a CDC stream from PostgreSQL to a consumer that writes to Cassandra, and backfill history in batches keyed by conversation ID, with writes carrying a version so backfill never overwrites newer data. Then we shadow-read for a week, comparing results, and move reads over by feature flag, 1% at a time. Once 100% of reads are on Cassandra with no mismatches, Cassandra becomes the source of truth and we stop the CDC stream. Every step can be rolled back by flipping a flag."