ftb-project-management/docs/runbooks/migration-rollback.md

Migration Rollback Runbook

Use this when a production release, Prisma migration, or data migration causes failed smoke tests, missing data, bad query performance, or unsafe writes.

First Response

1. Freeze new releases and ask product owners to pause bulk edits.
2. Capture current state before changing anything.

date -u
git rev-parse HEAD
docker compose --env-file .env.production -f docker-compose.prod.yml ps
pnpm backup:postgres -- --env-file .env.production
pnpm backup:server-data -- --env-file .env.production

3. Run the read-only release smoke test.

pnpm deploy:smoke -- --base-url http://localhost --expected-version "$APP_VERSION"

4. Check the database and latest server logs.

docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres pg_isready -U "$POSTGRES_USER" -d "$POSTGRES_DB"
docker compose --env-file .env.production -f docker-compose.prod.yml logs --tail=200 server

Decision Points

• App code bad, database healthy: roll back WEB_IMAGE and SERVER_IMAGE to the previous commit image tags, then restart Compose.
• Migration applied but only additive: roll back app images first; leave schema in place if old code remains compatible.
• Migration changed or removed data: restore a fresh database from the last known-good backup into a temporary database, compare row counts, then decide whether to restore production.
• AppData compatibility issue: keep production database intact, re-enable the previous app image that still reads the AppData fallback, and preserve the failing release backup for analysis.

App Image Rollback

Set previous image tags in .env.production. Use the previous successful GitHub Actions run to identify the image tags.

export PREVIOUS_WEB_IMAGE=ghcr.io/company/ftb-project-management/web:abc1234
export PREVIOUS_SERVER_IMAGE=ghcr.io/company/ftb-project-management/server:abc1234
sed -i "s|^WEB_IMAGE=.*|WEB_IMAGE=${PREVIOUS_WEB_IMAGE}|" .env.production
sed -i "s|^SERVER_IMAGE=.*|SERVER_IMAGE=${PREVIOUS_SERVER_IMAGE}|" .env.production
docker compose --env-file .env.production -f docker-compose.prod.yml pull web server
docker compose --env-file .env.production -f docker-compose.prod.yml up -d --remove-orphans
pnpm deploy:smoke -- --base-url http://localhost --expected-version abc1234

Fresh Database Restore

Never restore over production until the backup has been rehearsed into a temporary database.

export BACKUP_FILE=backups/postgres/ftb_pm-postgres-20260708T120000Z.dump
pnpm restore:postgres -- \
  --dry-run \
  --confirm-overwrite \
  --env-file .env.production \
  --input "$BACKUP_FILE" \
  --target-db ftb_pm_restore_check
pnpm restore:postgres -- \
  --confirm-overwrite \
  --env-file .env.production \
  --input "$BACKUP_FILE" \
  --target-db ftb_pm_restore_check

Compare critical row counts before touching production.

docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres psql -U "$POSTGRES_USER" -d ftb_pm_restore_check -c "select 'products' as table_name, count(*) from products union all select 'requirements', count(*) from requirements union all select 'versions', count(*) from versions union all select 'dev_tasks', count(*) from dev_tasks union all select 'test_cases', count(*) from test_cases union all select 'bugs', count(*) from bugs;"

If the temporary restore is healthy and production data is unsafe, restore production with explicit overwrite confirmation.

pnpm restore:postgres -- \
  --confirm-overwrite \
  --env-file .env.production \
  --input "$BACKUP_FILE"
docker compose --env-file .env.production -f docker-compose.prod.yml up -d --remove-orphans
pnpm deploy:smoke -- --base-url http://localhost

Data Risks

• PostgreSQL restore is destructive for the target database because the script terminates connections, drops the target database, recreates it, and runs pg_restore.
• Restoring PostgreSQL does not restore server_data; keep AI provider config backup files with the same incident bundle.
• AppData and relation tables can diverge during compatibility windows. Before deleting or restoring, preserve both the failing production backup and the known-good backup.
• If users continued editing during the incident, record the time window and decide whether those edits must be replayed manually after restore.

Closeout

1. Save the failed release SHA, rollback SHA, backup file names, smoke output, and row-count evidence in the incident notes.
2. Keep the failed backup until the next successful release has completed smoke tests and one business-day observation.
3. Add a regression test or runbook correction before re-attempting the migration.