ftb-project-management/docs/runbooks/migration-rollback.md

# Migration Rollback Runbook

Use this when a production release, Prisma migration, or data migration causes failed smoke tests, missing data, bad query performance, or unsafe writes.

## First Response

1. Freeze new releases and ask product owners to pause bulk edits.
2. Capture current state before changing anything.

```bash
date -u
git rev-parse HEAD
docker compose --env-file .env.production -f docker-compose.prod.yml ps
pnpm backup:postgres -- --env-file .env.production
pnpm backup:server-data -- --env-file .env.production
```

3. Run the read-only release smoke test.

```bash
pnpm deploy:smoke -- --base-url http://localhost --expected-version "$APP_VERSION"
```

4. Check the database and latest server logs.

```bash
docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres pg_isready -U "$POSTGRES_USER" -d "$POSTGRES_DB"
docker compose --env-file .env.production -f docker-compose.prod.yml logs --tail=200 server
```

## Decision Points

- App code bad, database healthy: roll back `WEB_IMAGE` and `SERVER_IMAGE` to the previous commit image tags, then restart Compose.
- Migration applied but only additive: roll back app images first; leave schema in place if old code remains compatible.
- Migration changed or removed data: restore a fresh database from the last known-good backup into a temporary database, compare row counts, then decide whether to restore production.
- AppData compatibility issue: keep production database intact, re-enable the previous app image that still reads the AppData fallback, and preserve the failing release backup for analysis.

## App Image Rollback

Set previous image tags in `.env.production`. Use the previous successful GitHub Actions run to identify the image tags.

```bash
export PREVIOUS_WEB_IMAGE=ghcr.io/company/ftb-project-management/web:abc1234
export PREVIOUS_SERVER_IMAGE=ghcr.io/company/ftb-project-management/server:abc1234
sed -i "s|^WEB_IMAGE=.*|WEB_IMAGE=${PREVIOUS_WEB_IMAGE}|" .env.production
sed -i "s|^SERVER_IMAGE=.*|SERVER_IMAGE=${PREVIOUS_SERVER_IMAGE}|" .env.production
docker compose --env-file .env.production -f docker-compose.prod.yml pull web server
docker compose --env-file .env.production -f docker-compose.prod.yml up -d --remove-orphans
pnpm deploy:smoke -- --base-url http://localhost --expected-version abc1234
```

## Fresh Database Restore

Never restore over production until the backup has been rehearsed into a temporary database.

```bash
export BACKUP_FILE=backups/postgres/ftb_pm-postgres-20260708T120000Z.dump
pnpm restore:postgres -- \
  --dry-run \
  --confirm-overwrite \
  --env-file .env.production \
  --input "$BACKUP_FILE" \
  --target-db ftb_pm_restore_check
pnpm restore:postgres -- \
  --confirm-overwrite \
  --env-file .env.production \
  --input "$BACKUP_FILE" \
  --target-db ftb_pm_restore_check
```

Compare critical row counts before touching production.

```bash
docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres psql -U "$POSTGRES_USER" -d ftb_pm_restore_check -c "select 'products' as table_name, count(*) from products union all select 'requirements', count(*) from requirements union all select 'versions', count(*) from versions union all select 'dev_tasks', count(*) from dev_tasks union all select 'test_cases', count(*) from test_cases union all select 'bugs', count(*) from bugs;"
```

If the temporary restore is healthy and production data is unsafe, restore production with explicit overwrite confirmation.

```bash
pnpm restore:postgres -- \
  --confirm-overwrite \
  --env-file .env.production \
  --input "$BACKUP_FILE"
docker compose --env-file .env.production -f docker-compose.prod.yml up -d --remove-orphans
pnpm deploy:smoke -- --base-url http://localhost
```

## Data Risks

- PostgreSQL restore is destructive for the target database because the script terminates connections, drops the target database, recreates it, and runs `pg_restore`.
- Restoring PostgreSQL does not restore `server_data`; keep AI provider config backup files with the same incident bundle.
- AppData and relation tables can diverge during compatibility windows. Before deleting or restoring, preserve both the failing production backup and the known-good backup.
- If users continued editing during the incident, record the time window and decide whether those edits must be replayed manually after restore.

## Closeout

1. Save the failed release SHA, rollback SHA, backup file names, smoke output, and row-count evidence in the incident notes.
2. Keep the failed backup until the next successful release has completed smoke tests and one business-day observation.
3. Add a regression test or runbook correction before re-attempting the migration.