docs(ops): 补齐生产 runbook 和 readiness 清单

- 新增迁移回滚、AppData 退场、小宝后台任务 runbook\n- 新增生产 readiness 证据清单和 runbook placeholder 扫描\n- 更新部署文档与路线图到 V2.8 运维闭环阶段\n\nCo-Authored-By: GPT-5 Codex <codex@openai.com>
This commit is contained in:
2026-07-08 16:28:02 +08:00
parent 7837a809ca
commit 72a59f125c
10 changed files with 420 additions and 15 deletions

View File

@@ -0,0 +1,100 @@
# Migration Rollback Runbook
Use this when a production release, Prisma migration, or data migration causes failed smoke tests, missing data, bad query performance, or unsafe writes.
## First Response
1. Freeze new releases and ask product owners to pause bulk edits.
2. Capture current state before changing anything.
```bash
date -u
git rev-parse HEAD
docker compose --env-file .env.production -f docker-compose.prod.yml ps
pnpm backup:postgres -- --env-file .env.production
pnpm backup:server-data -- --env-file .env.production
```
3. Run the read-only release smoke test.
```bash
pnpm deploy:smoke -- --base-url http://localhost --expected-version "$APP_VERSION"
```
4. Check the database and latest server logs.
```bash
docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres pg_isready -U "$POSTGRES_USER" -d "$POSTGRES_DB"
docker compose --env-file .env.production -f docker-compose.prod.yml logs --tail=200 server
```
## Decision Points
- App code bad, database healthy: roll back `WEB_IMAGE` and `SERVER_IMAGE` to the previous commit image tags, then restart Compose.
- Migration applied but only additive: roll back app images first; leave schema in place if old code remains compatible.
- Migration changed or removed data: restore a fresh database from the last known-good backup into a temporary database, compare row counts, then decide whether to restore production.
- AppData compatibility issue: keep production database intact, re-enable the previous app image that still reads the AppData fallback, and preserve the failing release backup for analysis.
## App Image Rollback
Set previous image tags in `.env.production`. Use the previous successful GitHub Actions run to identify the image tags.
```bash
export PREVIOUS_WEB_IMAGE=ghcr.io/company/ftb-project-management/web:abc1234
export PREVIOUS_SERVER_IMAGE=ghcr.io/company/ftb-project-management/server:abc1234
sed -i "s|^WEB_IMAGE=.*|WEB_IMAGE=${PREVIOUS_WEB_IMAGE}|" .env.production
sed -i "s|^SERVER_IMAGE=.*|SERVER_IMAGE=${PREVIOUS_SERVER_IMAGE}|" .env.production
docker compose --env-file .env.production -f docker-compose.prod.yml pull web server
docker compose --env-file .env.production -f docker-compose.prod.yml up -d --remove-orphans
pnpm deploy:smoke -- --base-url http://localhost --expected-version abc1234
```
## Fresh Database Restore
Never restore over production until the backup has been rehearsed into a temporary database.
```bash
export BACKUP_FILE=backups/postgres/ftb_pm-postgres-20260708T120000Z.dump
pnpm restore:postgres -- \
--dry-run \
--confirm-overwrite \
--env-file .env.production \
--input "$BACKUP_FILE" \
--target-db ftb_pm_restore_check
pnpm restore:postgres -- \
--confirm-overwrite \
--env-file .env.production \
--input "$BACKUP_FILE" \
--target-db ftb_pm_restore_check
```
Compare critical row counts before touching production.
```bash
docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres psql -U "$POSTGRES_USER" -d ftb_pm_restore_check -c "select 'products' as table_name, count(*) from products union all select 'requirements', count(*) from requirements union all select 'versions', count(*) from versions union all select 'dev_tasks', count(*) from dev_tasks union all select 'test_cases', count(*) from test_cases union all select 'bugs', count(*) from bugs;"
```
If the temporary restore is healthy and production data is unsafe, restore production with explicit overwrite confirmation.
```bash
pnpm restore:postgres -- \
--confirm-overwrite \
--env-file .env.production \
--input "$BACKUP_FILE"
docker compose --env-file .env.production -f docker-compose.prod.yml up -d --remove-orphans
pnpm deploy:smoke -- --base-url http://localhost
```
## Data Risks
- PostgreSQL restore is destructive for the target database because the script terminates connections, drops the target database, recreates it, and runs `pg_restore`.
- Restoring PostgreSQL does not restore `server_data`; keep AI provider config backup files with the same incident bundle.
- AppData and relation tables can diverge during compatibility windows. Before deleting or restoring, preserve both the failing production backup and the known-good backup.
- If users continued editing during the incident, record the time window and decide whether those edits must be replayed manually after restore.
## Closeout
1. Save the failed release SHA, rollback SHA, backup file names, smoke output, and row-count evidence in the incident notes.
2. Keep the failed backup until the next successful release has completed smoke tests and one business-day observation.
3. Add a regression test or runbook correction before re-attempting the migration.