docs(ops): 补齐生产 runbook 和 readiness 清单

- 新增迁移回滚、AppData 退场、小宝后台任务 runbook\n- 新增生产 readiness 证据清单和 runbook placeholder 扫描\n- 更新部署文档与路线图到 V2.8 运维闭环阶段\n\nCo-Authored-By: GPT-5 Codex <codex@openai.com>
2026-07-08 16:28:02 +08:00
parent 7837a809ca
commit 72a59f125c
10 changed files with 420 additions and 15 deletions
--- a/docs/runbooks/appdata-retirement.md
+++ b/docs/runbooks/appdata-retirement.md
@@ -0,0 +1,72 @@
+# AppData Retirement Runbook
+
+Use this when retiring an AppData key after its domain writes have moved to relation-table APIs. The order is fixed: backup, measure, freeze writes, compare, remove fallback, archive.
+
+## Scope Gate
+
+Retire one AppData key family at a time. Good candidates have domain CRUD writes, read APIs, pagination boundaries, audit events, and a rollback path.
+
+## Preparation
+
+```bash
+pnpm backup:postgres -- --env-file .env.production
+pnpm backup:server-data -- --env-file .env.production
+pnpm deploy:smoke -- --base-url http://localhost
+```
+
+Record the key family being retired, the owning domain API, and the relation tables that replace it.
+
+## Count And Consistency Checks
+
+Run counts before disabling writes.
+
+```bash
+docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "select key, jsonb_array_length(value) as appdata_rows from app_data where key in ('products-overview','requirements','version-plans','dev-tasks','test-cases','bugs','members','task-categories','task-worklogs','overtime') order by key;"
+docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "select 'requirements' as table_name, count(*) from requirements union all select 'version_plans', count(*) from version_plans union all select 'dev_tasks', count(*) from dev_tasks union all select 'test_cases', count(*) from test_cases union all select 'bugs', count(*) from bugs;"
+```
+
+For partitioned entities, also check missing partition keys.
+
+```bash
+docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "select 'requirements_missing_product' as check_name, count(*) from requirements where product_id is null union all select 'dev_tasks_missing_version', count(*) from dev_tasks where version_id is null union all select 'test_cases_missing_version', count(*) from test_cases where version_id is null union all select 'bugs_missing_version', count(*) from bugs where version_id is null;"
+```
+
+## Disable AppData Writes
+
+1. Merge the domain-specific frontend store change that stops calling `saveServerData` for the retired key.
+2. Keep AppData read fallback for one release while relation reads are verified.
+3. Deploy and run smoke tests.
+
+```bash
+pnpm deploy:smoke -- --base-url http://localhost --expected-version "$APP_VERSION"
+```
+
+## Remove Fallback
+
+Remove AppData read fallback only after one successful release where:
+
+- Domain writes went through relation APIs.
+- AppData row counts did not grow for the retired key.
+- V2.2 read paths and page workflows returned expected data.
+- No `AppData relation sync failed` logs appeared during the observation window.
+
+## Archive
+
+Export retired keys before any later table cleanup.
+
+```bash
+docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "copy (select key, value, updated_at from app_data where key in ('dev-tasks','test-cases','bugs')) to stdout with csv header" > backups/postgres/appdata-retired-executions-20260708.csv
+```
+
+## Rollback
+
+- If relation writes fail but AppData still has fresh data, roll back to the previous app image that still writes AppData.
+- If AppData writes were already disabled and relation writes are bad, restore PostgreSQL from the backup made at the start of this runbook.
+- If only read fallback removal caused the issue, roll back app images first and leave the database unchanged.
+
+## Data Risks
+
+- Removing fallback too early can hide valid historical JSON rows that were never mapped into relation tables.
+- Re-enabling old AppData writes after relation writes have accepted new edits can overwrite newer relation state through compatibility sync.
+- AppData exports can contain business-sensitive text; store backup CSV files in the same restricted location as database dumps.
+
--- a/docs/runbooks/migration-rollback.md
+++ b/docs/runbooks/migration-rollback.md
@@ -0,0 +1,100 @@
+# Migration Rollback Runbook
+
+Use this when a production release, Prisma migration, or data migration causes failed smoke tests, missing data, bad query performance, or unsafe writes.
+
+## First Response
+
+1. Freeze new releases and ask product owners to pause bulk edits.
+2. Capture current state before changing anything.
+
+```bash
+date -u
+git rev-parse HEAD
+docker compose --env-file .env.production -f docker-compose.prod.yml ps
+pnpm backup:postgres -- --env-file .env.production
+pnpm backup:server-data -- --env-file .env.production
+```
+
+3. Run the read-only release smoke test.
+
+```bash
+pnpm deploy:smoke -- --base-url http://localhost --expected-version "$APP_VERSION"
+```
+
+4. Check the database and latest server logs.
+
+```bash
+docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres pg_isready -U "$POSTGRES_USER" -d "$POSTGRES_DB"
+docker compose --env-file .env.production -f docker-compose.prod.yml logs --tail=200 server
+```
+
+## Decision Points
+
+- App code bad, database healthy: roll back `WEB_IMAGE` and `SERVER_IMAGE` to the previous commit image tags, then restart Compose.
+- Migration applied but only additive: roll back app images first; leave schema in place if old code remains compatible.
+- Migration changed or removed data: restore a fresh database from the last known-good backup into a temporary database, compare row counts, then decide whether to restore production.
+- AppData compatibility issue: keep production database intact, re-enable the previous app image that still reads the AppData fallback, and preserve the failing release backup for analysis.
+
+## App Image Rollback
+
+Set previous image tags in `.env.production`. Use the previous successful GitHub Actions run to identify the image tags.
+
+```bash
+export PREVIOUS_WEB_IMAGE=ghcr.io/company/ftb-project-management/web:abc1234
+export PREVIOUS_SERVER_IMAGE=ghcr.io/company/ftb-project-management/server:abc1234
+sed -i "s|^WEB_IMAGE=.*|WEB_IMAGE=${PREVIOUS_WEB_IMAGE}|" .env.production
+sed -i "s|^SERVER_IMAGE=.*|SERVER_IMAGE=${PREVIOUS_SERVER_IMAGE}|" .env.production
+docker compose --env-file .env.production -f docker-compose.prod.yml pull web server
+docker compose --env-file .env.production -f docker-compose.prod.yml up -d --remove-orphans
+pnpm deploy:smoke -- --base-url http://localhost --expected-version abc1234
+```
+
+## Fresh Database Restore
+
+Never restore over production until the backup has been rehearsed into a temporary database.
+
+```bash
+export BACKUP_FILE=backups/postgres/ftb_pm-postgres-20260708T120000Z.dump
+pnpm restore:postgres -- \
+  --dry-run \
+  --confirm-overwrite \
+  --env-file .env.production \
+  --input "$BACKUP_FILE" \
+  --target-db ftb_pm_restore_check
+pnpm restore:postgres -- \
+  --confirm-overwrite \
+  --env-file .env.production \
+  --input "$BACKUP_FILE" \
+  --target-db ftb_pm_restore_check
+```
+
+Compare critical row counts before touching production.
+
+```bash
+docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres psql -U "$POSTGRES_USER" -d ftb_pm_restore_check -c "select 'products' as table_name, count(*) from products union all select 'requirements', count(*) from requirements union all select 'versions', count(*) from versions union all select 'dev_tasks', count(*) from dev_tasks union all select 'test_cases', count(*) from test_cases union all select 'bugs', count(*) from bugs;"
+```
+
+If the temporary restore is healthy and production data is unsafe, restore production with explicit overwrite confirmation.
+
+```bash
+pnpm restore:postgres -- \
+  --confirm-overwrite \
+  --env-file .env.production \
+  --input "$BACKUP_FILE"
+docker compose --env-file .env.production -f docker-compose.prod.yml up -d --remove-orphans
+pnpm deploy:smoke -- --base-url http://localhost
+```
+
+## Data Risks
+
+- PostgreSQL restore is destructive for the target database because the script terminates connections, drops the target database, recreates it, and runs `pg_restore`.
+- Restoring PostgreSQL does not restore `server_data`; keep AI provider config backup files with the same incident bundle.
+- AppData and relation tables can diverge during compatibility windows. Before deleting or restoring, preserve both the failing production backup and the known-good backup.
+- If users continued editing during the incident, record the time window and decide whether those edits must be replayed manually after restore.
+
+## Closeout
+
+1. Save the failed release SHA, rollback SHA, backup file names, smoke output, and row-count evidence in the incident notes.
+2. Keep the failed backup until the next successful release has completed smoke tests and one business-day observation.
+3. Add a regression test or runbook correction before re-attempting the migration.
+
--- a/docs/runbooks/xiaobao-background-jobs.md
+++ b/docs/runbooks/xiaobao-background-jobs.md
@@ -0,0 +1,55 @@
+# Xiaobao Background Jobs Runbook
+
+Current V2.8 production monitoring supports Xiaobao staleness detection. The first production implementation is still page-triggered: opening `/xiaobao-warning` computes risk, saves snapshots, and lets V2.3 sync refresh summaries. A future scheduler must preserve the same idempotent data contract.
+
+## Alert Triage
+
+When `FtbXiaobaoSummaryStale` fires:
+
+```bash
+docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "select version_id, dirty, updated_at, recomputed_at, risk_level, risk_score from xiaobao_risk_summaries where dirty = true or updated_at < now() - interval '6 hours' order by updated_at asc limit 20;"
+docker compose --env-file .env.production -f docker-compose.prod.yml logs --tail=200 server | grep -E "xiaobao|AppData relation sync failed|Slow Prisma query"
+```
+
+Decision points:
+
+- Rows are dirty after active version edits: ask a manager to open `/xiaobao-warning` once, then verify summaries refresh.
+- Rows remain dirty and server logs show sync failures: treat as AppData relation sync incident and follow `migration-rollback.md`.
+- Rows are stale but no product release is near: keep monitoring and schedule a manual refresh before the next release decision meeting.
+- Rows are stale for a release due today: refresh manually and have the release owner review the resulting risk explanation before ship/no-ship decision.
+
+## Manual Refresh Path
+
+1. Log in as a user with `xiaobao.warning:manage`.
+2. Open `/xiaobao-warning`.
+3. Wait until AI interpretation status is no longer generating for high-risk versions.
+4. Re-run the stale-summary query.
+5. Run the release smoke test.
+
+```bash
+pnpm deploy:smoke -- --base-url http://localhost
+```
+
+## Future Scheduler Rules
+
+When a background job is introduced, it must:
+
+- Read unfinished versions by relation-table scope, not by full AppData document scan.
+- Use one idempotency key per `versionId + riskSignature + snapshotDate`.
+- Write snapshots append-only and upsert summaries by `version_id`.
+- Mark failures with structured logs containing `xiaobao background job failed`.
+- Retry transient AI failures with backoff and keep rule-based risk output even when AI interpretation fails.
+- Never mutate Version, Requirement, DevTask, TestCase, Bug, or Member data.
+
+## Monitoring Expectations
+
+- `FtbXiaobaoSummaryStale` alerts on dirty or older-than-6-hour summaries.
+- `FtbJobFailureLogBurst` alerts when sync or future job failure log counters increase.
+- Grafana dashboard shows the stale summary count and matching server warning/error logs.
+
+## Data Risks
+
+- Recomputing Xiaobao summaries can change release risk badges and manager decisions; record manual refresh time in release notes.
+- AI interpretation cache is explanatory only. Do not restore or delete business entities to fix a bad explanation.
+- If stale summaries are caused by relation sync failure, refreshing the page can mask the symptom without fixing the underlying sync path. Preserve logs before restarting services.
+