From 72a59f125c04375cabe4ad1477e847457cdf00c1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E3=80=82?= Date: Wed, 8 Jul 2026 16:28:02 +0800 Subject: [PATCH] =?UTF-8?q?docs(ops):=20=E8=A1=A5=E9=BD=90=E7=94=9F?= =?UTF-8?q?=E4=BA=A7=20runbook=20=E5=92=8C=20readiness=20=E6=B8=85?= =?UTF-8?q?=E5=8D=95?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - 新增迁移回滚、AppData 退场、小宝后台任务 runbook\n- 新增生产 readiness 证据清单和 runbook placeholder 扫描\n- 更新部署文档与路线图到 V2.8 运维闭环阶段\n\nCo-Authored-By: GPT-5 Codex --- docs/deployment.md | 15 ++++ docs/production-readiness.md | 24 ++++++ docs/roadmap.md | 35 ++++---- docs/runbooks/appdata-retirement.md | 72 ++++++++++++++++ docs/runbooks/migration-rollback.md | 100 +++++++++++++++++++++++ docs/runbooks/xiaobao-background-jobs.md | 55 +++++++++++++ package.json | 1 + scripts/check-runbook-placeholders.mjs | 87 ++++++++++++++++++++ scripts/ops-scripts.test.mjs | 11 +++ scripts/verify-production-deploy.mjs | 35 ++++++++ 10 files changed, 420 insertions(+), 15 deletions(-) create mode 100644 docs/production-readiness.md create mode 100644 docs/runbooks/appdata-retirement.md create mode 100644 docs/runbooks/migration-rollback.md create mode 100644 docs/runbooks/xiaobao-background-jobs.md create mode 100644 scripts/check-runbook-placeholders.mjs diff --git a/docs/deployment.md b/docs/deployment.md index ec1a67c..7603237 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -283,6 +283,21 @@ pnpm restore:postgres -- \ --confirm-overwrite ``` +## 运维 Runbooks + +生产发布、迁移和异常处置优先使用这些手册： + +- `docs/runbooks/migration-rollback.md`：发布失败、迁移失败、数据恢复和镜像回滚。 +- `docs/runbooks/appdata-retirement.md`：AppData key 分阶段退场、双读核对、归档和回滚。 +- `docs/runbooks/xiaobao-background-jobs.md`：小宝摘要 stale 告警、手动刷新和未来后台任务规则。 +- `docs/production-readiness.md`：生产发布前后证据清单。 + +提交前运行 runbook 扫描： + +```bash +pnpm docs:check-runbooks +``` + ## 升级流程 ```bash diff --git a/docs/production-readiness.md b/docs/production-readiness.md new file mode 100644 index 0000000..afcc2ab --- /dev/null +++ b/docs/production-readiness.md @@ -0,0 +1,24 @@ +# Production Readiness Checklist + +Use this before a production release and again after the release smoke test. Each item requires evidence, not a verbal assertion. + +| Area | Gate | Evidence | +| --- | --- | --- | +| Backup and restore | PostgreSQL backup dry-run and server_data backup dry-run are reviewed. | `pnpm backup:postgres -- --dry-run --env-file .env.production` and `pnpm backup:server-data -- --dry-run --env-file .env.production` output saved in release notes. | +| Backup and restore | Restore command refuses overwrite without explicit confirmation. | `pnpm restore:postgres -- --env-file .env.production --input backups/postgres/ftb_pm-postgres-20260708T120000Z.dump` exits non-zero and names `--confirm-overwrite`. | +| Backup and restore | Fresh DB restore rehearsal completed before destructive restore. | Temporary database restore command and row-count comparison from `docs/runbooks/migration-rollback.md`. | +| Smoke tests | Release smoke is wired into GitHub Actions. | `.github/workflows/deploy-production.yml` runs `scripts/smoke-test-release.mjs` with `--expected-version`. | +| Smoke tests | Manual smoke can be run against the target. | `pnpm deploy:smoke -- --base-url http://localhost --expected-version "$APP_VERSION"` output. | +| Monitoring | Monitoring profile renders and can start without committed secrets. | `docker compose --env-file .env.production -f docker-compose.prod.yml --profile monitoring config` exits 0. | +| Monitoring | Required alert rules exist. | `FtbPostgresDown`, `FtbDiskPressure`, `FtbSlowApiLogBurst`, `FtbSlowPrismaLogBurst`, `FtbJobFailureLogBurst`, and `FtbXiaobaoSummaryStale` visible in Prometheus. | +| Audit | Domain write APIs record actor and resource scope where implemented. | API request sample or audit log sample for Product, Requirement, Project, Version, execution entities, and dictionary writes. | +| RBAC | Project and version operations enforce Owner, Admin, Member, Viewer boundaries where enabled. | Permission matrix test result or manual account walkthrough attached to release notes. | +| Consistency | AppData and relation tables are checked for migrated domains. | Count and missing-partition-key queries from `docs/runbooks/appdata-retirement.md`. | +| Performance | Slow API and slow Prisma thresholds are configured. | `API_SLOW_REQUEST_MS` and `PRISMA_SLOW_QUERY_MS` values recorded, plus Grafana slow-log panels checked. | +| Performance | V2.2 hot reads avoid full-table scans. | Requirement pool smoke uses `productId`; query plan or service test evidence attached for high-volume domains. | +| Post-release | Runtime version matches the release SHA. | `/api/v1/health/version` payload or `pnpm deploy:check-runtime` output. | +| Post-release | Product, requirement, workspace, Xiaobao, and AI config read paths respond. | `pnpm deploy:smoke` output attached. | +| Post-release | On-call rollback path is known. | `docs/runbooks/migration-rollback.md` link included in release notes. | + +Release owner signs off only after all required evidence is attached to the release notes or incident record. + diff --git a/docs/roadmap.md b/docs/roadmap.md index be4761c..f2d7a7c 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -1,17 +1,16 @@ # 开发路线图 -## 当前阶段：V2.4 — 领域 CRUD 主写迁移 +## 当前阶段：V2.8 — 生产硬化稳定版 + 运维闭环 -V2.4 的目标是把业务主数据源从 AppData JSONB 文档切换到 PostgreSQL 领域关系表。AppData 继续保留为迁移、回填、兼容读取和排查入口，但不再作为长期主写入源；新增业务能力必须优先设计关系表、领域 CRUD API、索引/分区键和权限边界。V2.4 做逐领域主写迁移，并随 CRUD 入口埋好基础权限、`actorId` 和审计事件骨架；完整 RBAC、审计覆盖和 AppData 退场收口放到 V2.5。 +V2.8 的目标是在既有生产 CI/CD 基线上补齐运维闭环：备份恢复演练、发布 smoke test、监控告警、日志检索、迁移回滚 runbook、AppData 退场 runbook、小宝后台化 runbook 和生产 readiness 证据清单。当前执行前提是 V2.4 领域 CRUD 主写迁移已由上游验收完成；本阶段不重新设计业务流程，专注把生产发布和故障处置做成可验证、可复盘、可回滚的标准流程。 ### 当前重点 -1. **领域写 API**：按模块补齐 Product/Project/Version/Requirement/VersionPlan/DevTask/TestCase/Bug/Member/TaskCategory/Worklog/Overtime 的关系表写入 API。 -2. **前端持久化切换**：Zustand store 保留状态管理，但保存入口从 `saveServerData(key)` 迁到领域 API；读取优先 V2.2/V2.3 关系表接口。 -3. **AppData 迁移工具化**：把 AppData → 关系表同步做成可重复运行、可计数校验、可回滚的运维脚本，不提交真实 `.env`。 -4. **主写切换闸门**：每个领域完成双读核对后，先停止该领域 JSON 主写入，再进入 V2.5 的 fallback 移除和归档退场。 -5. **数据一致性校验**：为每个迁移领域补 counts、抽样记录、孤儿引用、分区键完整性和唯一约束校验。 -6. **权限/审计骨架**：领域 API 必须携带当前用户、产品/项目/版本作用域和审计事件入口，避免 V2.5 做 RBAC 时返工。 +1. **备份恢复自动化**：PostgreSQL dump、`server_data` volume 备份、fresh DB restore dry-run 和显式覆盖确认。 +2. **发布 smoke test**：GitHub Actions 部署后自动校验 runtime version、前端根页、产品页、产品 API、V2.2 读路径和 AI 配置。 +3. **监控告警基线**：Prometheus/Grafana/Loki/Promtail 可选 profile，覆盖慢 API、慢 Prisma、任务失败、小宝摘要 stale、磁盘压力和 DB 可用性。 +4. **迁移和后台任务 runbook**：迁移回滚、AppData 退场、小宝后台化处置步骤、决策点和数据风险。 +5. **生产 readiness 清单**：backup/restore、smoke、monitoring、audit、RBAC、consistency、performance 和 post-release verification 都要有证据项。 ## V2 分阶段交付链路 @@ -34,12 +33,18 @@ V2.4 的目标是把业务主数据源从 AppData JSONB 文档切换到 PostgreS - 版本详情已有需求、调研、产品方案、UI、开发任务、测试用例、Bug、概览等核心 Tab；渲染重的路径优先接入 V2.2 关系表快读，并保留 AppData fallback。 - 后端已落地 Product、Requirement 领域 CRUD，DataModule AppData 乐观锁，V2.2 快读 API，V2.3 AppData 写后同步关系表，AI Provider 抽象和健康版本接口。 - Prisma schema 已包含 Product、Project、Version、Requirement、VersionPlan、DevTask、TestCase、Bug、WorkActivity、Xiaobao、AiLog、AppData 等关系模型；高增长表的分区 migration 已落地。 -- 主写入源仍处在兼容窗口：多数前端 store 继续通过 `apps/web/lib/server-data.ts` 的 `loadServerData` / `saveServerData` 写 AppData；`useProductStore` 仍以 `products-overview` 文档作为产品/项目/版本树主写入。 -- Project、Version、VersionPlan、DevTask、TestCase、Bug、Member、TaskCategory、TaskWorklog、Overtime 等领域写 API 尚未完整替代 AppData Store。若后续称为 V2.4，应理解为“领域 CRUD 迁移阶段”，不是 V2.3 已完成内容。 -- `packages/shared` 中仍保留早期枚举口径；切换领域 API 时需要统一为当前前端业务状态机。 +- 本 V2.8 执行线程以前提“V2.4 领域 CRUD 主写迁移已完成验收”推进；本阶段不重新逐项复核领域 API 清单。 +- V2.8 新增运维交付物集中在 `scripts/`、`.github/workflows/deploy-production.yml`、`deploy/monitoring/`、`docs/runbooks/`、`docs/deployment.md` 和 `docs/production-readiness.md`。 +- AppData 退场、RBAC/审计、性能和小宝后台化仍通过 production readiness 证据项追踪，避免把运维稳定版误当成业务治理已全部完成。 ### 已完成（按时间倒序） +**2026-07-08** +- V2.8 production ops closure started: added PostgreSQL backup, fresh DB restore with explicit overwrite confirmation, and `server_data` volume backup automation. +- Added release smoke suite and wired GitHub Actions deployment verification to runtime version, frontend root, products, V2.2 read path, and AI config checks. +- Added optional monitoring profile with Prometheus, Grafana, Loki, Promtail, postgres-exporter, node-exporter, cAdvisor, and blackbox-exporter. +- Added migration rollback, AppData retirement, Xiaobao background jobs runbooks, and production readiness evidence checklist. + **2026-07-06** - Added production CI/CD flow: GitHub Actions builds `web` and `server` Docker images, pushes immutable commit-SHA tags to GHCR, deploys by SSH, pulls images on the server, runs `pnpm --filter server db:deploy`, restarts Compose, and verifies `/api/v1/health/version`. - Added runtime version metadata: backend `GET /api/v1/health/version`, Docker build args/env, and a frontend refresh banner when browser assets are older than the server runtime. @@ -117,12 +122,12 @@ V2.4 的目标是把业务主数据源从 AppData JSONB 文档切换到 PostgreS ### 进行中 -- V2.4 领域 CRUD 主写迁移：从 AppData JSONB 主写入切换到关系表 API，并随 API 落基础权限、审计和查询性能边界。 +- V2.8 生产硬化稳定版：补齐备份恢复、发布 smoke、监控告警、日志检索、迁移回滚和运维证据闭环。 - 项目详情页 VersionCard 状态胶囊数据联动（部分已完成） ## V2 — 后端接入 -NestJS + Prisma + PostgreSQL 已接入到 V2.3。第一阶段用 `app_data` JSONB 文档表承接现有 store 数据形状，避免浏览器清站点数据导致业务数据丢失；第二阶段已建立分区关系表、V2.2 快读 API 和 V2.3 AppData 写后同步。当前 V2.4 才是逐领域启用写 API，让前端 store 从 AppData 主写入迁移到领域 CRUD；AppData 后续只保留为迁移兼容层。 +NestJS + Prisma + PostgreSQL 已完成 V2.1 至 V2.4 的后端迁移主线。第一阶段用 `app_data` JSONB 文档表承接现有 store 数据形状，避免浏览器清站点数据导致业务数据丢失；第二阶段建立分区关系表、V2.2 快读 API 和 V2.3 AppData 写后同步；V2.4 完成领域 CRUD 主写迁移验收。V2.8 不再新增业务主写迁移范围，而是把生产运维、备份恢复、监控告警和回滚手册补齐。 ### 关键任务 @@ -141,7 +146,7 @@ NestJS + Prisma + PostgreSQL 已接入到 V2.3。第一阶段用 `app_data` JSON 当前不做本地导入导出。清站点数据后浏览器旧数据无法恢复，后续新增数据直接写入 PostgreSQL。若以后需要迁移旧浏览器数据，再单独做管理员导入工具。 -## V2.4 — 领域 CRUD 迁移（当前阶段） +## V2.4 — 领域 CRUD 迁移（已完成前提） 目标是让关系表从“快读 + AppData 同步副本”逐步升级为主写入路径。迁移顺序应优先选择写入频率高、实体边界清晰、已经在 V2.2 mapper 中稳定的领域： @@ -221,7 +226,7 @@ V2.4 推进前必须先统一 `packages/shared` 的状态枚举与当前前端 |------|------| | V1 业务流程打磨 | 进行中 | | V1 朋友试用反馈 | 持续中 | -| V2 后端接入 | 进行中（V2.1/V2.2/V2.3 已完成，V2.4 主写迁移中） | +| V2 后端接入 | V2.1-V2.4 已完成；V2.8 运维闭环进行中 | | V3 AI 集成 | 等 V2 数据沉淀 | | 公开发布 | TBD | **2026-06-26** diff --git a/docs/runbooks/appdata-retirement.md b/docs/runbooks/appdata-retirement.md new file mode 100644 index 0000000..ec2d1ac --- /dev/null +++ b/docs/runbooks/appdata-retirement.md @@ -0,0 +1,72 @@ +# AppData Retirement Runbook + +Use this when retiring an AppData key after its domain writes have moved to relation-table APIs. The order is fixed: backup, measure, freeze writes, compare, remove fallback, archive. + +## Scope Gate + +Retire one AppData key family at a time. Good candidates have domain CRUD writes, read APIs, pagination boundaries, audit events, and a rollback path. + +## Preparation + +```bash +pnpm backup:postgres -- --env-file .env.production +pnpm backup:server-data -- --env-file .env.production +pnpm deploy:smoke -- --base-url http://localhost +``` + +Record the key family being retired, the owning domain API, and the relation tables that replace it. + +## Count And Consistency Checks + +Run counts before disabling writes. + +```bash +docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "select key, jsonb_array_length(value) as appdata_rows from app_data where key in ('products-overview','requirements','version-plans','dev-tasks','test-cases','bugs','members','task-categories','task-worklogs','overtime') order by key;" +docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "select 'requirements' as table_name, count(*) from requirements union all select 'version_plans', count(*) from version_plans union all select 'dev_tasks', count(*) from dev_tasks union all select 'test_cases', count(*) from test_cases union all select 'bugs', count(*) from bugs;" +``` + +For partitioned entities, also check missing partition keys. + +```bash +docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "select 'requirements_missing_product' as check_name, count(*) from requirements where product_id is null union all select 'dev_tasks_missing_version', count(*) from dev_tasks where version_id is null union all select 'test_cases_missing_version', count(*) from test_cases where version_id is null union all select 'bugs_missing_version', count(*) from bugs where version_id is null;" +``` + +## Disable AppData Writes + +1. Merge the domain-specific frontend store change that stops calling `saveServerData` for the retired key. +2. Keep AppData read fallback for one release while relation reads are verified. +3. Deploy and run smoke tests. + +```bash +pnpm deploy:smoke -- --base-url http://localhost --expected-version "$APP_VERSION" +``` + +## Remove Fallback + +Remove AppData read fallback only after one successful release where: + +- Domain writes went through relation APIs. +- AppData row counts did not grow for the retired key. +- V2.2 read paths and page workflows returned expected data. +- No `AppData relation sync failed` logs appeared during the observation window. + +## Archive + +Export retired keys before any later table cleanup. + +```bash +docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "copy (select key, value, updated_at from app_data where key in ('dev-tasks','test-cases','bugs')) to stdout with csv header" > backups/postgres/appdata-retired-executions-20260708.csv +``` + +## Rollback + +- If relation writes fail but AppData still has fresh data, roll back to the previous app image that still writes AppData. +- If AppData writes were already disabled and relation writes are bad, restore PostgreSQL from the backup made at the start of this runbook. +- If only read fallback removal caused the issue, roll back app images first and leave the database unchanged. + +## Data Risks + +- Removing fallback too early can hide valid historical JSON rows that were never mapped into relation tables. +- Re-enabling old AppData writes after relation writes have accepted new edits can overwrite newer relation state through compatibility sync. +- AppData exports can contain business-sensitive text; store backup CSV files in the same restricted location as database dumps. + diff --git a/docs/runbooks/migration-rollback.md b/docs/runbooks/migration-rollback.md new file mode 100644 index 0000000..2ffac4b --- /dev/null +++ b/docs/runbooks/migration-rollback.md @@ -0,0 +1,100 @@ +# Migration Rollback Runbook + +Use this when a production release, Prisma migration, or data migration causes failed smoke tests, missing data, bad query performance, or unsafe writes. + +## First Response + +1. Freeze new releases and ask product owners to pause bulk edits. +2. Capture current state before changing anything. + +```bash +date -u +git rev-parse HEAD +docker compose --env-file .env.production -f docker-compose.prod.yml ps +pnpm backup:postgres -- --env-file .env.production +pnpm backup:server-data -- --env-file .env.production +``` + +3. Run the read-only release smoke test. + +```bash +pnpm deploy:smoke -- --base-url http://localhost --expected-version "$APP_VERSION" +``` + +4. Check the database and latest server logs. + +```bash +docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres pg_isready -U "$POSTGRES_USER" -d "$POSTGRES_DB" +docker compose --env-file .env.production -f docker-compose.prod.yml logs --tail=200 server +``` + +## Decision Points + +- App code bad, database healthy: roll back `WEB_IMAGE` and `SERVER_IMAGE` to the previous commit image tags, then restart Compose. +- Migration applied but only additive: roll back app images first; leave schema in place if old code remains compatible. +- Migration changed or removed data: restore a fresh database from the last known-good backup into a temporary database, compare row counts, then decide whether to restore production. +- AppData compatibility issue: keep production database intact, re-enable the previous app image that still reads the AppData fallback, and preserve the failing release backup for analysis. + +## App Image Rollback + +Set previous image tags in `.env.production`. Use the previous successful GitHub Actions run to identify the image tags. + +```bash +export PREVIOUS_WEB_IMAGE=ghcr.io/company/ftb-project-management/web:abc1234 +export PREVIOUS_SERVER_IMAGE=ghcr.io/company/ftb-project-management/server:abc1234 +sed -i "s|^WEB_IMAGE=.*|WEB_IMAGE=${PREVIOUS_WEB_IMAGE}|" .env.production +sed -i "s|^SERVER_IMAGE=.*|SERVER_IMAGE=${PREVIOUS_SERVER_IMAGE}|" .env.production +docker compose --env-file .env.production -f docker-compose.prod.yml pull web server +docker compose --env-file .env.production -f docker-compose.prod.yml up -d --remove-orphans +pnpm deploy:smoke -- --base-url http://localhost --expected-version abc1234 +``` + +## Fresh Database Restore + +Never restore over production until the backup has been rehearsed into a temporary database. + +```bash +export BACKUP_FILE=backups/postgres/ftb_pm-postgres-20260708T120000Z.dump +pnpm restore:postgres -- \ + --dry-run \ + --confirm-overwrite \ + --env-file .env.production \ + --input "$BACKUP_FILE" \ + --target-db ftb_pm_restore_check +pnpm restore:postgres -- \ + --confirm-overwrite \ + --env-file .env.production \ + --input "$BACKUP_FILE" \ + --target-db ftb_pm_restore_check +``` + +Compare critical row counts before touching production. + +```bash +docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres psql -U "$POSTGRES_USER" -d ftb_pm_restore_check -c "select 'products' as table_name, count(*) from products union all select 'requirements', count(*) from requirements union all select 'versions', count(*) from versions union all select 'dev_tasks', count(*) from dev_tasks union all select 'test_cases', count(*) from test_cases union all select 'bugs', count(*) from bugs;" +``` + +If the temporary restore is healthy and production data is unsafe, restore production with explicit overwrite confirmation. + +```bash +pnpm restore:postgres -- \ + --confirm-overwrite \ + --env-file .env.production \ + --input "$BACKUP_FILE" +docker compose --env-file .env.production -f docker-compose.prod.yml up -d --remove-orphans +pnpm deploy:smoke -- --base-url http://localhost +``` + +## Data Risks + +- PostgreSQL restore is destructive for the target database because the script terminates connections, drops the target database, recreates it, and runs `pg_restore`. +- Restoring PostgreSQL does not restore `server_data`; keep AI provider config backup files with the same incident bundle. +- AppData and relation tables can diverge during compatibility windows. Before deleting or restoring, preserve both the failing production backup and the known-good backup. +- If users continued editing during the incident, record the time window and decide whether those edits must be replayed manually after restore. + +## Closeout + +1. Save the failed release SHA, rollback SHA, backup file names, smoke output, and row-count evidence in the incident notes. +2. Keep the failed backup until the next successful release has completed smoke tests and one business-day observation. +3. Add a regression test or runbook correction before re-attempting the migration. + diff --git a/docs/runbooks/xiaobao-background-jobs.md b/docs/runbooks/xiaobao-background-jobs.md new file mode 100644 index 0000000..c2fce9f --- /dev/null +++ b/docs/runbooks/xiaobao-background-jobs.md @@ -0,0 +1,55 @@ +# Xiaobao Background Jobs Runbook + +Current V2.8 production monitoring supports Xiaobao staleness detection. The first production implementation is still page-triggered: opening `/xiaobao-warning` computes risk, saves snapshots, and lets V2.3 sync refresh summaries. A future scheduler must preserve the same idempotent data contract. + +## Alert Triage + +When `FtbXiaobaoSummaryStale` fires: + +```bash +docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "select version_id, dirty, updated_at, recomputed_at, risk_level, risk_score from xiaobao_risk_summaries where dirty = true or updated_at < now() - interval '6 hours' order by updated_at asc limit 20;" +docker compose --env-file .env.production -f docker-compose.prod.yml logs --tail=200 server | grep -E "xiaobao|AppData relation sync failed|Slow Prisma query" +``` + +Decision points: + +- Rows are dirty after active version edits: ask a manager to open `/xiaobao-warning` once, then verify summaries refresh. +- Rows remain dirty and server logs show sync failures: treat as AppData relation sync incident and follow `migration-rollback.md`. +- Rows are stale but no product release is near: keep monitoring and schedule a manual refresh before the next release decision meeting. +- Rows are stale for a release due today: refresh manually and have the release owner review the resulting risk explanation before ship/no-ship decision. + +## Manual Refresh Path + +1. Log in as a user with `xiaobao.warning:manage`. +2. Open `/xiaobao-warning`. +3. Wait until AI interpretation status is no longer generating for high-risk versions. +4. Re-run the stale-summary query. +5. Run the release smoke test. + +```bash +pnpm deploy:smoke -- --base-url http://localhost +``` + +## Future Scheduler Rules + +When a background job is introduced, it must: + +- Read unfinished versions by relation-table scope, not by full AppData document scan. +- Use one idempotency key per `versionId + riskSignature + snapshotDate`. +- Write snapshots append-only and upsert summaries by `version_id`. +- Mark failures with structured logs containing `xiaobao background job failed`. +- Retry transient AI failures with backoff and keep rule-based risk output even when AI interpretation fails. +- Never mutate Version, Requirement, DevTask, TestCase, Bug, or Member data. + +## Monitoring Expectations + +- `FtbXiaobaoSummaryStale` alerts on dirty or older-than-6-hour summaries. +- `FtbJobFailureLogBurst` alerts when sync or future job failure log counters increase. +- Grafana dashboard shows the stale summary count and matching server warning/error logs. + +## Data Risks + +- Recomputing Xiaobao summaries can change release risk badges and manager decisions; record manual refresh time in release notes. +- AI interpretation cache is explanatory only. Do not restore or delete business entities to fix a bad explanation. +- If stale summaries are caused by relation sync failure, refreshing the page can mask the symptom without fixing the underlying sync path. Preserve logs before restarting services. + diff --git a/package.json b/package.json index 213d37c..57f7506 100644 --- a/package.json +++ b/package.json @@ -14,6 +14,7 @@ "backup:postgres": "node scripts/backup-postgres.mjs", "restore:postgres": "node scripts/restore-postgres.mjs", "backup:server-data": "node scripts/backup-server-data.mjs", + "docs:check-runbooks": "node scripts/check-runbook-placeholders.mjs --paths docs/runbooks docs/production-readiness.md", "deploy:local:build": "docker compose --env-file .env.local-server -f docker-compose.local.yml build", "deploy:local:up": "docker compose --env-file .env.local-server -f docker-compose.local.yml up -d", "deploy:local:down": "docker compose --env-file .env.local-server -f docker-compose.local.yml down", diff --git a/scripts/check-runbook-placeholders.mjs b/scripts/check-runbook-placeholders.mjs new file mode 100644 index 0000000..f99a3fe --- /dev/null +++ b/scripts/check-runbook-placeholders.mjs @@ -0,0 +1,87 @@ +#!/usr/bin/env node +import { existsSync, readFileSync, readdirSync, statSync } from 'node:fs'; +import { extname, join } from 'node:path'; +import { pathToFileURL } from 'node:url'; +import { option, parseArgs } from './ops-utils.mjs'; + +const forbiddenPatterns = [ + { name: 'TBD', pattern: /\bTBD\b/i }, + { name: 'TODO', pattern: /\bTODO\b/i }, + { name: 'fill-in', pattern: /fill in|fill-in/i }, + { name: 'angle-token', pattern: /<[^>\n]+>/ }, + { name: 'Chinese pending marker', pattern: /待定|占位/ }, +]; + +function usage() { + return `Usage: node scripts/check-runbook-placeholders.mjs --paths [path...] + +Scans Markdown runbooks for unresolved placeholder markers. +`; +} + +function collectMarkdownFiles(path) { + if (!existsSync(path)) { + throw new Error(`Path not found: ${path}`); + } + + const stat = statSync(path); + if (stat.isFile()) return extname(path) === '.md' ? [path] : []; + if (!stat.isDirectory()) return []; + + const files = []; + for (const entry of readdirSync(path)) { + files.push(...collectMarkdownFiles(join(path, entry))); + } + return files; +} + +function requestedPaths(argv) { + const args = parseArgs(argv); + if (args.flags.has('help')) return { help: true, paths: [] }; + const first = option(args, 'paths'); + const paths = [first, ...args.positionals].filter(Boolean); + if (paths.length === 0) { + throw new Error('Missing --paths [path...]'); + } + return { help: false, paths }; +} + +export function scanFiles(paths) { + const files = paths.flatMap(collectMarkdownFiles); + const findings = []; + for (const file of files) { + const lines = readFileSync(file, 'utf8').split(/\r?\n/); + lines.forEach((line, index) => { + for (const forbidden of forbiddenPatterns) { + if (forbidden.pattern.test(line)) { + findings.push(`${file}:${index + 1} ${forbidden.name}: ${line.trim()}`); + } + } + }); + } + return { files, findings }; +} + +export async function main(argv = process.argv.slice(2)) { + const { help, paths } = requestedPaths(argv); + if (help) { + process.stdout.write(usage()); + return; + } + + const { files, findings } = scanFiles(paths); + if (findings.length > 0) { + process.stderr.write(`${findings.join('\n')}\n`); + throw new Error(`Runbook placeholder scan failed: ${findings.length} finding(s)`); + } + + process.stdout.write(`Runbook placeholder scan passed (${files.length} file(s)).\n`); +} + +if (import.meta.url === pathToFileURL(process.argv[1]).href) { + main().catch((error) => { + process.stderr.write(`${error.message}\n`); + process.exit(1); + }); +} + diff --git a/scripts/ops-scripts.test.mjs b/scripts/ops-scripts.test.mjs index 3d21014..a70d0f8 100644 --- a/scripts/ops-scripts.test.mjs +++ b/scripts/ops-scripts.test.mjs @@ -138,4 +138,15 @@ describe('production ops scripts', () => { assert.doesNotMatch(combined, /sk-ant-[A-Za-z0-9]/); assert.doesNotMatch(combined, /hooks\.slack\.com\/services\//); }); + + it('passes the runbook placeholder scan', () => { + const result = runScript('scripts/check-runbook-placeholders.mjs', [ + '--paths', + 'docs/runbooks', + 'docs/production-readiness.md', + ]); + + assert.equal(result.status, 0, result.stderr); + assert.match(result.stdout, /Runbook placeholder scan passed/); + }); }); diff --git a/scripts/verify-production-deploy.mjs b/scripts/verify-production-deploy.mjs index 9759a63..957505f 100644 --- a/scripts/verify-production-deploy.mjs +++ b/scripts/verify-production-deploy.mjs @@ -93,6 +93,14 @@ const checks = [ file: 'scripts/backup-server-data.mjs', snippets: ['server_data', 'tar -czf', '--dry-run'], }, + { + file: 'scripts/check-runbook-placeholders.mjs', + snippets: ['Runbook placeholder scan passed', 'Runbook placeholder scan failed', '--paths'], + }, + { + file: 'package.json', + snippets: ['docs:check-runbooks', 'docs/runbooks', 'docs/production-readiness.md'], + }, { file: 'docker-compose.local.yml', snippets: [ @@ -138,6 +146,31 @@ const checks = [ 'FtbXiaobaoSummaryStale', ], }, + { + file: 'docs/runbooks/migration-rollback.md', + snippets: ['Fresh Database Restore', 'Data Risks', 'pnpm restore:postgres'], + }, + { + file: 'docs/runbooks/appdata-retirement.md', + snippets: ['Disable AppData Writes', 'Remove Fallback', 'Data Risks'], + }, + { + file: 'docs/runbooks/xiaobao-background-jobs.md', + snippets: ['FtbXiaobaoSummaryStale', 'Future Scheduler Rules', 'Data Risks'], + }, + { + file: 'docs/production-readiness.md', + snippets: [ + 'Backup and restore', + 'Smoke tests', + 'Monitoring', + 'Audit', + 'RBAC', + 'Consistency', + 'Performance', + 'Post-release', + ], + }, { file: 'deploy/monitoring/postgres/postgres-queries.yml', snippets: ['xiaobao_risk_summaries', 'dirty = true', 'stale_summary_count'], @@ -183,6 +216,8 @@ const checks = [ 'pnpm backup:postgres', 'pnpm restore:postgres', 'pnpm backup:server-data', + 'docs/runbooks/migration-rollback.md', + 'docs/production-readiness.md', '--profile monitoring', 'pnpm db:migrate', 'NEXT_PUBLIC_API_URL',