docs(ops): 补齐生产 runbook 和 readiness 清单

- 新增迁移回滚、AppData 退场、小宝后台任务 runbook\n- 新增生产 readiness 证据清单和 runbook placeholder 扫描\n- 更新部署文档与路线图到 V2.8 运维闭环阶段\n\nCo-Authored-By: GPT-5 Codex <codex@openai.com>
This commit is contained in:
2026-07-08 16:28:02 +08:00
parent 7837a809ca
commit 72a59f125c
10 changed files with 420 additions and 15 deletions

View File

@@ -283,6 +283,21 @@ pnpm restore:postgres -- \
--confirm-overwrite --confirm-overwrite
``` ```
## 运维 Runbooks
生产发布、迁移和异常处置优先使用这些手册:
- `docs/runbooks/migration-rollback.md`:发布失败、迁移失败、数据恢复和镜像回滚。
- `docs/runbooks/appdata-retirement.md`AppData key 分阶段退场、双读核对、归档和回滚。
- `docs/runbooks/xiaobao-background-jobs.md`:小宝摘要 stale 告警、手动刷新和未来后台任务规则。
- `docs/production-readiness.md`:生产发布前后证据清单。
提交前运行 runbook 扫描:
```bash
pnpm docs:check-runbooks
```
## 升级流程 ## 升级流程
```bash ```bash

View File

@@ -0,0 +1,24 @@
# Production Readiness Checklist
Use this before a production release and again after the release smoke test. Each item requires evidence, not a verbal assertion.
| Area | Gate | Evidence |
| --- | --- | --- |
| Backup and restore | PostgreSQL backup dry-run and server_data backup dry-run are reviewed. | `pnpm backup:postgres -- --dry-run --env-file .env.production` and `pnpm backup:server-data -- --dry-run --env-file .env.production` output saved in release notes. |
| Backup and restore | Restore command refuses overwrite without explicit confirmation. | `pnpm restore:postgres -- --env-file .env.production --input backups/postgres/ftb_pm-postgres-20260708T120000Z.dump` exits non-zero and names `--confirm-overwrite`. |
| Backup and restore | Fresh DB restore rehearsal completed before destructive restore. | Temporary database restore command and row-count comparison from `docs/runbooks/migration-rollback.md`. |
| Smoke tests | Release smoke is wired into GitHub Actions. | `.github/workflows/deploy-production.yml` runs `scripts/smoke-test-release.mjs` with `--expected-version`. |
| Smoke tests | Manual smoke can be run against the target. | `pnpm deploy:smoke -- --base-url http://localhost --expected-version "$APP_VERSION"` output. |
| Monitoring | Monitoring profile renders and can start without committed secrets. | `docker compose --env-file .env.production -f docker-compose.prod.yml --profile monitoring config` exits 0. |
| Monitoring | Required alert rules exist. | `FtbPostgresDown`, `FtbDiskPressure`, `FtbSlowApiLogBurst`, `FtbSlowPrismaLogBurst`, `FtbJobFailureLogBurst`, and `FtbXiaobaoSummaryStale` visible in Prometheus. |
| Audit | Domain write APIs record actor and resource scope where implemented. | API request sample or audit log sample for Product, Requirement, Project, Version, execution entities, and dictionary writes. |
| RBAC | Project and version operations enforce Owner, Admin, Member, Viewer boundaries where enabled. | Permission matrix test result or manual account walkthrough attached to release notes. |
| Consistency | AppData and relation tables are checked for migrated domains. | Count and missing-partition-key queries from `docs/runbooks/appdata-retirement.md`. |
| Performance | Slow API and slow Prisma thresholds are configured. | `API_SLOW_REQUEST_MS` and `PRISMA_SLOW_QUERY_MS` values recorded, plus Grafana slow-log panels checked. |
| Performance | V2.2 hot reads avoid full-table scans. | Requirement pool smoke uses `productId`; query plan or service test evidence attached for high-volume domains. |
| Post-release | Runtime version matches the release SHA. | `/api/v1/health/version` payload or `pnpm deploy:check-runtime` output. |
| Post-release | Product, requirement, workspace, Xiaobao, and AI config read paths respond. | `pnpm deploy:smoke` output attached. |
| Post-release | On-call rollback path is known. | `docs/runbooks/migration-rollback.md` link included in release notes. |
Release owner signs off only after all required evidence is attached to the release notes or incident record.

View File

@@ -1,17 +1,16 @@
# 开发路线图 # 开发路线图
## 当前阶段V2.4领域 CRUD 主写迁移 ## 当前阶段V2.8生产硬化稳定版 + 运维闭环
V2.4 的目标是把业务主数据源从 AppData JSONB 文档切换到 PostgreSQL 领域关系表。AppData 继续保留为迁移、回填、兼容读取和排查入口,但不再作为长期主写入源;新增业务能力必须优先设计关系表、领域 CRUD API、索引/分区键和权限边界。V2.4 做逐领域主写迁移,并随 CRUD 入口埋好基础权限、`actorId` 和审计事件骨架;完整 RBAC、审计覆盖和 AppData 退场收口放到 V2.5 V2.8 的目标是在既有生产 CI/CD 基线上补齐运维闭环:备份恢复演练、发布 smoke test、监控告警、日志检索、迁移回滚 runbook、AppData 退场 runbook、小宝后台化 runbook 和生产 readiness 证据清单。当前执行前提是 V2.4 领域 CRUD 主写迁移已由上游验收完成;本阶段不重新设计业务流程,专注把生产发布和故障处置做成可验证、可复盘、可回滚的标准流程
### 当前重点 ### 当前重点
1. **领域写 API**:按模块补齐 Product/Project/Version/Requirement/VersionPlan/DevTask/TestCase/Bug/Member/TaskCategory/Worklog/Overtime 的关系表写入 API 1. **备份恢复自动化**PostgreSQL dump、`server_data` volume 备份、fresh DB restore dry-run 和显式覆盖确认
2. **前端持久化切换**Zustand store 保留状态管理,但保存入口从 `saveServerData(key)` 迁到领域 API读取优先 V2.2/V2.3 关系表接口 2. **发布 smoke test**GitHub Actions 部署后自动校验 runtime version、前端根页、产品页、产品 API、V2.2 读路径和 AI 配置
3. **AppData 迁移工具化**:把 AppData → 关系表同步做成可重复运行、可计数校验、可回滚的运维脚本,不提交真实 `.env` 3. **监控告警基线**Prometheus/Grafana/Loki/Promtail 可选 profile覆盖慢 API、慢 Prisma、任务失败、小宝摘要 stale、磁盘压力和 DB 可用性
4. **主写切换闸门**:每个领域完成双读核对后,先停止该领域 JSON 主写入,再进入 V2.5 的 fallback 移除和归档退场 4. **迁移和后台任务 runbook**迁移回滚、AppData 退场、小宝后台化处置步骤、决策点和数据风险
5. **数据一致性校验**:为每个迁移领域补 counts、抽样记录、孤儿引用、分区键完整性和唯一约束校验 5. **生产 readiness 清单**backup/restore、smoke、monitoring、audit、RBAC、consistency、performance 和 post-release verification 都要有证据项
6. **权限/审计骨架**:领域 API 必须携带当前用户、产品/项目/版本作用域和审计事件入口,避免 V2.5 做 RBAC 时返工。
## V2 分阶段交付链路 ## V2 分阶段交付链路
@@ -34,12 +33,18 @@ V2.4 的目标是把业务主数据源从 AppData JSONB 文档切换到 PostgreS
- 版本详情已有需求、调研、产品方案、UI、开发任务、测试用例、Bug、概览等核心 Tab渲染重的路径优先接入 V2.2 关系表快读,并保留 AppData fallback。 - 版本详情已有需求、调研、产品方案、UI、开发任务、测试用例、Bug、概览等核心 Tab渲染重的路径优先接入 V2.2 关系表快读,并保留 AppData fallback。
- 后端已落地 Product、Requirement 领域 CRUDDataModule AppData 乐观锁V2.2 快读 APIV2.3 AppData 写后同步关系表AI Provider 抽象和健康版本接口。 - 后端已落地 Product、Requirement 领域 CRUDDataModule AppData 乐观锁V2.2 快读 APIV2.3 AppData 写后同步关系表AI Provider 抽象和健康版本接口。
- Prisma schema 已包含 Product、Project、Version、Requirement、VersionPlan、DevTask、TestCase、Bug、WorkActivity、Xiaobao、AiLog、AppData 等关系模型;高增长表的分区 migration 已落地。 - Prisma schema 已包含 Product、Project、Version、Requirement、VersionPlan、DevTask、TestCase、Bug、WorkActivity、Xiaobao、AiLog、AppData 等关系模型;高增长表的分区 migration 已落地。
- 主写入源仍处在兼容窗口:多数前端 store 继续通过 `apps/web/lib/server-data.ts``loadServerData` / `saveServerData` 写 AppData`useProductStore` 仍以 `products-overview` 文档作为产品/项目/版本树主写入 - 本 V2.8 执行线程以前提“V2.4 领域 CRUD 主写迁移已完成验收”推进;本阶段不重新逐项复核领域 API 清单
- Project、Version、VersionPlan、DevTask、TestCase、Bug、Member、TaskCategory、TaskWorklog、Overtime 等领域写 API 尚未完整替代 AppData Store。若后续称为 V2.4,应理解为“领域 CRUD 迁移阶段”,不是 V2.3 已完成内容 - V2.8 新增运维交付物集中在 `scripts/``.github/workflows/deploy-production.yml``deploy/monitoring/``docs/runbooks/``docs/deployment.md``docs/production-readiness.md`
- `packages/shared` 中仍保留早期枚举口径;切换领域 API 时需要统一为当前前端业务状态机 - AppData 退场、RBAC/审计、性能和小宝后台化仍通过 production readiness 证据项追踪,避免把运维稳定版误当成业务治理已全部完成
### 已完成(按时间倒序) ### 已完成(按时间倒序)
**2026-07-08**
- V2.8 production ops closure started: added PostgreSQL backup, fresh DB restore with explicit overwrite confirmation, and `server_data` volume backup automation.
- Added release smoke suite and wired GitHub Actions deployment verification to runtime version, frontend root, products, V2.2 read path, and AI config checks.
- Added optional monitoring profile with Prometheus, Grafana, Loki, Promtail, postgres-exporter, node-exporter, cAdvisor, and blackbox-exporter.
- Added migration rollback, AppData retirement, Xiaobao background jobs runbooks, and production readiness evidence checklist.
**2026-07-06** **2026-07-06**
- Added production CI/CD flow: GitHub Actions builds `web` and `server` Docker images, pushes immutable commit-SHA tags to GHCR, deploys by SSH, pulls images on the server, runs `pnpm --filter server db:deploy`, restarts Compose, and verifies `/api/v1/health/version`. - Added production CI/CD flow: GitHub Actions builds `web` and `server` Docker images, pushes immutable commit-SHA tags to GHCR, deploys by SSH, pulls images on the server, runs `pnpm --filter server db:deploy`, restarts Compose, and verifies `/api/v1/health/version`.
- Added runtime version metadata: backend `GET /api/v1/health/version`, Docker build args/env, and a frontend refresh banner when browser assets are older than the server runtime. - Added runtime version metadata: backend `GET /api/v1/health/version`, Docker build args/env, and a frontend refresh banner when browser assets are older than the server runtime.
@@ -117,12 +122,12 @@ V2.4 的目标是把业务主数据源从 AppData JSONB 文档切换到 PostgreS
### 进行中 ### 进行中
- V2.4 领域 CRUD 主写迁移:从 AppData JSONB 主写入切换到关系表 API并随 API 落基础权限、审计和查询性能边界 - V2.8 生产硬化稳定版:补齐备份恢复、发布 smoke、监控告警、日志检索、迁移回滚和运维证据闭环
- 项目详情页 VersionCard 状态胶囊数据联动(部分已完成) - 项目详情页 VersionCard 状态胶囊数据联动(部分已完成)
## V2 — 后端接入 ## V2 — 后端接入
NestJS + Prisma + PostgreSQL 已接入到 V2.3。第一阶段用 `app_data` JSONB 文档表承接现有 store 数据形状,避免浏览器清站点数据导致业务数据丢失;第二阶段建立分区关系表、V2.2 快读 API 和 V2.3 AppData 写后同步。当前 V2.4 才是逐领域启用写 API让前端 store 从 AppData 主写迁移到领域 CRUDAppData 后续只保留为迁移兼容层 NestJS + Prisma + PostgreSQL 已完成 V2.1 至 V2.4 的后端迁移主线。第一阶段用 `app_data` JSONB 文档表承接现有 store 数据形状避免浏览器清站点数据导致业务数据丢失第二阶段建立分区关系表、V2.2 快读 API 和 V2.3 AppData 写后同步V2.4 完成领域 CRUD 主写迁移验收。V2.8 不再新增业务主写迁移范围,而是把生产运维、备份恢复、监控告警和回滚手册补齐
### 关键任务 ### 关键任务
@@ -141,7 +146,7 @@ NestJS + Prisma + PostgreSQL 已接入到 V2.3。第一阶段用 `app_data` JSON
当前不做本地导入导出。清站点数据后浏览器旧数据无法恢复,后续新增数据直接写入 PostgreSQL。若以后需要迁移旧浏览器数据再单独做管理员导入工具。 当前不做本地导入导出。清站点数据后浏览器旧数据无法恢复,后续新增数据直接写入 PostgreSQL。若以后需要迁移旧浏览器数据再单独做管理员导入工具。
## V2.4 — 领域 CRUD 迁移(当前阶段 ## V2.4 — 领域 CRUD 迁移(已完成前提
目标是让关系表从“快读 + AppData 同步副本”逐步升级为主写入路径。迁移顺序应优先选择写入频率高、实体边界清晰、已经在 V2.2 mapper 中稳定的领域: 目标是让关系表从“快读 + AppData 同步副本”逐步升级为主写入路径。迁移顺序应优先选择写入频率高、实体边界清晰、已经在 V2.2 mapper 中稳定的领域:
@@ -221,7 +226,7 @@ V2.4 推进前必须先统一 `packages/shared` 的状态枚举与当前前端
|------|------| |------|------|
| V1 业务流程打磨 | 进行中 | | V1 业务流程打磨 | 进行中 |
| V1 朋友试用反馈 | 持续中 | | V1 朋友试用反馈 | 持续中 |
| V2 后端接入 | 进行中V2.1/V2.2/V2.3 已完成V2.4 主写迁移中) | | V2 后端接入 | V2.1-V2.4 已完成V2.8 运维闭环进行中 |
| V3 AI 集成 | 等 V2 数据沉淀 | | V3 AI 集成 | 等 V2 数据沉淀 |
| 公开发布 | TBD | | 公开发布 | TBD |
**2026-06-26** **2026-06-26**

View File

@@ -0,0 +1,72 @@
# AppData Retirement Runbook
Use this when retiring an AppData key after its domain writes have moved to relation-table APIs. The order is fixed: backup, measure, freeze writes, compare, remove fallback, archive.
## Scope Gate
Retire one AppData key family at a time. Good candidates have domain CRUD writes, read APIs, pagination boundaries, audit events, and a rollback path.
## Preparation
```bash
pnpm backup:postgres -- --env-file .env.production
pnpm backup:server-data -- --env-file .env.production
pnpm deploy:smoke -- --base-url http://localhost
```
Record the key family being retired, the owning domain API, and the relation tables that replace it.
## Count And Consistency Checks
Run counts before disabling writes.
```bash
docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "select key, jsonb_array_length(value) as appdata_rows from app_data where key in ('products-overview','requirements','version-plans','dev-tasks','test-cases','bugs','members','task-categories','task-worklogs','overtime') order by key;"
docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "select 'requirements' as table_name, count(*) from requirements union all select 'version_plans', count(*) from version_plans union all select 'dev_tasks', count(*) from dev_tasks union all select 'test_cases', count(*) from test_cases union all select 'bugs', count(*) from bugs;"
```
For partitioned entities, also check missing partition keys.
```bash
docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "select 'requirements_missing_product' as check_name, count(*) from requirements where product_id is null union all select 'dev_tasks_missing_version', count(*) from dev_tasks where version_id is null union all select 'test_cases_missing_version', count(*) from test_cases where version_id is null union all select 'bugs_missing_version', count(*) from bugs where version_id is null;"
```
## Disable AppData Writes
1. Merge the domain-specific frontend store change that stops calling `saveServerData` for the retired key.
2. Keep AppData read fallback for one release while relation reads are verified.
3. Deploy and run smoke tests.
```bash
pnpm deploy:smoke -- --base-url http://localhost --expected-version "$APP_VERSION"
```
## Remove Fallback
Remove AppData read fallback only after one successful release where:
- Domain writes went through relation APIs.
- AppData row counts did not grow for the retired key.
- V2.2 read paths and page workflows returned expected data.
- No `AppData relation sync failed` logs appeared during the observation window.
## Archive
Export retired keys before any later table cleanup.
```bash
docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "copy (select key, value, updated_at from app_data where key in ('dev-tasks','test-cases','bugs')) to stdout with csv header" > backups/postgres/appdata-retired-executions-20260708.csv
```
## Rollback
- If relation writes fail but AppData still has fresh data, roll back to the previous app image that still writes AppData.
- If AppData writes were already disabled and relation writes are bad, restore PostgreSQL from the backup made at the start of this runbook.
- If only read fallback removal caused the issue, roll back app images first and leave the database unchanged.
## Data Risks
- Removing fallback too early can hide valid historical JSON rows that were never mapped into relation tables.
- Re-enabling old AppData writes after relation writes have accepted new edits can overwrite newer relation state through compatibility sync.
- AppData exports can contain business-sensitive text; store backup CSV files in the same restricted location as database dumps.

View File

@@ -0,0 +1,100 @@
# Migration Rollback Runbook
Use this when a production release, Prisma migration, or data migration causes failed smoke tests, missing data, bad query performance, or unsafe writes.
## First Response
1. Freeze new releases and ask product owners to pause bulk edits.
2. Capture current state before changing anything.
```bash
date -u
git rev-parse HEAD
docker compose --env-file .env.production -f docker-compose.prod.yml ps
pnpm backup:postgres -- --env-file .env.production
pnpm backup:server-data -- --env-file .env.production
```
3. Run the read-only release smoke test.
```bash
pnpm deploy:smoke -- --base-url http://localhost --expected-version "$APP_VERSION"
```
4. Check the database and latest server logs.
```bash
docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres pg_isready -U "$POSTGRES_USER" -d "$POSTGRES_DB"
docker compose --env-file .env.production -f docker-compose.prod.yml logs --tail=200 server
```
## Decision Points
- App code bad, database healthy: roll back `WEB_IMAGE` and `SERVER_IMAGE` to the previous commit image tags, then restart Compose.
- Migration applied but only additive: roll back app images first; leave schema in place if old code remains compatible.
- Migration changed or removed data: restore a fresh database from the last known-good backup into a temporary database, compare row counts, then decide whether to restore production.
- AppData compatibility issue: keep production database intact, re-enable the previous app image that still reads the AppData fallback, and preserve the failing release backup for analysis.
## App Image Rollback
Set previous image tags in `.env.production`. Use the previous successful GitHub Actions run to identify the image tags.
```bash
export PREVIOUS_WEB_IMAGE=ghcr.io/company/ftb-project-management/web:abc1234
export PREVIOUS_SERVER_IMAGE=ghcr.io/company/ftb-project-management/server:abc1234
sed -i "s|^WEB_IMAGE=.*|WEB_IMAGE=${PREVIOUS_WEB_IMAGE}|" .env.production
sed -i "s|^SERVER_IMAGE=.*|SERVER_IMAGE=${PREVIOUS_SERVER_IMAGE}|" .env.production
docker compose --env-file .env.production -f docker-compose.prod.yml pull web server
docker compose --env-file .env.production -f docker-compose.prod.yml up -d --remove-orphans
pnpm deploy:smoke -- --base-url http://localhost --expected-version abc1234
```
## Fresh Database Restore
Never restore over production until the backup has been rehearsed into a temporary database.
```bash
export BACKUP_FILE=backups/postgres/ftb_pm-postgres-20260708T120000Z.dump
pnpm restore:postgres -- \
--dry-run \
--confirm-overwrite \
--env-file .env.production \
--input "$BACKUP_FILE" \
--target-db ftb_pm_restore_check
pnpm restore:postgres -- \
--confirm-overwrite \
--env-file .env.production \
--input "$BACKUP_FILE" \
--target-db ftb_pm_restore_check
```
Compare critical row counts before touching production.
```bash
docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres psql -U "$POSTGRES_USER" -d ftb_pm_restore_check -c "select 'products' as table_name, count(*) from products union all select 'requirements', count(*) from requirements union all select 'versions', count(*) from versions union all select 'dev_tasks', count(*) from dev_tasks union all select 'test_cases', count(*) from test_cases union all select 'bugs', count(*) from bugs;"
```
If the temporary restore is healthy and production data is unsafe, restore production with explicit overwrite confirmation.
```bash
pnpm restore:postgres -- \
--confirm-overwrite \
--env-file .env.production \
--input "$BACKUP_FILE"
docker compose --env-file .env.production -f docker-compose.prod.yml up -d --remove-orphans
pnpm deploy:smoke -- --base-url http://localhost
```
## Data Risks
- PostgreSQL restore is destructive for the target database because the script terminates connections, drops the target database, recreates it, and runs `pg_restore`.
- Restoring PostgreSQL does not restore `server_data`; keep AI provider config backup files with the same incident bundle.
- AppData and relation tables can diverge during compatibility windows. Before deleting or restoring, preserve both the failing production backup and the known-good backup.
- If users continued editing during the incident, record the time window and decide whether those edits must be replayed manually after restore.
## Closeout
1. Save the failed release SHA, rollback SHA, backup file names, smoke output, and row-count evidence in the incident notes.
2. Keep the failed backup until the next successful release has completed smoke tests and one business-day observation.
3. Add a regression test or runbook correction before re-attempting the migration.

View File

@@ -0,0 +1,55 @@
# Xiaobao Background Jobs Runbook
Current V2.8 production monitoring supports Xiaobao staleness detection. The first production implementation is still page-triggered: opening `/xiaobao-warning` computes risk, saves snapshots, and lets V2.3 sync refresh summaries. A future scheduler must preserve the same idempotent data contract.
## Alert Triage
When `FtbXiaobaoSummaryStale` fires:
```bash
docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "select version_id, dirty, updated_at, recomputed_at, risk_level, risk_score from xiaobao_risk_summaries where dirty = true or updated_at < now() - interval '6 hours' order by updated_at asc limit 20;"
docker compose --env-file .env.production -f docker-compose.prod.yml logs --tail=200 server | grep -E "xiaobao|AppData relation sync failed|Slow Prisma query"
```
Decision points:
- Rows are dirty after active version edits: ask a manager to open `/xiaobao-warning` once, then verify summaries refresh.
- Rows remain dirty and server logs show sync failures: treat as AppData relation sync incident and follow `migration-rollback.md`.
- Rows are stale but no product release is near: keep monitoring and schedule a manual refresh before the next release decision meeting.
- Rows are stale for a release due today: refresh manually and have the release owner review the resulting risk explanation before ship/no-ship decision.
## Manual Refresh Path
1. Log in as a user with `xiaobao.warning:manage`.
2. Open `/xiaobao-warning`.
3. Wait until AI interpretation status is no longer generating for high-risk versions.
4. Re-run the stale-summary query.
5. Run the release smoke test.
```bash
pnpm deploy:smoke -- --base-url http://localhost
```
## Future Scheduler Rules
When a background job is introduced, it must:
- Read unfinished versions by relation-table scope, not by full AppData document scan.
- Use one idempotency key per `versionId + riskSignature + snapshotDate`.
- Write snapshots append-only and upsert summaries by `version_id`.
- Mark failures with structured logs containing `xiaobao background job failed`.
- Retry transient AI failures with backoff and keep rule-based risk output even when AI interpretation fails.
- Never mutate Version, Requirement, DevTask, TestCase, Bug, or Member data.
## Monitoring Expectations
- `FtbXiaobaoSummaryStale` alerts on dirty or older-than-6-hour summaries.
- `FtbJobFailureLogBurst` alerts when sync or future job failure log counters increase.
- Grafana dashboard shows the stale summary count and matching server warning/error logs.
## Data Risks
- Recomputing Xiaobao summaries can change release risk badges and manager decisions; record manual refresh time in release notes.
- AI interpretation cache is explanatory only. Do not restore or delete business entities to fix a bad explanation.
- If stale summaries are caused by relation sync failure, refreshing the page can mask the symptom without fixing the underlying sync path. Preserve logs before restarting services.

View File

@@ -14,6 +14,7 @@
"backup:postgres": "node scripts/backup-postgres.mjs", "backup:postgres": "node scripts/backup-postgres.mjs",
"restore:postgres": "node scripts/restore-postgres.mjs", "restore:postgres": "node scripts/restore-postgres.mjs",
"backup:server-data": "node scripts/backup-server-data.mjs", "backup:server-data": "node scripts/backup-server-data.mjs",
"docs:check-runbooks": "node scripts/check-runbook-placeholders.mjs --paths docs/runbooks docs/production-readiness.md",
"deploy:local:build": "docker compose --env-file .env.local-server -f docker-compose.local.yml build", "deploy:local:build": "docker compose --env-file .env.local-server -f docker-compose.local.yml build",
"deploy:local:up": "docker compose --env-file .env.local-server -f docker-compose.local.yml up -d", "deploy:local:up": "docker compose --env-file .env.local-server -f docker-compose.local.yml up -d",
"deploy:local:down": "docker compose --env-file .env.local-server -f docker-compose.local.yml down", "deploy:local:down": "docker compose --env-file .env.local-server -f docker-compose.local.yml down",

View File

@@ -0,0 +1,87 @@
#!/usr/bin/env node
import { existsSync, readFileSync, readdirSync, statSync } from 'node:fs';
import { extname, join } from 'node:path';
import { pathToFileURL } from 'node:url';
import { option, parseArgs } from './ops-utils.mjs';
const forbiddenPatterns = [
{ name: 'TBD', pattern: /\bTBD\b/i },
{ name: 'TODO', pattern: /\bTODO\b/i },
{ name: 'fill-in', pattern: /fill in|fill-in/i },
{ name: 'angle-token', pattern: /<[^>\n]+>/ },
{ name: 'Chinese pending marker', pattern: /待定|占位/ },
];
function usage() {
return `Usage: node scripts/check-runbook-placeholders.mjs --paths <path> [path...]
Scans Markdown runbooks for unresolved placeholder markers.
`;
}
function collectMarkdownFiles(path) {
if (!existsSync(path)) {
throw new Error(`Path not found: ${path}`);
}
const stat = statSync(path);
if (stat.isFile()) return extname(path) === '.md' ? [path] : [];
if (!stat.isDirectory()) return [];
const files = [];
for (const entry of readdirSync(path)) {
files.push(...collectMarkdownFiles(join(path, entry)));
}
return files;
}
function requestedPaths(argv) {
const args = parseArgs(argv);
if (args.flags.has('help')) return { help: true, paths: [] };
const first = option(args, 'paths');
const paths = [first, ...args.positionals].filter(Boolean);
if (paths.length === 0) {
throw new Error('Missing --paths <path> [path...]');
}
return { help: false, paths };
}
export function scanFiles(paths) {
const files = paths.flatMap(collectMarkdownFiles);
const findings = [];
for (const file of files) {
const lines = readFileSync(file, 'utf8').split(/\r?\n/);
lines.forEach((line, index) => {
for (const forbidden of forbiddenPatterns) {
if (forbidden.pattern.test(line)) {
findings.push(`${file}:${index + 1} ${forbidden.name}: ${line.trim()}`);
}
}
});
}
return { files, findings };
}
export async function main(argv = process.argv.slice(2)) {
const { help, paths } = requestedPaths(argv);
if (help) {
process.stdout.write(usage());
return;
}
const { files, findings } = scanFiles(paths);
if (findings.length > 0) {
process.stderr.write(`${findings.join('\n')}\n`);
throw new Error(`Runbook placeholder scan failed: ${findings.length} finding(s)`);
}
process.stdout.write(`Runbook placeholder scan passed (${files.length} file(s)).\n`);
}
if (import.meta.url === pathToFileURL(process.argv[1]).href) {
main().catch((error) => {
process.stderr.write(`${error.message}\n`);
process.exit(1);
});
}

View File

@@ -138,4 +138,15 @@ describe('production ops scripts', () => {
assert.doesNotMatch(combined, /sk-ant-[A-Za-z0-9]/); assert.doesNotMatch(combined, /sk-ant-[A-Za-z0-9]/);
assert.doesNotMatch(combined, /hooks\.slack\.com\/services\//); assert.doesNotMatch(combined, /hooks\.slack\.com\/services\//);
}); });
it('passes the runbook placeholder scan', () => {
const result = runScript('scripts/check-runbook-placeholders.mjs', [
'--paths',
'docs/runbooks',
'docs/production-readiness.md',
]);
assert.equal(result.status, 0, result.stderr);
assert.match(result.stdout, /Runbook placeholder scan passed/);
});
}); });

View File

@@ -93,6 +93,14 @@ const checks = [
file: 'scripts/backup-server-data.mjs', file: 'scripts/backup-server-data.mjs',
snippets: ['server_data', 'tar -czf', '--dry-run'], snippets: ['server_data', 'tar -czf', '--dry-run'],
}, },
{
file: 'scripts/check-runbook-placeholders.mjs',
snippets: ['Runbook placeholder scan passed', 'Runbook placeholder scan failed', '--paths'],
},
{
file: 'package.json',
snippets: ['docs:check-runbooks', 'docs/runbooks', 'docs/production-readiness.md'],
},
{ {
file: 'docker-compose.local.yml', file: 'docker-compose.local.yml',
snippets: [ snippets: [
@@ -138,6 +146,31 @@ const checks = [
'FtbXiaobaoSummaryStale', 'FtbXiaobaoSummaryStale',
], ],
}, },
{
file: 'docs/runbooks/migration-rollback.md',
snippets: ['Fresh Database Restore', 'Data Risks', 'pnpm restore:postgres'],
},
{
file: 'docs/runbooks/appdata-retirement.md',
snippets: ['Disable AppData Writes', 'Remove Fallback', 'Data Risks'],
},
{
file: 'docs/runbooks/xiaobao-background-jobs.md',
snippets: ['FtbXiaobaoSummaryStale', 'Future Scheduler Rules', 'Data Risks'],
},
{
file: 'docs/production-readiness.md',
snippets: [
'Backup and restore',
'Smoke tests',
'Monitoring',
'Audit',
'RBAC',
'Consistency',
'Performance',
'Post-release',
],
},
{ {
file: 'deploy/monitoring/postgres/postgres-queries.yml', file: 'deploy/monitoring/postgres/postgres-queries.yml',
snippets: ['xiaobao_risk_summaries', 'dirty = true', 'stale_summary_count'], snippets: ['xiaobao_risk_summaries', 'dirty = true', 'stale_summary_count'],
@@ -183,6 +216,8 @@ const checks = [
'pnpm backup:postgres', 'pnpm backup:postgres',
'pnpm restore:postgres', 'pnpm restore:postgres',
'pnpm backup:server-data', 'pnpm backup:server-data',
'docs/runbooks/migration-rollback.md',
'docs/production-readiness.md',
'--profile monitoring', '--profile monitoring',
'pnpm db:migrate', 'pnpm db:migrate',
'NEXT_PUBLIC_API_URL', 'NEXT_PUBLIC_API_URL',