docs(ops): 补齐生产 runbook 和 readiness 清单
- 新增迁移回滚、AppData 退场、小宝后台任务 runbook\n- 新增生产 readiness 证据清单和 runbook placeholder 扫描\n- 更新部署文档与路线图到 V2.8 运维闭环阶段\n\nCo-Authored-By: GPT-5 Codex <codex@openai.com>
This commit is contained in:
@@ -283,6 +283,21 @@ pnpm restore:postgres -- \
|
|||||||
--confirm-overwrite
|
--confirm-overwrite
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## 运维 Runbooks
|
||||||
|
|
||||||
|
生产发布、迁移和异常处置优先使用这些手册:
|
||||||
|
|
||||||
|
- `docs/runbooks/migration-rollback.md`:发布失败、迁移失败、数据恢复和镜像回滚。
|
||||||
|
- `docs/runbooks/appdata-retirement.md`:AppData key 分阶段退场、双读核对、归档和回滚。
|
||||||
|
- `docs/runbooks/xiaobao-background-jobs.md`:小宝摘要 stale 告警、手动刷新和未来后台任务规则。
|
||||||
|
- `docs/production-readiness.md`:生产发布前后证据清单。
|
||||||
|
|
||||||
|
提交前运行 runbook 扫描:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pnpm docs:check-runbooks
|
||||||
|
```
|
||||||
|
|
||||||
## 升级流程
|
## 升级流程
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|||||||
24
docs/production-readiness.md
Normal file
24
docs/production-readiness.md
Normal file
@@ -0,0 +1,24 @@
|
|||||||
|
# Production Readiness Checklist
|
||||||
|
|
||||||
|
Use this before a production release and again after the release smoke test. Each item requires evidence, not a verbal assertion.
|
||||||
|
|
||||||
|
| Area | Gate | Evidence |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Backup and restore | PostgreSQL backup dry-run and server_data backup dry-run are reviewed. | `pnpm backup:postgres -- --dry-run --env-file .env.production` and `pnpm backup:server-data -- --dry-run --env-file .env.production` output saved in release notes. |
|
||||||
|
| Backup and restore | Restore command refuses overwrite without explicit confirmation. | `pnpm restore:postgres -- --env-file .env.production --input backups/postgres/ftb_pm-postgres-20260708T120000Z.dump` exits non-zero and names `--confirm-overwrite`. |
|
||||||
|
| Backup and restore | Fresh DB restore rehearsal completed before destructive restore. | Temporary database restore command and row-count comparison from `docs/runbooks/migration-rollback.md`. |
|
||||||
|
| Smoke tests | Release smoke is wired into GitHub Actions. | `.github/workflows/deploy-production.yml` runs `scripts/smoke-test-release.mjs` with `--expected-version`. |
|
||||||
|
| Smoke tests | Manual smoke can be run against the target. | `pnpm deploy:smoke -- --base-url http://localhost --expected-version "$APP_VERSION"` output. |
|
||||||
|
| Monitoring | Monitoring profile renders and can start without committed secrets. | `docker compose --env-file .env.production -f docker-compose.prod.yml --profile monitoring config` exits 0. |
|
||||||
|
| Monitoring | Required alert rules exist. | `FtbPostgresDown`, `FtbDiskPressure`, `FtbSlowApiLogBurst`, `FtbSlowPrismaLogBurst`, `FtbJobFailureLogBurst`, and `FtbXiaobaoSummaryStale` visible in Prometheus. |
|
||||||
|
| Audit | Domain write APIs record actor and resource scope where implemented. | API request sample or audit log sample for Product, Requirement, Project, Version, execution entities, and dictionary writes. |
|
||||||
|
| RBAC | Project and version operations enforce Owner, Admin, Member, Viewer boundaries where enabled. | Permission matrix test result or manual account walkthrough attached to release notes. |
|
||||||
|
| Consistency | AppData and relation tables are checked for migrated domains. | Count and missing-partition-key queries from `docs/runbooks/appdata-retirement.md`. |
|
||||||
|
| Performance | Slow API and slow Prisma thresholds are configured. | `API_SLOW_REQUEST_MS` and `PRISMA_SLOW_QUERY_MS` values recorded, plus Grafana slow-log panels checked. |
|
||||||
|
| Performance | V2.2 hot reads avoid full-table scans. | Requirement pool smoke uses `productId`; query plan or service test evidence attached for high-volume domains. |
|
||||||
|
| Post-release | Runtime version matches the release SHA. | `/api/v1/health/version` payload or `pnpm deploy:check-runtime` output. |
|
||||||
|
| Post-release | Product, requirement, workspace, Xiaobao, and AI config read paths respond. | `pnpm deploy:smoke` output attached. |
|
||||||
|
| Post-release | On-call rollback path is known. | `docs/runbooks/migration-rollback.md` link included in release notes. |
|
||||||
|
|
||||||
|
Release owner signs off only after all required evidence is attached to the release notes or incident record.
|
||||||
|
|
||||||
@@ -1,17 +1,16 @@
|
|||||||
# 开发路线图
|
# 开发路线图
|
||||||
|
|
||||||
## 当前阶段:V2.4 — 领域 CRUD 主写迁移
|
## 当前阶段:V2.8 — 生产硬化稳定版 + 运维闭环
|
||||||
|
|
||||||
V2.4 的目标是把业务主数据源从 AppData JSONB 文档切换到 PostgreSQL 领域关系表。AppData 继续保留为迁移、回填、兼容读取和排查入口,但不再作为长期主写入源;新增业务能力必须优先设计关系表、领域 CRUD API、索引/分区键和权限边界。V2.4 做逐领域主写迁移,并随 CRUD 入口埋好基础权限、`actorId` 和审计事件骨架;完整 RBAC、审计覆盖和 AppData 退场收口放到 V2.5。
|
V2.8 的目标是在既有生产 CI/CD 基线上补齐运维闭环:备份恢复演练、发布 smoke test、监控告警、日志检索、迁移回滚 runbook、AppData 退场 runbook、小宝后台化 runbook 和生产 readiness 证据清单。当前执行前提是 V2.4 领域 CRUD 主写迁移已由上游验收完成;本阶段不重新设计业务流程,专注把生产发布和故障处置做成可验证、可复盘、可回滚的标准流程。
|
||||||
|
|
||||||
### 当前重点
|
### 当前重点
|
||||||
|
|
||||||
1. **领域写 API**:按模块补齐 Product/Project/Version/Requirement/VersionPlan/DevTask/TestCase/Bug/Member/TaskCategory/Worklog/Overtime 的关系表写入 API。
|
1. **备份恢复自动化**:PostgreSQL dump、`server_data` volume 备份、fresh DB restore dry-run 和显式覆盖确认。
|
||||||
2. **前端持久化切换**:Zustand store 保留状态管理,但保存入口从 `saveServerData(key)` 迁到领域 API;读取优先 V2.2/V2.3 关系表接口。
|
2. **发布 smoke test**:GitHub Actions 部署后自动校验 runtime version、前端根页、产品页、产品 API、V2.2 读路径和 AI 配置。
|
||||||
3. **AppData 迁移工具化**:把 AppData → 关系表同步做成可重复运行、可计数校验、可回滚的运维脚本,不提交真实 `.env`。
|
3. **监控告警基线**:Prometheus/Grafana/Loki/Promtail 可选 profile,覆盖慢 API、慢 Prisma、任务失败、小宝摘要 stale、磁盘压力和 DB 可用性。
|
||||||
4. **主写切换闸门**:每个领域完成双读核对后,先停止该领域 JSON 主写入,再进入 V2.5 的 fallback 移除和归档退场。
|
4. **迁移和后台任务 runbook**:迁移回滚、AppData 退场、小宝后台化处置步骤、决策点和数据风险。
|
||||||
5. **数据一致性校验**:为每个迁移领域补 counts、抽样记录、孤儿引用、分区键完整性和唯一约束校验。
|
5. **生产 readiness 清单**:backup/restore、smoke、monitoring、audit、RBAC、consistency、performance 和 post-release verification 都要有证据项。
|
||||||
6. **权限/审计骨架**:领域 API 必须携带当前用户、产品/项目/版本作用域和审计事件入口,避免 V2.5 做 RBAC 时返工。
|
|
||||||
|
|
||||||
## V2 分阶段交付链路
|
## V2 分阶段交付链路
|
||||||
|
|
||||||
@@ -34,12 +33,18 @@ V2.4 的目标是把业务主数据源从 AppData JSONB 文档切换到 PostgreS
|
|||||||
- 版本详情已有需求、调研、产品方案、UI、开发任务、测试用例、Bug、概览等核心 Tab;渲染重的路径优先接入 V2.2 关系表快读,并保留 AppData fallback。
|
- 版本详情已有需求、调研、产品方案、UI、开发任务、测试用例、Bug、概览等核心 Tab;渲染重的路径优先接入 V2.2 关系表快读,并保留 AppData fallback。
|
||||||
- 后端已落地 Product、Requirement 领域 CRUD,DataModule AppData 乐观锁,V2.2 快读 API,V2.3 AppData 写后同步关系表,AI Provider 抽象和健康版本接口。
|
- 后端已落地 Product、Requirement 领域 CRUD,DataModule AppData 乐观锁,V2.2 快读 API,V2.3 AppData 写后同步关系表,AI Provider 抽象和健康版本接口。
|
||||||
- Prisma schema 已包含 Product、Project、Version、Requirement、VersionPlan、DevTask、TestCase、Bug、WorkActivity、Xiaobao、AiLog、AppData 等关系模型;高增长表的分区 migration 已落地。
|
- Prisma schema 已包含 Product、Project、Version、Requirement、VersionPlan、DevTask、TestCase、Bug、WorkActivity、Xiaobao、AiLog、AppData 等关系模型;高增长表的分区 migration 已落地。
|
||||||
- 主写入源仍处在兼容窗口:多数前端 store 继续通过 `apps/web/lib/server-data.ts` 的 `loadServerData` / `saveServerData` 写 AppData;`useProductStore` 仍以 `products-overview` 文档作为产品/项目/版本树主写入。
|
- 本 V2.8 执行线程以前提“V2.4 领域 CRUD 主写迁移已完成验收”推进;本阶段不重新逐项复核领域 API 清单。
|
||||||
- Project、Version、VersionPlan、DevTask、TestCase、Bug、Member、TaskCategory、TaskWorklog、Overtime 等领域写 API 尚未完整替代 AppData Store。若后续称为 V2.4,应理解为“领域 CRUD 迁移阶段”,不是 V2.3 已完成内容。
|
- V2.8 新增运维交付物集中在 `scripts/`、`.github/workflows/deploy-production.yml`、`deploy/monitoring/`、`docs/runbooks/`、`docs/deployment.md` 和 `docs/production-readiness.md`。
|
||||||
- `packages/shared` 中仍保留早期枚举口径;切换领域 API 时需要统一为当前前端业务状态机。
|
- AppData 退场、RBAC/审计、性能和小宝后台化仍通过 production readiness 证据项追踪,避免把运维稳定版误当成业务治理已全部完成。
|
||||||
|
|
||||||
### 已完成(按时间倒序)
|
### 已完成(按时间倒序)
|
||||||
|
|
||||||
|
**2026-07-08**
|
||||||
|
- V2.8 production ops closure started: added PostgreSQL backup, fresh DB restore with explicit overwrite confirmation, and `server_data` volume backup automation.
|
||||||
|
- Added release smoke suite and wired GitHub Actions deployment verification to runtime version, frontend root, products, V2.2 read path, and AI config checks.
|
||||||
|
- Added optional monitoring profile with Prometheus, Grafana, Loki, Promtail, postgres-exporter, node-exporter, cAdvisor, and blackbox-exporter.
|
||||||
|
- Added migration rollback, AppData retirement, Xiaobao background jobs runbooks, and production readiness evidence checklist.
|
||||||
|
|
||||||
**2026-07-06**
|
**2026-07-06**
|
||||||
- Added production CI/CD flow: GitHub Actions builds `web` and `server` Docker images, pushes immutable commit-SHA tags to GHCR, deploys by SSH, pulls images on the server, runs `pnpm --filter server db:deploy`, restarts Compose, and verifies `/api/v1/health/version`.
|
- Added production CI/CD flow: GitHub Actions builds `web` and `server` Docker images, pushes immutable commit-SHA tags to GHCR, deploys by SSH, pulls images on the server, runs `pnpm --filter server db:deploy`, restarts Compose, and verifies `/api/v1/health/version`.
|
||||||
- Added runtime version metadata: backend `GET /api/v1/health/version`, Docker build args/env, and a frontend refresh banner when browser assets are older than the server runtime.
|
- Added runtime version metadata: backend `GET /api/v1/health/version`, Docker build args/env, and a frontend refresh banner when browser assets are older than the server runtime.
|
||||||
@@ -117,12 +122,12 @@ V2.4 的目标是把业务主数据源从 AppData JSONB 文档切换到 PostgreS
|
|||||||
|
|
||||||
### 进行中
|
### 进行中
|
||||||
|
|
||||||
- V2.4 领域 CRUD 主写迁移:从 AppData JSONB 主写入切换到关系表 API,并随 API 落基础权限、审计和查询性能边界。
|
- V2.8 生产硬化稳定版:补齐备份恢复、发布 smoke、监控告警、日志检索、迁移回滚和运维证据闭环。
|
||||||
- 项目详情页 VersionCard 状态胶囊数据联动(部分已完成)
|
- 项目详情页 VersionCard 状态胶囊数据联动(部分已完成)
|
||||||
|
|
||||||
## V2 — 后端接入
|
## V2 — 后端接入
|
||||||
|
|
||||||
NestJS + Prisma + PostgreSQL 已接入到 V2.3。第一阶段用 `app_data` JSONB 文档表承接现有 store 数据形状,避免浏览器清站点数据导致业务数据丢失;第二阶段已建立分区关系表、V2.2 快读 API 和 V2.3 AppData 写后同步。当前 V2.4 才是逐领域启用写 API,让前端 store 从 AppData 主写入迁移到领域 CRUD;AppData 后续只保留为迁移兼容层。
|
NestJS + Prisma + PostgreSQL 已完成 V2.1 至 V2.4 的后端迁移主线。第一阶段用 `app_data` JSONB 文档表承接现有 store 数据形状,避免浏览器清站点数据导致业务数据丢失;第二阶段建立分区关系表、V2.2 快读 API 和 V2.3 AppData 写后同步;V2.4 完成领域 CRUD 主写迁移验收。V2.8 不再新增业务主写迁移范围,而是把生产运维、备份恢复、监控告警和回滚手册补齐。
|
||||||
|
|
||||||
### 关键任务
|
### 关键任务
|
||||||
|
|
||||||
@@ -141,7 +146,7 @@ NestJS + Prisma + PostgreSQL 已接入到 V2.3。第一阶段用 `app_data` JSON
|
|||||||
|
|
||||||
当前不做本地导入导出。清站点数据后浏览器旧数据无法恢复,后续新增数据直接写入 PostgreSQL。若以后需要迁移旧浏览器数据,再单独做管理员导入工具。
|
当前不做本地导入导出。清站点数据后浏览器旧数据无法恢复,后续新增数据直接写入 PostgreSQL。若以后需要迁移旧浏览器数据,再单独做管理员导入工具。
|
||||||
|
|
||||||
## V2.4 — 领域 CRUD 迁移(当前阶段)
|
## V2.4 — 领域 CRUD 迁移(已完成前提)
|
||||||
|
|
||||||
目标是让关系表从“快读 + AppData 同步副本”逐步升级为主写入路径。迁移顺序应优先选择写入频率高、实体边界清晰、已经在 V2.2 mapper 中稳定的领域:
|
目标是让关系表从“快读 + AppData 同步副本”逐步升级为主写入路径。迁移顺序应优先选择写入频率高、实体边界清晰、已经在 V2.2 mapper 中稳定的领域:
|
||||||
|
|
||||||
@@ -221,7 +226,7 @@ V2.4 推进前必须先统一 `packages/shared` 的状态枚举与当前前端
|
|||||||
|------|------|
|
|------|------|
|
||||||
| V1 业务流程打磨 | 进行中 |
|
| V1 业务流程打磨 | 进行中 |
|
||||||
| V1 朋友试用反馈 | 持续中 |
|
| V1 朋友试用反馈 | 持续中 |
|
||||||
| V2 后端接入 | 进行中(V2.1/V2.2/V2.3 已完成,V2.4 主写迁移中) |
|
| V2 后端接入 | V2.1-V2.4 已完成;V2.8 运维闭环进行中 |
|
||||||
| V3 AI 集成 | 等 V2 数据沉淀 |
|
| V3 AI 集成 | 等 V2 数据沉淀 |
|
||||||
| 公开发布 | TBD |
|
| 公开发布 | TBD |
|
||||||
**2026-06-26**
|
**2026-06-26**
|
||||||
|
|||||||
72
docs/runbooks/appdata-retirement.md
Normal file
72
docs/runbooks/appdata-retirement.md
Normal file
@@ -0,0 +1,72 @@
|
|||||||
|
# AppData Retirement Runbook
|
||||||
|
|
||||||
|
Use this when retiring an AppData key after its domain writes have moved to relation-table APIs. The order is fixed: backup, measure, freeze writes, compare, remove fallback, archive.
|
||||||
|
|
||||||
|
## Scope Gate
|
||||||
|
|
||||||
|
Retire one AppData key family at a time. Good candidates have domain CRUD writes, read APIs, pagination boundaries, audit events, and a rollback path.
|
||||||
|
|
||||||
|
## Preparation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pnpm backup:postgres -- --env-file .env.production
|
||||||
|
pnpm backup:server-data -- --env-file .env.production
|
||||||
|
pnpm deploy:smoke -- --base-url http://localhost
|
||||||
|
```
|
||||||
|
|
||||||
|
Record the key family being retired, the owning domain API, and the relation tables that replace it.
|
||||||
|
|
||||||
|
## Count And Consistency Checks
|
||||||
|
|
||||||
|
Run counts before disabling writes.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "select key, jsonb_array_length(value) as appdata_rows from app_data where key in ('products-overview','requirements','version-plans','dev-tasks','test-cases','bugs','members','task-categories','task-worklogs','overtime') order by key;"
|
||||||
|
docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "select 'requirements' as table_name, count(*) from requirements union all select 'version_plans', count(*) from version_plans union all select 'dev_tasks', count(*) from dev_tasks union all select 'test_cases', count(*) from test_cases union all select 'bugs', count(*) from bugs;"
|
||||||
|
```
|
||||||
|
|
||||||
|
For partitioned entities, also check missing partition keys.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "select 'requirements_missing_product' as check_name, count(*) from requirements where product_id is null union all select 'dev_tasks_missing_version', count(*) from dev_tasks where version_id is null union all select 'test_cases_missing_version', count(*) from test_cases where version_id is null union all select 'bugs_missing_version', count(*) from bugs where version_id is null;"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Disable AppData Writes
|
||||||
|
|
||||||
|
1. Merge the domain-specific frontend store change that stops calling `saveServerData` for the retired key.
|
||||||
|
2. Keep AppData read fallback for one release while relation reads are verified.
|
||||||
|
3. Deploy and run smoke tests.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pnpm deploy:smoke -- --base-url http://localhost --expected-version "$APP_VERSION"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Remove Fallback
|
||||||
|
|
||||||
|
Remove AppData read fallback only after one successful release where:
|
||||||
|
|
||||||
|
- Domain writes went through relation APIs.
|
||||||
|
- AppData row counts did not grow for the retired key.
|
||||||
|
- V2.2 read paths and page workflows returned expected data.
|
||||||
|
- No `AppData relation sync failed` logs appeared during the observation window.
|
||||||
|
|
||||||
|
## Archive
|
||||||
|
|
||||||
|
Export retired keys before any later table cleanup.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "copy (select key, value, updated_at from app_data where key in ('dev-tasks','test-cases','bugs')) to stdout with csv header" > backups/postgres/appdata-retired-executions-20260708.csv
|
||||||
|
```
|
||||||
|
|
||||||
|
## Rollback
|
||||||
|
|
||||||
|
- If relation writes fail but AppData still has fresh data, roll back to the previous app image that still writes AppData.
|
||||||
|
- If AppData writes were already disabled and relation writes are bad, restore PostgreSQL from the backup made at the start of this runbook.
|
||||||
|
- If only read fallback removal caused the issue, roll back app images first and leave the database unchanged.
|
||||||
|
|
||||||
|
## Data Risks
|
||||||
|
|
||||||
|
- Removing fallback too early can hide valid historical JSON rows that were never mapped into relation tables.
|
||||||
|
- Re-enabling old AppData writes after relation writes have accepted new edits can overwrite newer relation state through compatibility sync.
|
||||||
|
- AppData exports can contain business-sensitive text; store backup CSV files in the same restricted location as database dumps.
|
||||||
|
|
||||||
100
docs/runbooks/migration-rollback.md
Normal file
100
docs/runbooks/migration-rollback.md
Normal file
@@ -0,0 +1,100 @@
|
|||||||
|
# Migration Rollback Runbook
|
||||||
|
|
||||||
|
Use this when a production release, Prisma migration, or data migration causes failed smoke tests, missing data, bad query performance, or unsafe writes.
|
||||||
|
|
||||||
|
## First Response
|
||||||
|
|
||||||
|
1. Freeze new releases and ask product owners to pause bulk edits.
|
||||||
|
2. Capture current state before changing anything.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
date -u
|
||||||
|
git rev-parse HEAD
|
||||||
|
docker compose --env-file .env.production -f docker-compose.prod.yml ps
|
||||||
|
pnpm backup:postgres -- --env-file .env.production
|
||||||
|
pnpm backup:server-data -- --env-file .env.production
|
||||||
|
```
|
||||||
|
|
||||||
|
3. Run the read-only release smoke test.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pnpm deploy:smoke -- --base-url http://localhost --expected-version "$APP_VERSION"
|
||||||
|
```
|
||||||
|
|
||||||
|
4. Check the database and latest server logs.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres pg_isready -U "$POSTGRES_USER" -d "$POSTGRES_DB"
|
||||||
|
docker compose --env-file .env.production -f docker-compose.prod.yml logs --tail=200 server
|
||||||
|
```
|
||||||
|
|
||||||
|
## Decision Points
|
||||||
|
|
||||||
|
- App code bad, database healthy: roll back `WEB_IMAGE` and `SERVER_IMAGE` to the previous commit image tags, then restart Compose.
|
||||||
|
- Migration applied but only additive: roll back app images first; leave schema in place if old code remains compatible.
|
||||||
|
- Migration changed or removed data: restore a fresh database from the last known-good backup into a temporary database, compare row counts, then decide whether to restore production.
|
||||||
|
- AppData compatibility issue: keep production database intact, re-enable the previous app image that still reads the AppData fallback, and preserve the failing release backup for analysis.
|
||||||
|
|
||||||
|
## App Image Rollback
|
||||||
|
|
||||||
|
Set previous image tags in `.env.production`. Use the previous successful GitHub Actions run to identify the image tags.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export PREVIOUS_WEB_IMAGE=ghcr.io/company/ftb-project-management/web:abc1234
|
||||||
|
export PREVIOUS_SERVER_IMAGE=ghcr.io/company/ftb-project-management/server:abc1234
|
||||||
|
sed -i "s|^WEB_IMAGE=.*|WEB_IMAGE=${PREVIOUS_WEB_IMAGE}|" .env.production
|
||||||
|
sed -i "s|^SERVER_IMAGE=.*|SERVER_IMAGE=${PREVIOUS_SERVER_IMAGE}|" .env.production
|
||||||
|
docker compose --env-file .env.production -f docker-compose.prod.yml pull web server
|
||||||
|
docker compose --env-file .env.production -f docker-compose.prod.yml up -d --remove-orphans
|
||||||
|
pnpm deploy:smoke -- --base-url http://localhost --expected-version abc1234
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fresh Database Restore
|
||||||
|
|
||||||
|
Never restore over production until the backup has been rehearsed into a temporary database.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export BACKUP_FILE=backups/postgres/ftb_pm-postgres-20260708T120000Z.dump
|
||||||
|
pnpm restore:postgres -- \
|
||||||
|
--dry-run \
|
||||||
|
--confirm-overwrite \
|
||||||
|
--env-file .env.production \
|
||||||
|
--input "$BACKUP_FILE" \
|
||||||
|
--target-db ftb_pm_restore_check
|
||||||
|
pnpm restore:postgres -- \
|
||||||
|
--confirm-overwrite \
|
||||||
|
--env-file .env.production \
|
||||||
|
--input "$BACKUP_FILE" \
|
||||||
|
--target-db ftb_pm_restore_check
|
||||||
|
```
|
||||||
|
|
||||||
|
Compare critical row counts before touching production.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres psql -U "$POSTGRES_USER" -d ftb_pm_restore_check -c "select 'products' as table_name, count(*) from products union all select 'requirements', count(*) from requirements union all select 'versions', count(*) from versions union all select 'dev_tasks', count(*) from dev_tasks union all select 'test_cases', count(*) from test_cases union all select 'bugs', count(*) from bugs;"
|
||||||
|
```
|
||||||
|
|
||||||
|
If the temporary restore is healthy and production data is unsafe, restore production with explicit overwrite confirmation.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pnpm restore:postgres -- \
|
||||||
|
--confirm-overwrite \
|
||||||
|
--env-file .env.production \
|
||||||
|
--input "$BACKUP_FILE"
|
||||||
|
docker compose --env-file .env.production -f docker-compose.prod.yml up -d --remove-orphans
|
||||||
|
pnpm deploy:smoke -- --base-url http://localhost
|
||||||
|
```
|
||||||
|
|
||||||
|
## Data Risks
|
||||||
|
|
||||||
|
- PostgreSQL restore is destructive for the target database because the script terminates connections, drops the target database, recreates it, and runs `pg_restore`.
|
||||||
|
- Restoring PostgreSQL does not restore `server_data`; keep AI provider config backup files with the same incident bundle.
|
||||||
|
- AppData and relation tables can diverge during compatibility windows. Before deleting or restoring, preserve both the failing production backup and the known-good backup.
|
||||||
|
- If users continued editing during the incident, record the time window and decide whether those edits must be replayed manually after restore.
|
||||||
|
|
||||||
|
## Closeout
|
||||||
|
|
||||||
|
1. Save the failed release SHA, rollback SHA, backup file names, smoke output, and row-count evidence in the incident notes.
|
||||||
|
2. Keep the failed backup until the next successful release has completed smoke tests and one business-day observation.
|
||||||
|
3. Add a regression test or runbook correction before re-attempting the migration.
|
||||||
|
|
||||||
55
docs/runbooks/xiaobao-background-jobs.md
Normal file
55
docs/runbooks/xiaobao-background-jobs.md
Normal file
@@ -0,0 +1,55 @@
|
|||||||
|
# Xiaobao Background Jobs Runbook
|
||||||
|
|
||||||
|
Current V2.8 production monitoring supports Xiaobao staleness detection. The first production implementation is still page-triggered: opening `/xiaobao-warning` computes risk, saves snapshots, and lets V2.3 sync refresh summaries. A future scheduler must preserve the same idempotent data contract.
|
||||||
|
|
||||||
|
## Alert Triage
|
||||||
|
|
||||||
|
When `FtbXiaobaoSummaryStale` fires:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "select version_id, dirty, updated_at, recomputed_at, risk_level, risk_score from xiaobao_risk_summaries where dirty = true or updated_at < now() - interval '6 hours' order by updated_at asc limit 20;"
|
||||||
|
docker compose --env-file .env.production -f docker-compose.prod.yml logs --tail=200 server | grep -E "xiaobao|AppData relation sync failed|Slow Prisma query"
|
||||||
|
```
|
||||||
|
|
||||||
|
Decision points:
|
||||||
|
|
||||||
|
- Rows are dirty after active version edits: ask a manager to open `/xiaobao-warning` once, then verify summaries refresh.
|
||||||
|
- Rows remain dirty and server logs show sync failures: treat as AppData relation sync incident and follow `migration-rollback.md`.
|
||||||
|
- Rows are stale but no product release is near: keep monitoring and schedule a manual refresh before the next release decision meeting.
|
||||||
|
- Rows are stale for a release due today: refresh manually and have the release owner review the resulting risk explanation before ship/no-ship decision.
|
||||||
|
|
||||||
|
## Manual Refresh Path
|
||||||
|
|
||||||
|
1. Log in as a user with `xiaobao.warning:manage`.
|
||||||
|
2. Open `/xiaobao-warning`.
|
||||||
|
3. Wait until AI interpretation status is no longer generating for high-risk versions.
|
||||||
|
4. Re-run the stale-summary query.
|
||||||
|
5. Run the release smoke test.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pnpm deploy:smoke -- --base-url http://localhost
|
||||||
|
```
|
||||||
|
|
||||||
|
## Future Scheduler Rules
|
||||||
|
|
||||||
|
When a background job is introduced, it must:
|
||||||
|
|
||||||
|
- Read unfinished versions by relation-table scope, not by full AppData document scan.
|
||||||
|
- Use one idempotency key per `versionId + riskSignature + snapshotDate`.
|
||||||
|
- Write snapshots append-only and upsert summaries by `version_id`.
|
||||||
|
- Mark failures with structured logs containing `xiaobao background job failed`.
|
||||||
|
- Retry transient AI failures with backoff and keep rule-based risk output even when AI interpretation fails.
|
||||||
|
- Never mutate Version, Requirement, DevTask, TestCase, Bug, or Member data.
|
||||||
|
|
||||||
|
## Monitoring Expectations
|
||||||
|
|
||||||
|
- `FtbXiaobaoSummaryStale` alerts on dirty or older-than-6-hour summaries.
|
||||||
|
- `FtbJobFailureLogBurst` alerts when sync or future job failure log counters increase.
|
||||||
|
- Grafana dashboard shows the stale summary count and matching server warning/error logs.
|
||||||
|
|
||||||
|
## Data Risks
|
||||||
|
|
||||||
|
- Recomputing Xiaobao summaries can change release risk badges and manager decisions; record manual refresh time in release notes.
|
||||||
|
- AI interpretation cache is explanatory only. Do not restore or delete business entities to fix a bad explanation.
|
||||||
|
- If stale summaries are caused by relation sync failure, refreshing the page can mask the symptom without fixing the underlying sync path. Preserve logs before restarting services.
|
||||||
|
|
||||||
@@ -14,6 +14,7 @@
|
|||||||
"backup:postgres": "node scripts/backup-postgres.mjs",
|
"backup:postgres": "node scripts/backup-postgres.mjs",
|
||||||
"restore:postgres": "node scripts/restore-postgres.mjs",
|
"restore:postgres": "node scripts/restore-postgres.mjs",
|
||||||
"backup:server-data": "node scripts/backup-server-data.mjs",
|
"backup:server-data": "node scripts/backup-server-data.mjs",
|
||||||
|
"docs:check-runbooks": "node scripts/check-runbook-placeholders.mjs --paths docs/runbooks docs/production-readiness.md",
|
||||||
"deploy:local:build": "docker compose --env-file .env.local-server -f docker-compose.local.yml build",
|
"deploy:local:build": "docker compose --env-file .env.local-server -f docker-compose.local.yml build",
|
||||||
"deploy:local:up": "docker compose --env-file .env.local-server -f docker-compose.local.yml up -d",
|
"deploy:local:up": "docker compose --env-file .env.local-server -f docker-compose.local.yml up -d",
|
||||||
"deploy:local:down": "docker compose --env-file .env.local-server -f docker-compose.local.yml down",
|
"deploy:local:down": "docker compose --env-file .env.local-server -f docker-compose.local.yml down",
|
||||||
|
|||||||
87
scripts/check-runbook-placeholders.mjs
Normal file
87
scripts/check-runbook-placeholders.mjs
Normal file
@@ -0,0 +1,87 @@
|
|||||||
|
#!/usr/bin/env node
|
||||||
|
import { existsSync, readFileSync, readdirSync, statSync } from 'node:fs';
|
||||||
|
import { extname, join } from 'node:path';
|
||||||
|
import { pathToFileURL } from 'node:url';
|
||||||
|
import { option, parseArgs } from './ops-utils.mjs';
|
||||||
|
|
||||||
|
const forbiddenPatterns = [
|
||||||
|
{ name: 'TBD', pattern: /\bTBD\b/i },
|
||||||
|
{ name: 'TODO', pattern: /\bTODO\b/i },
|
||||||
|
{ name: 'fill-in', pattern: /fill in|fill-in/i },
|
||||||
|
{ name: 'angle-token', pattern: /<[^>\n]+>/ },
|
||||||
|
{ name: 'Chinese pending marker', pattern: /待定|占位/ },
|
||||||
|
];
|
||||||
|
|
||||||
|
function usage() {
|
||||||
|
return `Usage: node scripts/check-runbook-placeholders.mjs --paths <path> [path...]
|
||||||
|
|
||||||
|
Scans Markdown runbooks for unresolved placeholder markers.
|
||||||
|
`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function collectMarkdownFiles(path) {
|
||||||
|
if (!existsSync(path)) {
|
||||||
|
throw new Error(`Path not found: ${path}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
const stat = statSync(path);
|
||||||
|
if (stat.isFile()) return extname(path) === '.md' ? [path] : [];
|
||||||
|
if (!stat.isDirectory()) return [];
|
||||||
|
|
||||||
|
const files = [];
|
||||||
|
for (const entry of readdirSync(path)) {
|
||||||
|
files.push(...collectMarkdownFiles(join(path, entry)));
|
||||||
|
}
|
||||||
|
return files;
|
||||||
|
}
|
||||||
|
|
||||||
|
function requestedPaths(argv) {
|
||||||
|
const args = parseArgs(argv);
|
||||||
|
if (args.flags.has('help')) return { help: true, paths: [] };
|
||||||
|
const first = option(args, 'paths');
|
||||||
|
const paths = [first, ...args.positionals].filter(Boolean);
|
||||||
|
if (paths.length === 0) {
|
||||||
|
throw new Error('Missing --paths <path> [path...]');
|
||||||
|
}
|
||||||
|
return { help: false, paths };
|
||||||
|
}
|
||||||
|
|
||||||
|
export function scanFiles(paths) {
|
||||||
|
const files = paths.flatMap(collectMarkdownFiles);
|
||||||
|
const findings = [];
|
||||||
|
for (const file of files) {
|
||||||
|
const lines = readFileSync(file, 'utf8').split(/\r?\n/);
|
||||||
|
lines.forEach((line, index) => {
|
||||||
|
for (const forbidden of forbiddenPatterns) {
|
||||||
|
if (forbidden.pattern.test(line)) {
|
||||||
|
findings.push(`${file}:${index + 1} ${forbidden.name}: ${line.trim()}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return { files, findings };
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function main(argv = process.argv.slice(2)) {
|
||||||
|
const { help, paths } = requestedPaths(argv);
|
||||||
|
if (help) {
|
||||||
|
process.stdout.write(usage());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const { files, findings } = scanFiles(paths);
|
||||||
|
if (findings.length > 0) {
|
||||||
|
process.stderr.write(`${findings.join('\n')}\n`);
|
||||||
|
throw new Error(`Runbook placeholder scan failed: ${findings.length} finding(s)`);
|
||||||
|
}
|
||||||
|
|
||||||
|
process.stdout.write(`Runbook placeholder scan passed (${files.length} file(s)).\n`);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (import.meta.url === pathToFileURL(process.argv[1]).href) {
|
||||||
|
main().catch((error) => {
|
||||||
|
process.stderr.write(`${error.message}\n`);
|
||||||
|
process.exit(1);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
@@ -138,4 +138,15 @@ describe('production ops scripts', () => {
|
|||||||
assert.doesNotMatch(combined, /sk-ant-[A-Za-z0-9]/);
|
assert.doesNotMatch(combined, /sk-ant-[A-Za-z0-9]/);
|
||||||
assert.doesNotMatch(combined, /hooks\.slack\.com\/services\//);
|
assert.doesNotMatch(combined, /hooks\.slack\.com\/services\//);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it('passes the runbook placeholder scan', () => {
|
||||||
|
const result = runScript('scripts/check-runbook-placeholders.mjs', [
|
||||||
|
'--paths',
|
||||||
|
'docs/runbooks',
|
||||||
|
'docs/production-readiness.md',
|
||||||
|
]);
|
||||||
|
|
||||||
|
assert.equal(result.status, 0, result.stderr);
|
||||||
|
assert.match(result.stdout, /Runbook placeholder scan passed/);
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -93,6 +93,14 @@ const checks = [
|
|||||||
file: 'scripts/backup-server-data.mjs',
|
file: 'scripts/backup-server-data.mjs',
|
||||||
snippets: ['server_data', 'tar -czf', '--dry-run'],
|
snippets: ['server_data', 'tar -czf', '--dry-run'],
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
file: 'scripts/check-runbook-placeholders.mjs',
|
||||||
|
snippets: ['Runbook placeholder scan passed', 'Runbook placeholder scan failed', '--paths'],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
file: 'package.json',
|
||||||
|
snippets: ['docs:check-runbooks', 'docs/runbooks', 'docs/production-readiness.md'],
|
||||||
|
},
|
||||||
{
|
{
|
||||||
file: 'docker-compose.local.yml',
|
file: 'docker-compose.local.yml',
|
||||||
snippets: [
|
snippets: [
|
||||||
@@ -138,6 +146,31 @@ const checks = [
|
|||||||
'FtbXiaobaoSummaryStale',
|
'FtbXiaobaoSummaryStale',
|
||||||
],
|
],
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
file: 'docs/runbooks/migration-rollback.md',
|
||||||
|
snippets: ['Fresh Database Restore', 'Data Risks', 'pnpm restore:postgres'],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
file: 'docs/runbooks/appdata-retirement.md',
|
||||||
|
snippets: ['Disable AppData Writes', 'Remove Fallback', 'Data Risks'],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
file: 'docs/runbooks/xiaobao-background-jobs.md',
|
||||||
|
snippets: ['FtbXiaobaoSummaryStale', 'Future Scheduler Rules', 'Data Risks'],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
file: 'docs/production-readiness.md',
|
||||||
|
snippets: [
|
||||||
|
'Backup and restore',
|
||||||
|
'Smoke tests',
|
||||||
|
'Monitoring',
|
||||||
|
'Audit',
|
||||||
|
'RBAC',
|
||||||
|
'Consistency',
|
||||||
|
'Performance',
|
||||||
|
'Post-release',
|
||||||
|
],
|
||||||
|
},
|
||||||
{
|
{
|
||||||
file: 'deploy/monitoring/postgres/postgres-queries.yml',
|
file: 'deploy/monitoring/postgres/postgres-queries.yml',
|
||||||
snippets: ['xiaobao_risk_summaries', 'dirty = true', 'stale_summary_count'],
|
snippets: ['xiaobao_risk_summaries', 'dirty = true', 'stale_summary_count'],
|
||||||
@@ -183,6 +216,8 @@ const checks = [
|
|||||||
'pnpm backup:postgres',
|
'pnpm backup:postgres',
|
||||||
'pnpm restore:postgres',
|
'pnpm restore:postgres',
|
||||||
'pnpm backup:server-data',
|
'pnpm backup:server-data',
|
||||||
|
'docs/runbooks/migration-rollback.md',
|
||||||
|
'docs/production-readiness.md',
|
||||||
'--profile monitoring',
|
'--profile monitoring',
|
||||||
'pnpm db:migrate',
|
'pnpm db:migrate',
|
||||||
'NEXT_PUBLIC_API_URL',
|
'NEXT_PUBLIC_API_URL',
|
||||||
|
|||||||
Reference in New Issue
Block a user