merge: 集成V2.8 生产硬化与运维闭环
# Conflicts: # .gitignore # docs/deployment.md # docs/roadmap.md # package.json
This commit is contained in:
。
@@ -175,12 +175,13 @@ cp .env.production.example .env.production
|
||||
5. 执行 `docker compose --env-file .env.production -f docker-compose.prod.yml pull web server` 拉取本次 SHA 镜像。
|
||||
6. 启动数据库与 Redis，执行 `pnpm --filter server db:deploy`。
|
||||
7. 执行 `docker compose --env-file .env.production -f docker-compose.prod.yml up -d --remove-orphans` 重启服务。
|
||||
8. 通过 `/api/v1/health/version` 校验运行中的后端版本是否等于本次 commit SHA。
|
||||
8. 运行发布 smoke test：校验 `/api/v1/health/version`、前端首页、产品页、产品 API、V2.2 读路径和 AI 配置端点，并确认运行中的后端版本等于本次 commit SHA。
|
||||
|
||||
如果最后一步失败，Actions 会红掉，说明“代码已合并”不等于“线上容器已更新”。本地或服务器也可以手工运行：
|
||||
如果最后一步失败，Actions 会红掉，说明“代码已合并”不等于“线上容器已更新”或关键读路径不可用。本地或服务器也可以手工运行：
|
||||
|
||||
```bash
|
||||
pnpm deploy:check-runtime http://localhost/api/v1/health/version <expected-commit-sha>
|
||||
pnpm deploy:smoke -- --base-url http://localhost --expected-version <expected-commit-sha>
|
||||
```
|
||||
|
||||
## Nginx 路由
|
||||
@@ -193,6 +194,30 @@ pnpm deploy:check-runtime http://localhost/api/v1/health/version <expected-commi
|
||||
|
||||
生产 Compose 默认只监听 HTTP 80。HTTPS 建议优先交给云负载均衡、CDN 或宿主机外层证书管理工具；如果要让本 Compose 内的 Nginx 直接处理 HTTPS，可以在后续增加证书 volume 和 443 server block。
|
||||
|
||||
## 监控与告警基线
|
||||
|
||||
V2.8 提供可选 `monitoring` profile，不影响默认生产启动。启用前先在 `.env.production` 设置 `GRAFANA_ADMIN_PASSWORD`，不要使用示例密码对外暴露 Grafana。
|
||||
|
||||
```bash
|
||||
docker compose --env-file .env.production -f docker-compose.prod.yml --profile monitoring up -d
|
||||
```
|
||||
|
||||
默认入口：
|
||||
|
||||
- Prometheus: `http://localhost:9090`
|
||||
- Grafana: `http://localhost:3002`
|
||||
|
||||
配置目录在 `deploy/monitoring/`。基线覆盖：
|
||||
|
||||
- `FtbPostgresDown`：PostgreSQL 不可用。
|
||||
- `FtbDiskPressure`：宿主机根分区低于 15% 可用空间。
|
||||
- `FtbSlowApiLogBurst`：服务端慢 API 日志在 10 分钟内超过阈值。
|
||||
- `FtbSlowPrismaLogBurst`：慢 Prisma 查询日志在 10 分钟内超过阈值。
|
||||
- `FtbJobFailureLogBurst`：AppData 同步、AI 调用或后台任务失败日志出现。
|
||||
- `FtbXiaobaoSummaryStale`：`xiaobao_risk_summaries` 存在 dirty 或超过 6 小时未更新的摘要。
|
||||
|
||||
Prometheus 只加载本地规则，不提交真实通知密钥。接入 Slack、企业微信、邮件等通知时，把 Alertmanager receiver 放在未跟踪的服务器文件或环境变量中。
|
||||
|
||||
## 数据持久化
|
||||
|
||||
生产 Compose 使用三个命名 volume：
|
||||
@@ -235,6 +260,76 @@ pnpm consistency:v25 -- --url http://localhost/api/v1/consistency
|
||||
- `warn` 需要记录原因；历史数据没有审计事件属于预期 warning，不阻断 V2.5。
|
||||
- 后台页面 `/admin/consistency` 展示同一份报告，需要当前用户具备 `consistency:view`。
|
||||
|
||||
## 备份与恢复
|
||||
|
||||
备份分两类：PostgreSQL 业务数据和 `server_data` volume 中的运行时配置。真实备份文件默认写到 `backups/`，该目录已加入 `.gitignore`，不要把 dump 或 tar 包提交到仓库。
|
||||
|
||||
先演练命令，不写文件：
|
||||
|
||||
```bash
|
||||
pnpm backup:postgres -- --dry-run --env-file .env.production
|
||||
pnpm backup:server-data -- --dry-run --env-file .env.production
|
||||
```
|
||||
|
||||
执行正式备份：
|
||||
|
||||
```bash
|
||||
pnpm backup:postgres -- --env-file .env.production
|
||||
pnpm backup:server-data -- --env-file .env.production
|
||||
```
|
||||
|
||||
PostgreSQL 备份使用 `pg_dump --format=custom --no-owner --no-acl`，便于跨环境恢复。`server_data` 备份使用只读 volume mount 和 `tar -czf`，覆盖 AI Provider 配置等后端运行时文件。
|
||||
|
||||
恢复数据库必须显式确认覆盖。默认命令会拒绝执行，防止误删现有库：
|
||||
|
||||
```bash
|
||||
pnpm restore:postgres -- --env-file .env.production --input backups/postgres/ftb_pm-postgres-20260708T120000Z.dump
|
||||
```
|
||||
|
||||
确认要把目标数据库重建为 fresh DB 后，再运行：
|
||||
|
||||
```bash
|
||||
pnpm restore:postgres -- \
|
||||
--env-file .env.production \
|
||||
--input backups/postgres/ftb_pm-postgres-20260708T120000Z.dump \
|
||||
--confirm-overwrite
|
||||
```
|
||||
|
||||
恢复前建议先 dry-run 看清将执行的 `psql/dropdb/createdb/pg_restore` 步骤：
|
||||
|
||||
```bash
|
||||
pnpm restore:postgres -- \
|
||||
--dry-run \
|
||||
--confirm-overwrite \
|
||||
--env-file .env.production \
|
||||
--input backups/postgres/ftb_pm-postgres-20260708T120000Z.dump
|
||||
```
|
||||
|
||||
如果要恢复到临时库做校验，不覆盖当前生产库：
|
||||
|
||||
```bash
|
||||
pnpm restore:postgres -- \
|
||||
--env-file .env.production \
|
||||
--input backups/postgres/ftb_pm-postgres-20260708T120000Z.dump \
|
||||
--target-db ftb_pm_restore_check \
|
||||
--confirm-overwrite
|
||||
```
|
||||
|
||||
## 运维 Runbooks
|
||||
|
||||
生产发布、迁移和异常处置优先使用这些手册：
|
||||
|
||||
- `docs/runbooks/migration-rollback.md`：发布失败、迁移失败、数据恢复和镜像回滚。
|
||||
- `docs/runbooks/appdata-retirement.md`：AppData key 分阶段退场、双读核对、归档和回滚。
|
||||
- `docs/runbooks/xiaobao-background-jobs.md`：小宝摘要 stale 告警、手动刷新和未来后台任务规则。
|
||||
- `docs/production-readiness.md`：生产发布前后证据清单。
|
||||
|
||||
提交前运行 runbook 扫描：
|
||||
|
||||
```bash
|
||||
pnpm docs:check-runbooks
|
||||
```
|
||||
|
||||
## 升级流程
|
||||
|
||||
```bash
|
||||
@@ -248,7 +343,8 @@ docker compose --env-file .env.production -f docker-compose.prod.yml exec server
|
||||
升级前建议先备份数据库：
|
||||
|
||||
```bash
|
||||
docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres pg_dump -U "$POSTGRES_USER" "$POSTGRES_DB" > ftb_pm_backup.sql
|
||||
pnpm backup:postgres -- --env-file .env.production
|
||||
pnpm backup:server-data -- --env-file .env.production
|
||||
```
|
||||
|
||||
## 常见排查
|
||||
|
||||
24
docs/production-readiness.md
Normal file
24
docs/production-readiness.md
Normal file
@@ -0,0 +1,24 @@
|
||||
# Production Readiness Checklist
|
||||
|
||||
Use this before a production release and again after the release smoke test. Each item requires evidence, not a verbal assertion.
|
||||
|
||||
| Area | Gate | Evidence |
|
||||
| --- | --- | --- |
|
||||
| Backup and restore | PostgreSQL backup dry-run and server_data backup dry-run are reviewed. | `pnpm backup:postgres -- --dry-run --env-file .env.production` and `pnpm backup:server-data -- --dry-run --env-file .env.production` output saved in release notes. |
|
||||
| Backup and restore | Restore command refuses overwrite without explicit confirmation. | `pnpm restore:postgres -- --env-file .env.production --input backups/postgres/ftb_pm-postgres-20260708T120000Z.dump` exits non-zero and names `--confirm-overwrite`. |
|
||||
| Backup and restore | Fresh DB restore rehearsal completed before destructive restore. | Temporary database restore command and row-count comparison from `docs/runbooks/migration-rollback.md`. |
|
||||
| Smoke tests | Release smoke is wired into GitHub Actions. | `.github/workflows/deploy-production.yml` runs `scripts/smoke-test-release.mjs` with `--expected-version`. |
|
||||
| Smoke tests | Manual smoke can be run against the target. | `pnpm deploy:smoke -- --base-url http://localhost --expected-version "$APP_VERSION"` output. |
|
||||
| Monitoring | Monitoring profile renders and can start without committed secrets. | `docker compose --env-file .env.production -f docker-compose.prod.yml --profile monitoring config` exits 0. |
|
||||
| Monitoring | Required alert rules exist. | `FtbPostgresDown`, `FtbDiskPressure`, `FtbSlowApiLogBurst`, `FtbSlowPrismaLogBurst`, `FtbJobFailureLogBurst`, and `FtbXiaobaoSummaryStale` visible in Prometheus. |
|
||||
| Audit | Domain write APIs record actor and resource scope where implemented. | API request sample or audit log sample for Product, Requirement, Project, Version, execution entities, and dictionary writes. |
|
||||
| RBAC | Project and version operations enforce Owner, Admin, Member, Viewer boundaries where enabled. | Permission matrix test result or manual account walkthrough attached to release notes. |
|
||||
| Consistency | AppData and relation tables are checked for migrated domains. | Count and missing-partition-key queries from `docs/runbooks/appdata-retirement.md`. |
|
||||
| Performance | Slow API and slow Prisma thresholds are configured. | `API_SLOW_REQUEST_MS` and `PRISMA_SLOW_QUERY_MS` values recorded, plus Grafana slow-log panels checked. |
|
||||
| Performance | V2.2 hot reads avoid full-table scans. | Requirement pool smoke uses `productId`; query plan or service test evidence attached for high-volume domains. |
|
||||
| Post-release | Runtime version matches the release SHA. | `/api/v1/health/version` payload or `pnpm deploy:check-runtime` output. |
|
||||
| Post-release | Product, requirement, workspace, Xiaobao, and AI config read paths respond. | `pnpm deploy:smoke` output attached. |
|
||||
| Post-release | On-call rollback path is known. | `docs/runbooks/migration-rollback.md` link included in release notes. |
|
||||
|
||||
Release owner signs off only after all required evidence is attached to the release notes or incident record.
|
||||
|
||||
@@ -1,26 +1,12 @@
|
||||
# 开发路线图
|
||||
|
||||
## 当前阶段：V2.7 已完成 — 下一阶段 V2.8 生产硬化与运维闭环集成
|
||||
## 当前阶段：V2.8 已完成 — 统一验证与生产交付收口
|
||||
|
||||
V2.7 已在关系表主源方向上补齐企业协作和治理能力：通知、评论与提及、项目成员治理、管理驾驶舱、治理字典、以及统一 RBAC/audit 适配器。V2.7 不新增 AppData 主存储。
|
||||
V2.8 已在既有生产 CI/CD 基线上补齐运维闭环：备份恢复演练、发布 smoke test、监控告警、日志检索、迁移回滚 runbook、AppData 退场 runbook、小宝后台化 runbook 和生产 readiness 证据清单。本阶段不重新设计业务流程，专注把生产发布和故障处置做成可验证、可复盘、可回滚的标准流程。
|
||||
|
||||
### 当前重点
|
||||
V2.4 已将高增长和核心业务领域从“AppData 主写 + 关系表同步副本”推进到“领域 CRUD 主写关系表 + AppData 兼容/迁移兜底”。V2.5 已收口后端权限、审计、AppData 禁写和一致性核对。V2.6 已完成大数据性能增强、小宝风险后台化、AI 解读队列和运行时 Ops 看板。V2.7 已补齐企业协作和治理能力，且不新增 AppData 主存储。
|
||||
|
||||
1. **协作通知**：通知记录、已读状态、NotificationBell，并覆盖 assignment / mention / risk_alert / overdue_item 稳定事件类型。
|
||||
2. **通用评论**：DevTask/TestCase/Bug/Requirement/VersionPlan 统一评论面板，支持 `@成员名` 和显式成员选择，创建/删除写 audit。
|
||||
3. **项目成员治理**：Owner/Admin/Member/Viewer 服务端强校验，禁止移除最后 Owner，角色变更写 audit。
|
||||
4. **管理驾驶舱**：只读关系表和 summary，聚合活跃版本、逾期、阻塞、风险和成员负载。
|
||||
5. **治理设置**：集中维护 task category、requirement type/platform/source，使用中的字典不可硬删，支持导入导出。
|
||||
|
||||
V2.4 已将高增长和核心业务领域从“AppData 主写 + 关系表同步副本”推进到“领域 CRUD 主写关系表 + AppData 兼容/迁移兜底”。V2.2 快读 API 和 V2.3 AppData 写后同步继续保留，但它们现在是兼容基础设施，不再是已迁移领域的数据新鲜度主链路。
|
||||
|
||||
V2.5 的目标是正式收口后端权限、审计、AppData 禁写和一致性核对。AppData 不能直接删除，必须按“禁写 → 双读核对 → 移除 fallback → 只读归档/导出 → 后续删表”的顺序推进。
|
||||
|
||||
V2.6 的目标是在关系表主源稳定后完成大数据性能增强、小宝风险后台化、AI 解读队列和运行时 Ops 看板，让高增长热路径、后台任务和风险摘要不再依赖页面打开。
|
||||
|
||||
V2.8 的目标是在现有生产部署基线上补齐备份恢复演练、发布 smoke test、监控告警、日志检索、迁移回滚和运维手册，形成生产交付稳定版。
|
||||
|
||||
### V2.5-V2.7 完成范围
|
||||
### V2.5-V2.8 完成范围
|
||||
|
||||
1. **RBAC 收口**：领域 mutation API 已接入服务端权限校验、资源作用域和当前用户上下文。
|
||||
2. **审计事件**：领域 mutation 通过 `audit_events` 写 append-only audit event，支持后台查询和敏感字段脱敏。
|
||||
@@ -37,6 +23,11 @@ V2.8 的目标是在现有生产部署基线上补齐备份恢复演练、发布
|
||||
13. **项目成员治理**：已补项目成员 Owner/Admin/Member/Viewer 服务端治理，禁止移除最后 Owner，角色变更写审计。
|
||||
14. **管理驾驶舱**：已补只读关系表和 summary 的管理概览，聚合活跃版本、逾期、阻塞、风险和成员负载。
|
||||
15. **治理设置**：已补 task category、requirement type/platform/source 等治理字典能力，使用中的字典不可硬删，支持导入导出。
|
||||
16. **备份恢复自动化**：已补 PostgreSQL dump、`server_data` volume 备份、fresh DB restore dry-run 和显式覆盖确认。
|
||||
17. **发布 smoke test**：已补部署后 runtime version、前端根页、产品页、产品 API、V2.2 读路径和 AI 配置校验。
|
||||
18. **监控告警基线**：已补 Prometheus/Grafana/Loki/Promtail 可选 profile，覆盖慢 API、慢 Prisma、任务失败、小宝摘要 stale、磁盘压力和 DB 可用性。
|
||||
19. **运维 runbook**：已补迁移回滚、AppData 退场、小宝后台化处置步骤、决策点和数据风险。
|
||||
20. **生产 readiness 清单**：已补 backup/restore、smoke、monitoring、audit、RBAC、consistency、performance 和 post-release verification 证据项。
|
||||
|
||||
## V2 分阶段交付链路
|
||||
|
||||
@@ -58,7 +49,7 @@ V2.8 的目标是在现有生产部署基线上补齐备份恢复演练、发布
|
||||
- 项目已经不是早期骨架。前端业务功能已覆盖产品、项目、版本详情、需求池、工作台、成员/角色/任务类型、加班、小宝预警和 AI 配置等主要管理端路由。
|
||||
- 版本详情已有需求、调研、产品方案、UI、开发任务、测试用例、Bug、概览等核心 Tab；渲染重的路径优先接入关系表快读，并保留 AppData fallback。
|
||||
- 后端已落地 Product、Project、Version、Requirement、VersionPlan、DevTask、TestCase、Bug、Member、TaskCategory、TaskWorklog、Overtime、WorkActivity 领域 CRUD/write API。
|
||||
- Prisma schema 已包含 Product、Project、Version、Requirement、VersionPlan、DevTask、TestCase、Bug、WorkActivity、TaskWorklog、Overtime、Xiaobao、AiLog、AppData 等关系模型；高增长表的分区 migration 已落地。
|
||||
- Prisma schema 已包含 Product、Project、Version、Requirement、VersionPlan、DevTask、TestCase、Bug、WorkActivity、TaskWorklog、Overtime、Xiaobao、AiLog、AppData、Notification、Comment、ProjectMember、GovernanceDictionary 等关系模型；高增长表的分区 migration 已落地。
|
||||
- 主写入源已经切到领域 API：前端 store 优先调用 `apps/web/lib/domain-api.ts`，AppData 只保留兼容读取、历史核对和少量旧配置形状。
|
||||
- V2.5 后端服务端权限、审计和一致性控制面已启用：领域 mutation 使用 `@ProtectedMutation()`，审计写 `audit_events`，后台查询需要 `audit:view` / `consistency:view`。
|
||||
- 所有业务 AppData key 已明确冻结或只读归档；`PUT /api/v1/data/:key` 对这些 key 返回 `APP_DATA_WRITE_FROZEN`，`GET` 留作历史核对与归档。
|
||||
@@ -76,10 +67,16 @@ V2.8 的目标是在现有生产部署基线上补齐备份恢复演练、发布
|
||||
- V2.7.3 已新增项目成员治理 API 和项目页成员面板，服务端强校验 Owner/Admin/Member/Viewer 边界。
|
||||
- V2.7.4 已新增管理驾驶舱和治理设置，聚合关系表指标并维护治理字典。
|
||||
- V2.7.5 已新增协作治理 RBAC/audit adapter，避免新增模块绕开服务端权限和审计边界。
|
||||
- V2.8 新增运维交付物集中在 `scripts/`、`.github/workflows/deploy-production.yml`、`deploy/monitoring/`、`docs/runbooks/`、`docs/deployment.md` 和 `docs/production-readiness.md`。
|
||||
- AppData 退场、RBAC/审计、性能、小宝后台化、协作治理和生产发布都通过 production readiness 证据项追踪，避免把运维稳定版误当成一次性口头验收。
|
||||
|
||||
### 已完成（按时间倒序）
|
||||
|
||||
**2026-07-08**
|
||||
- V2.8 added PostgreSQL backup, fresh DB restore with explicit overwrite confirmation, and `server_data` volume backup automation.
|
||||
- V2.8 added release smoke suite and wired GitHub Actions deployment verification to runtime version, frontend root, products, V2.2 read path, and AI config checks.
|
||||
- V2.8 added optional monitoring profile with Prometheus, Grafana, Loki, Promtail, postgres-exporter, node-exporter, cAdvisor, and blackbox-exporter.
|
||||
- V2.8 added migration rollback, AppData retirement, Xiaobao background jobs runbooks, and production readiness evidence checklist.
|
||||
- V2.7.5 added shared collaboration/governance RBAC and audit adapters so notification, comment, project-member, management, and governance modules keep a single permission/audit boundary.
|
||||
- V2.7.4 added management and governance admin pages for relation-backed overview metrics and dictionary governance.
|
||||
- V2.7.3 added project-member governance APIs and project member panel with Owner/Admin/Member/Viewer safeguards.
|
||||
@@ -188,12 +185,12 @@ V2.8 的目标是在现有生产部署基线上补齐备份恢复演练、发布
|
||||
|
||||
### 进行中
|
||||
|
||||
- V2.8 生产硬化与运维闭环：备份恢复演练、发布 smoke test、监控告警、日志检索、迁移回滚和运维手册。
|
||||
- V2.8 统一验证与发布收口：合并后集中跑类型、测试、Prisma、部署、runbook、备份恢复和 smoke 验证。
|
||||
- 项目详情页 VersionCard 状态胶囊数据联动（部分已完成）
|
||||
|
||||
## V2 — 后端接入
|
||||
|
||||
NestJS + Prisma + PostgreSQL 已推进到 V2.7。第一阶段用 `app_data` JSONB 文档表承接现有 store 数据形状，避免浏览器清站点数据导致业务数据丢失；第二阶段建立分区关系表、V2.2 快读 API 和 V2.3 AppData 写后同步；第三阶段 V2.4 已逐领域启用写 API，让前端 store 从 AppData 主写入迁移到领域 CRUD 主写；第四阶段 V2.5 已冻结 AppData 业务写入并收口服务端 RBAC、审计和一致性校验；第五阶段 V2.6 已完成大数据性能和小宝后台化；第六阶段 V2.7 已补齐协作治理能力。
|
||||
NestJS + Prisma + PostgreSQL 已推进到 V2.8。第一阶段用 `app_data` JSONB 文档表承接现有 store 数据形状，避免浏览器清站点数据导致业务数据丢失；第二阶段建立分区关系表、V2.2 快读 API 和 V2.3 AppData 写后同步；第三阶段 V2.4 已逐领域启用写 API，让前端 store 从 AppData 主写入迁移到领域 CRUD 主写；第四阶段 V2.5 已冻结 AppData 业务写入并收口服务端 RBAC、审计和一致性校验；第五阶段 V2.6 已完成大数据性能和小宝后台化；第六阶段 V2.7 已补齐协作治理能力；第七阶段 V2.8 已补齐生产运维、备份恢复、监控告警和回滚手册。
|
||||
|
||||
### 关键任务
|
||||
|
||||
@@ -306,6 +303,6 @@ V2.5 完成后的保留边界：`GET /api/v1/data/:key` 仍可读历史 JSON；X
|
||||
|------|------|
|
||||
| V1 业务流程打磨 | 进行中 |
|
||||
| V1 朋友试用反馈 | 持续中 |
|
||||
| V2 后端接入 | 进行中（V2.7 已完成；V2.8 生产硬化与运维闭环待集成） |
|
||||
| V2 后端接入 | V2.1-V2.8 已完成，等待统一验证与发布授权 |
|
||||
| V3 AI 集成 | 等 V2 数据沉淀 |
|
||||
| 公开发布 | TBD |
|
||||
|
||||
72
docs/runbooks/appdata-retirement.md
Normal file
72
docs/runbooks/appdata-retirement.md
Normal file
@@ -0,0 +1,72 @@
|
||||
# AppData Retirement Runbook
|
||||
|
||||
Use this when retiring an AppData key after its domain writes have moved to relation-table APIs. The order is fixed: backup, measure, freeze writes, compare, remove fallback, archive.
|
||||
|
||||
## Scope Gate
|
||||
|
||||
Retire one AppData key family at a time. Good candidates have domain CRUD writes, read APIs, pagination boundaries, audit events, and a rollback path.
|
||||
|
||||
## Preparation
|
||||
|
||||
```bash
|
||||
pnpm backup:postgres -- --env-file .env.production
|
||||
pnpm backup:server-data -- --env-file .env.production
|
||||
pnpm deploy:smoke -- --base-url http://localhost
|
||||
```
|
||||
|
||||
Record the key family being retired, the owning domain API, and the relation tables that replace it.
|
||||
|
||||
## Count And Consistency Checks
|
||||
|
||||
Run counts before disabling writes.
|
||||
|
||||
```bash
|
||||
docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "select key, jsonb_array_length(value) as appdata_rows from app_data where key in ('products-overview','requirements','version-plans','dev-tasks','test-cases','bugs','members','task-categories','task-worklogs','overtime') order by key;"
|
||||
docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "select 'requirements' as table_name, count(*) from requirements union all select 'version_plans', count(*) from version_plans union all select 'dev_tasks', count(*) from dev_tasks union all select 'test_cases', count(*) from test_cases union all select 'bugs', count(*) from bugs;"
|
||||
```
|
||||
|
||||
For partitioned entities, also check missing partition keys.
|
||||
|
||||
```bash
|
||||
docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "select 'requirements_missing_product' as check_name, count(*) from requirements where product_id is null union all select 'dev_tasks_missing_version', count(*) from dev_tasks where version_id is null union all select 'test_cases_missing_version', count(*) from test_cases where version_id is null union all select 'bugs_missing_version', count(*) from bugs where version_id is null;"
|
||||
```
|
||||
|
||||
## Disable AppData Writes
|
||||
|
||||
1. Merge the domain-specific frontend store change that stops calling `saveServerData` for the retired key.
|
||||
2. Keep AppData read fallback for one release while relation reads are verified.
|
||||
3. Deploy and run smoke tests.
|
||||
|
||||
```bash
|
||||
pnpm deploy:smoke -- --base-url http://localhost --expected-version "$APP_VERSION"
|
||||
```
|
||||
|
||||
## Remove Fallback
|
||||
|
||||
Remove AppData read fallback only after one successful release where:
|
||||
|
||||
- Domain writes went through relation APIs.
|
||||
- AppData row counts did not grow for the retired key.
|
||||
- V2.2 read paths and page workflows returned expected data.
|
||||
- No `AppData relation sync failed` logs appeared during the observation window.
|
||||
|
||||
## Archive
|
||||
|
||||
Export retired keys before any later table cleanup.
|
||||
|
||||
```bash
|
||||
docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "copy (select key, value, updated_at from app_data where key in ('dev-tasks','test-cases','bugs')) to stdout with csv header" > backups/postgres/appdata-retired-executions-20260708.csv
|
||||
```
|
||||
|
||||
## Rollback
|
||||
|
||||
- If relation writes fail but AppData still has fresh data, roll back to the previous app image that still writes AppData.
|
||||
- If AppData writes were already disabled and relation writes are bad, restore PostgreSQL from the backup made at the start of this runbook.
|
||||
- If only read fallback removal caused the issue, roll back app images first and leave the database unchanged.
|
||||
|
||||
## Data Risks
|
||||
|
||||
- Removing fallback too early can hide valid historical JSON rows that were never mapped into relation tables.
|
||||
- Re-enabling old AppData writes after relation writes have accepted new edits can overwrite newer relation state through compatibility sync.
|
||||
- AppData exports can contain business-sensitive text; store backup CSV files in the same restricted location as database dumps.
|
||||
|
||||
100
docs/runbooks/migration-rollback.md
Normal file
100
docs/runbooks/migration-rollback.md
Normal file
@@ -0,0 +1,100 @@
|
||||
# Migration Rollback Runbook
|
||||
|
||||
Use this when a production release, Prisma migration, or data migration causes failed smoke tests, missing data, bad query performance, or unsafe writes.
|
||||
|
||||
## First Response
|
||||
|
||||
1. Freeze new releases and ask product owners to pause bulk edits.
|
||||
2. Capture current state before changing anything.
|
||||
|
||||
```bash
|
||||
date -u
|
||||
git rev-parse HEAD
|
||||
docker compose --env-file .env.production -f docker-compose.prod.yml ps
|
||||
pnpm backup:postgres -- --env-file .env.production
|
||||
pnpm backup:server-data -- --env-file .env.production
|
||||
```
|
||||
|
||||
3. Run the read-only release smoke test.
|
||||
|
||||
```bash
|
||||
pnpm deploy:smoke -- --base-url http://localhost --expected-version "$APP_VERSION"
|
||||
```
|
||||
|
||||
4. Check the database and latest server logs.
|
||||
|
||||
```bash
|
||||
docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres pg_isready -U "$POSTGRES_USER" -d "$POSTGRES_DB"
|
||||
docker compose --env-file .env.production -f docker-compose.prod.yml logs --tail=200 server
|
||||
```
|
||||
|
||||
## Decision Points
|
||||
|
||||
- App code bad, database healthy: roll back `WEB_IMAGE` and `SERVER_IMAGE` to the previous commit image tags, then restart Compose.
|
||||
- Migration applied but only additive: roll back app images first; leave schema in place if old code remains compatible.
|
||||
- Migration changed or removed data: restore a fresh database from the last known-good backup into a temporary database, compare row counts, then decide whether to restore production.
|
||||
- AppData compatibility issue: keep production database intact, re-enable the previous app image that still reads the AppData fallback, and preserve the failing release backup for analysis.
|
||||
|
||||
## App Image Rollback
|
||||
|
||||
Set previous image tags in `.env.production`. Use the previous successful GitHub Actions run to identify the image tags.
|
||||
|
||||
```bash
|
||||
export PREVIOUS_WEB_IMAGE=ghcr.io/company/ftb-project-management/web:abc1234
|
||||
export PREVIOUS_SERVER_IMAGE=ghcr.io/company/ftb-project-management/server:abc1234
|
||||
sed -i "s|^WEB_IMAGE=.*|WEB_IMAGE=${PREVIOUS_WEB_IMAGE}|" .env.production
|
||||
sed -i "s|^SERVER_IMAGE=.*|SERVER_IMAGE=${PREVIOUS_SERVER_IMAGE}|" .env.production
|
||||
docker compose --env-file .env.production -f docker-compose.prod.yml pull web server
|
||||
docker compose --env-file .env.production -f docker-compose.prod.yml up -d --remove-orphans
|
||||
pnpm deploy:smoke -- --base-url http://localhost --expected-version abc1234
|
||||
```
|
||||
|
||||
## Fresh Database Restore
|
||||
|
||||
Never restore over production until the backup has been rehearsed into a temporary database.
|
||||
|
||||
```bash
|
||||
export BACKUP_FILE=backups/postgres/ftb_pm-postgres-20260708T120000Z.dump
|
||||
pnpm restore:postgres -- \
|
||||
--dry-run \
|
||||
--confirm-overwrite \
|
||||
--env-file .env.production \
|
||||
--input "$BACKUP_FILE" \
|
||||
--target-db ftb_pm_restore_check
|
||||
pnpm restore:postgres -- \
|
||||
--confirm-overwrite \
|
||||
--env-file .env.production \
|
||||
--input "$BACKUP_FILE" \
|
||||
--target-db ftb_pm_restore_check
|
||||
```
|
||||
|
||||
Compare critical row counts before touching production.
|
||||
|
||||
```bash
|
||||
docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres psql -U "$POSTGRES_USER" -d ftb_pm_restore_check -c "select 'products' as table_name, count(*) from products union all select 'requirements', count(*) from requirements union all select 'versions', count(*) from versions union all select 'dev_tasks', count(*) from dev_tasks union all select 'test_cases', count(*) from test_cases union all select 'bugs', count(*) from bugs;"
|
||||
```
|
||||
|
||||
If the temporary restore is healthy and production data is unsafe, restore production with explicit overwrite confirmation.
|
||||
|
||||
```bash
|
||||
pnpm restore:postgres -- \
|
||||
--confirm-overwrite \
|
||||
--env-file .env.production \
|
||||
--input "$BACKUP_FILE"
|
||||
docker compose --env-file .env.production -f docker-compose.prod.yml up -d --remove-orphans
|
||||
pnpm deploy:smoke -- --base-url http://localhost
|
||||
```
|
||||
|
||||
## Data Risks
|
||||
|
||||
- PostgreSQL restore is destructive for the target database because the script terminates connections, drops the target database, recreates it, and runs `pg_restore`.
|
||||
- Restoring PostgreSQL does not restore `server_data`; keep AI provider config backup files with the same incident bundle.
|
||||
- AppData and relation tables can diverge during compatibility windows. Before deleting or restoring, preserve both the failing production backup and the known-good backup.
|
||||
- If users continued editing during the incident, record the time window and decide whether those edits must be replayed manually after restore.
|
||||
|
||||
## Closeout
|
||||
|
||||
1. Save the failed release SHA, rollback SHA, backup file names, smoke output, and row-count evidence in the incident notes.
|
||||
2. Keep the failed backup until the next successful release has completed smoke tests and one business-day observation.
|
||||
3. Add a regression test or runbook correction before re-attempting the migration.
|
||||
|
||||
55
docs/runbooks/xiaobao-background-jobs.md
Normal file
55
docs/runbooks/xiaobao-background-jobs.md
Normal file
@@ -0,0 +1,55 @@
|
||||
# Xiaobao Background Jobs Runbook
|
||||
|
||||
Current V2.8 production monitoring supports Xiaobao staleness detection. The first production implementation is still page-triggered: opening `/xiaobao-warning` computes risk, saves snapshots, and lets V2.3 sync refresh summaries. A future scheduler must preserve the same idempotent data contract.
|
||||
|
||||
## Alert Triage
|
||||
|
||||
When `FtbXiaobaoSummaryStale` fires:
|
||||
|
||||
```bash
|
||||
docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "select version_id, dirty, updated_at, recomputed_at, risk_level, risk_score from xiaobao_risk_summaries where dirty = true or updated_at < now() - interval '6 hours' order by updated_at asc limit 20;"
|
||||
docker compose --env-file .env.production -f docker-compose.prod.yml logs --tail=200 server | grep -E "xiaobao|AppData relation sync failed|Slow Prisma query"
|
||||
```
|
||||
|
||||
Decision points:
|
||||
|
||||
- Rows are dirty after active version edits: ask a manager to open `/xiaobao-warning` once, then verify summaries refresh.
|
||||
- Rows remain dirty and server logs show sync failures: treat as AppData relation sync incident and follow `migration-rollback.md`.
|
||||
- Rows are stale but no product release is near: keep monitoring and schedule a manual refresh before the next release decision meeting.
|
||||
- Rows are stale for a release due today: refresh manually and have the release owner review the resulting risk explanation before ship/no-ship decision.
|
||||
|
||||
## Manual Refresh Path
|
||||
|
||||
1. Log in as a user with `xiaobao.warning:manage`.
|
||||
2. Open `/xiaobao-warning`.
|
||||
3. Wait until AI interpretation status is no longer generating for high-risk versions.
|
||||
4. Re-run the stale-summary query.
|
||||
5. Run the release smoke test.
|
||||
|
||||
```bash
|
||||
pnpm deploy:smoke -- --base-url http://localhost
|
||||
```
|
||||
|
||||
## Future Scheduler Rules
|
||||
|
||||
When a background job is introduced, it must:
|
||||
|
||||
- Read unfinished versions by relation-table scope, not by full AppData document scan.
|
||||
- Use one idempotency key per `versionId + riskSignature + snapshotDate`.
|
||||
- Write snapshots append-only and upsert summaries by `version_id`.
|
||||
- Mark failures with structured logs containing `xiaobao background job failed`.
|
||||
- Retry transient AI failures with backoff and keep rule-based risk output even when AI interpretation fails.
|
||||
- Never mutate Version, Requirement, DevTask, TestCase, Bug, or Member data.
|
||||
|
||||
## Monitoring Expectations
|
||||
|
||||
- `FtbXiaobaoSummaryStale` alerts on dirty or older-than-6-hour summaries.
|
||||
- `FtbJobFailureLogBurst` alerts when sync or future job failure log counters increase.
|
||||
- Grafana dashboard shows the stale summary count and matching server warning/error logs.
|
||||
|
||||
## Data Risks
|
||||
|
||||
- Recomputing Xiaobao summaries can change release risk badges and manager decisions; record manual refresh time in release notes.
|
||||
- AI interpretation cache is explanatory only. Do not restore or delete business entities to fix a bad explanation.
|
||||
- If stale summaries are caused by relation sync failure, refreshing the page can mask the symptom without fixing the underlying sync path. Preserve logs before restarting services.
|
||||
|
||||
Reference in New Issue
Block a user