feat(ops): 添加生产监控告警基线

- 新增 Prometheus/Grafana/Loki/Promtail 监控 profile\n- 覆盖 DB、磁盘、慢 API、慢 Prisma、任务失败和小宝摘要 stale 告警\n- 补充 postgres-exporter 自定义查询、Dashboard、部署文档和校验\n\nCo-Authored-By: GPT-5 Codex <codex@openai.com>
This commit is contained in:
2026-07-08 16:21:07 +08:00
parent eef09d5af4
commit 7837a809ca
15 changed files with 587 additions and 1 deletions

View File

@@ -0,0 +1,43 @@
# FTB Production Monitoring Baseline
This profile adds a deployable Prometheus/Grafana baseline for production operations. It is intentionally secret-free: no webhook URLs, API keys, SMTP passwords, or real alert receiver credentials are committed.
## Start
```bash
docker compose --env-file .env.production -f docker-compose.prod.yml --profile monitoring up -d
```
Default local ports:
- Prometheus: `http://localhost:9090`
- Grafana: `http://localhost:3002`
- Loki: internal only
Set `GRAFANA_ADMIN_USER` and `GRAFANA_ADMIN_PASSWORD` in `.env.production` before exposing Grafana beyond localhost. Keep real alert receivers in the server environment or an untracked Alertmanager file.
## Coverage
- DB availability: `pg_up` from postgres-exporter.
- Disk pressure: root filesystem availability from node-exporter.
- Slow API requests: Promtail turns `Slow API request` server logs into `ftb_slow_api_log_total`.
- Slow Prisma queries: Promtail turns `Slow Prisma query` server logs into `ftb_slow_prisma_log_total`.
- Job failures: Promtail turns `AppData relation sync failed` and AI call failure logs into `ftb_job_failure_log_total`.
- Xiaobao stale summaries: postgres-exporter custom query exposes `ftb_xiaobao_stale_summary_count` from `xiaobao_risk_summaries`.
## Alerts
Prometheus loads `prometheus/alert-rules.yml`. The rules evaluate locally and are visible in Prometheus/Grafana. To send notifications, add Alertmanager outside git or mount an environment-specific receiver file; do not commit webhook URLs or tokens.
Baseline alert names:
- `FtbPostgresDown`
- `FtbDiskPressure`
- `FtbSlowApiLogBurst`
- `FtbSlowPrismaLogBurst`
- `FtbJobFailureLogBurst`
- `FtbXiaobaoSummaryStale`
## Log Search
Promtail ships Docker logs to Loki with container labels. Grafana provisions both Prometheus and Loki data sources, so on-call checks can move from a firing alert to matching server logs without SSHing into the host.