feat(ops): 添加生产监控告警基线
- 新增 Prometheus/Grafana/Loki/Promtail 监控 profile\n- 覆盖 DB、磁盘、慢 API、慢 Prisma、任务失败和小宝摘要 stale 告警\n- 补充 postgres-exporter 自定义查询、Dashboard、部署文档和校验\n\nCo-Authored-By: GPT-5 Codex <codex@openai.com>
This commit is contained in:
43
deploy/monitoring/README.md
Normal file
43
deploy/monitoring/README.md
Normal file
@@ -0,0 +1,43 @@
|
||||
# FTB Production Monitoring Baseline
|
||||
|
||||
This profile adds a deployable Prometheus/Grafana baseline for production operations. It is intentionally secret-free: no webhook URLs, API keys, SMTP passwords, or real alert receiver credentials are committed.
|
||||
|
||||
## Start
|
||||
|
||||
```bash
|
||||
docker compose --env-file .env.production -f docker-compose.prod.yml --profile monitoring up -d
|
||||
```
|
||||
|
||||
Default local ports:
|
||||
|
||||
- Prometheus: `http://localhost:9090`
|
||||
- Grafana: `http://localhost:3002`
|
||||
- Loki: internal only
|
||||
|
||||
Set `GRAFANA_ADMIN_USER` and `GRAFANA_ADMIN_PASSWORD` in `.env.production` before exposing Grafana beyond localhost. Keep real alert receivers in the server environment or an untracked Alertmanager file.
|
||||
|
||||
## Coverage
|
||||
|
||||
- DB availability: `pg_up` from postgres-exporter.
|
||||
- Disk pressure: root filesystem availability from node-exporter.
|
||||
- Slow API requests: Promtail turns `Slow API request` server logs into `ftb_slow_api_log_total`.
|
||||
- Slow Prisma queries: Promtail turns `Slow Prisma query` server logs into `ftb_slow_prisma_log_total`.
|
||||
- Job failures: Promtail turns `AppData relation sync failed` and AI call failure logs into `ftb_job_failure_log_total`.
|
||||
- Xiaobao stale summaries: postgres-exporter custom query exposes `ftb_xiaobao_stale_summary_count` from `xiaobao_risk_summaries`.
|
||||
|
||||
## Alerts
|
||||
|
||||
Prometheus loads `prometheus/alert-rules.yml`. The rules evaluate locally and are visible in Prometheus/Grafana. To send notifications, add Alertmanager outside git or mount an environment-specific receiver file; do not commit webhook URLs or tokens.
|
||||
|
||||
Baseline alert names:
|
||||
|
||||
- `FtbPostgresDown`
|
||||
- `FtbDiskPressure`
|
||||
- `FtbSlowApiLogBurst`
|
||||
- `FtbSlowPrismaLogBurst`
|
||||
- `FtbJobFailureLogBurst`
|
||||
- `FtbXiaobaoSummaryStale`
|
||||
|
||||
## Log Search
|
||||
|
||||
Promtail ships Docker logs to Loki with container labels. Grafana provisions both Prometheus and Loki data sources, so on-call checks can move from a firing alert to matching server logs without SSHing into the host.
|
||||
Reference in New Issue
Block a user