7837a809ca
- 新增 Prometheus/Grafana/Loki/Promtail 监控 profile\n- 覆盖 DB、磁盘、慢 API、慢 Prisma、任务失败和小宝摘要 stale 告警\n- 补充 postgres-exporter 自定义查询、Dashboard、部署文档和校验\n\nCo-Authored-By: GPT-5 Codex <codex@openai.com>
44 lines
1.9 KiB
Markdown
44 lines
1.9 KiB
Markdown
# FTB Production Monitoring Baseline
|
|
|
|
This profile adds a deployable Prometheus/Grafana baseline for production operations. It is intentionally secret-free: no webhook URLs, API keys, SMTP passwords, or real alert receiver credentials are committed.
|
|
|
|
## Start
|
|
|
|
```bash
|
|
docker compose --env-file .env.production -f docker-compose.prod.yml --profile monitoring up -d
|
|
```
|
|
|
|
Default local ports:
|
|
|
|
- Prometheus: `http://localhost:9090`
|
|
- Grafana: `http://localhost:3002`
|
|
- Loki: internal only
|
|
|
|
Set `GRAFANA_ADMIN_USER` and `GRAFANA_ADMIN_PASSWORD` in `.env.production` before exposing Grafana beyond localhost. Keep real alert receivers in the server environment or an untracked Alertmanager file.
|
|
|
|
## Coverage
|
|
|
|
- DB availability: `pg_up` from postgres-exporter.
|
|
- Disk pressure: root filesystem availability from node-exporter.
|
|
- Slow API requests: Promtail turns `Slow API request` server logs into `ftb_slow_api_log_total`.
|
|
- Slow Prisma queries: Promtail turns `Slow Prisma query` server logs into `ftb_slow_prisma_log_total`.
|
|
- Job failures: Promtail turns `AppData relation sync failed` and AI call failure logs into `ftb_job_failure_log_total`.
|
|
- Xiaobao stale summaries: postgres-exporter custom query exposes `ftb_xiaobao_stale_summary_count` from `xiaobao_risk_summaries`.
|
|
|
|
## Alerts
|
|
|
|
Prometheus loads `prometheus/alert-rules.yml`. The rules evaluate locally and are visible in Prometheus/Grafana. To send notifications, add Alertmanager outside git or mount an environment-specific receiver file; do not commit webhook URLs or tokens.
|
|
|
|
Baseline alert names:
|
|
|
|
- `FtbPostgresDown`
|
|
- `FtbDiskPressure`
|
|
- `FtbSlowApiLogBurst`
|
|
- `FtbSlowPrismaLogBurst`
|
|
- `FtbJobFailureLogBurst`
|
|
- `FtbXiaobaoSummaryStale`
|
|
|
|
## Log Search
|
|
|
|
Promtail ships Docker logs to Loki with container labels. Grafana provisions both Prometheus and Loki data sources, so on-call checks can move from a firing alert to matching server logs without SSHing into the host.
|