ftb-project-management/docs/runbooks/xiaobao-background-jobs.md

Xiaobao Background Jobs Runbook

Current V2.8 production monitoring supports Xiaobao staleness detection. The first production implementation is still page-triggered: opening /xiaobao-warning computes risk, saves snapshots, and lets V2.3 sync refresh summaries. A future scheduler must preserve the same idempotent data contract.

Alert Triage

When FtbXiaobaoSummaryStale fires:

docker compose --env-file .env.production -f docker-compose.prod.yml exec postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "select version_id, dirty, updated_at, recomputed_at, risk_level, risk_score from xiaobao_risk_summaries where dirty = true or updated_at < now() - interval '6 hours' order by updated_at asc limit 20;"
docker compose --env-file .env.production -f docker-compose.prod.yml logs --tail=200 server | grep -E "xiaobao|AppData relation sync failed|Slow Prisma query"

Decision points:

• Rows are dirty after active version edits: ask a manager to open /xiaobao-warning once, then verify summaries refresh.
• Rows remain dirty and server logs show sync failures: treat as AppData relation sync incident and follow migration-rollback.md.
• Rows are stale but no product release is near: keep monitoring and schedule a manual refresh before the next release decision meeting.
• Rows are stale for a release due today: refresh manually and have the release owner review the resulting risk explanation before ship/no-ship decision.

Manual Refresh Path

1. Log in as a user with xiaobao.warning:manage.
2. Open /xiaobao-warning.
3. Wait until AI interpretation status is no longer generating for high-risk versions.
4. Re-run the stale-summary query.
5. Run the release smoke test.

pnpm deploy:smoke -- --base-url http://localhost

Future Scheduler Rules

When a background job is introduced, it must:

• Read unfinished versions by relation-table scope, not by full AppData document scan.
• Use one idempotency key per versionId + riskSignature + snapshotDate.
• Write snapshots append-only and upsert summaries by version_id.
• Mark failures with structured logs containing xiaobao background job failed.
• Retry transient AI failures with backoff and keep rule-based risk output even when AI interpretation fails.
• Never mutate Version, Requirement, DevTask, TestCase, Bug, or Member data.

Monitoring Expectations

• FtbXiaobaoSummaryStale alerts on dirty or older-than-6-hour summaries.
• FtbJobFailureLogBurst alerts when sync or future job failure log counters increase.
• Grafana dashboard shows the stale summary count and matching server warning/error logs.

Data Risks

• Recomputing Xiaobao summaries can change release risk badges and manager decisions; record manual refresh time in release notes.
• AI interpretation cache is explanatory only. Do not restore or delete business entities to fix a bad explanation.
• If stale summaries are caused by relation sync failure, refreshing the page can mask the symptom without fixing the underlying sync path. Preserve logs before restarting services.