The single-file stdlib server became the limit: no way to track what had been
done about an alert, no accounts, and a UI that had to be hand-rolled in
template strings. This restructures it into something deployable.
Backend (FastAPI)
- app/ holds config, database, auth, delivery and the routers; triagelib keeps
the triage engine unchanged, so the validated screening and runbook logic is
untouched.
- Cases persist per alert fingerprint with a status workflow (investigating,
customer contacted, escalated to Infra, waiting, remediated, resolved, won't
fix, false positive), an assignee, notes and an append-only history. An alert
that stops and re-fires lands back on the same case and counts as a reopen.
- Suppression rules move from a JSON file into the database.
Auth
- Signed session cookies over PBKDF2 local accounts, plus an OIDC flow ready for
Authentik: users are created on first login and admin follows a group claim.
Local login can be switched off entirely once SSO is live.
Zendesk and Jira
- Delivery is now implemented, behind three gates: the integration must be
configured, its feature flag on, and CX_FEATURE_SEND_ENABLED on. A demo
instance leaves the last off and cannot mail anyone. Both search before
creating, so re-diagnosing an alert updates one ticket rather than opening
several, and a rolling daily cap stops a loop mailing everybody.
Deployment
- Multi-stage Dockerfile builds the bundle and serves it from the API origin.
- docker-compose for local and single-host use; Gitea Actions runs the tests,
builds the image and renders deploy/k8s with envsubst.
Two fixes found while testing: assigning a case returned a null assignee, and
add_event could leave an already-loaded history collection stale.
Known gap: the engine reaches OpenStack via `docker exec <region>-osc`, which
does not work in a pod without the CX-Tools containers alongside it.
docs/DEPLOYMENT.md sets out the three ways to close that.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Read-only triage for the Infrahub error alerts. Pulls the Prometheus alert
queue, re-checks each alert's condition against live state to separate real
work from noise, diagnoses it using the CX runbooks, and drafts the customer
comms with contacts resolved from Infrahub.
Findings from validating against production:
- "Suspected Rogue VM" fires on spare GPU capacity, not rogue VMs: In_Use_Gpus
equals the physical count on 71 of 75 firing hosts, so the rule reduces to
"this host has a free GPU". Verified against OpenStack on 10 hosts.
- "Exists in Infrahub but does not exist in OpenStack" matches every VM because
openstack_nova_server_status returns no series; excluded as a rule defect.
- Prometheus activeAt is reset several times a day by dips in the Resources
metric, so alert ages are recovered from ALERTS history instead.
Takes ~2,650 firing alerts down to ~20 that need a decision.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
