Parham Monfared
1262690276
The single-file stdlib server became the limit: no way to track what had been done about an alert, no accounts, and a UI that had to be hand-rolled in template strings. This restructures it into something deployable. Backend (FastAPI) - app/ holds config, database, auth, delivery and the routers; triagelib keeps the triage engine unchanged, so the validated screening and runbook logic is untouched. - Cases persist per alert fingerprint with a status workflow (investigating, customer contacted, escalated to Infra, waiting, remediated, resolved, won't fix, false positive), an assignee, notes and an append-only history. An alert that stops and re-fires lands back on the same case and counts as a reopen. - Suppression rules move from a JSON file into the database. Auth - Signed session cookies over PBKDF2 local accounts, plus an OIDC flow ready for Authentik: users are created on first login and admin follows a group claim. Local login can be switched off entirely once SSO is live. Zendesk and Jira - Delivery is now implemented, behind three gates: the integration must be configured, its feature flag on, and CX_FEATURE_SEND_ENABLED on. A demo instance leaves the last off and cannot mail anyone. Both search before creating, so re-diagnosing an alert updates one ticket rather than opening several, and a rolling daily cap stops a loop mailing everybody. Deployment - Multi-stage Dockerfile builds the bundle and serves it from the API origin. - docker-compose for local and single-host use; Gitea Actions runs the tests, builds the image and renders deploy/k8s with envsubst. Two fixes found while testing: assigning a case returned a null assignee, and add_event could leave an already-loaded history collection stale. Known gap: the engine reaches OpenStack via `docker exec <region>-osc`, which does not work in a pod without the CX-Tools containers alongside it. docs/DEPLOYMENT.md sets out the three ways to close that. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2.6 KiB
2.6 KiB
CX Triage
Turns the Infrahub error-alert firehose into a short list of things that actually need doing — then helps you do them.
Python/FastAPI backend, React frontend, PostgreSQL for case state.
What it does
- Pulls the alert queue from Prometheus.
- Screens every alert by re-checking its condition against live state, so noise and already-resolved alerts drop out.
- Diagnoses what is left using the CX runbooks, reconciling Infrahub against OpenStack through the CX-Tools collectors.
- Drafts the customer email with contacts resolved from Infrahub, and the Jira escalation with the evidence attached.
- Tracks each case — who owns it, what was done, what was sent.
On live data this takes roughly 2,650 firing alerts down to ~20 that need a decision.
Findings that shaped it
Validated against production, not assumed:
Suspected Rogue VMis measuring spare capacity.In_Use_Gpusequals the physical GPU count on 71 of 75 firing hosts, so the rule reduces to "this host has a free GPU". Checked against OpenStack on 10 hosts: Infrahub and OpenStack agreed exactly on all of them. Those alerts are flagged as a rule defect.Exists in Infrahub but does not exist in OpenStackmatches every VM, becauseopenstack_nova_server_statusreturns no series. Excluded outright. The same gap meansSuspected Orphan VMcannot fire at all.- Prometheus alert ages are unreliable here. The
Resourcesmetric drops most of its series several times a day; every alert alive at the time resolves and re-fires, resettingactiveAt. Ages are recovered fromALERTShistory instead.
Read-only by design
The app queries and advises. It never deletes, shelves, or edits a VM — those stay copy-a-command. The only thing it can send is a Zendesk ticket or a Jira issue, behind three gates and a confirm step. See docs/INTEGRATIONS.md.
Run it
cp .env.example .env
docker compose up --build
More: docs/DEPLOYMENT.md · docs/LINKAGE.md · docs/PLAN.md
Layout
backend/
app/ FastAPI: config, db, models, auth, delivery, routers
triagelib/ the triage engine (screening, runbooks, comms, linkage)
tests/ pure-logic tests — no network, no CX-Tools
frontend/ React + TypeScript + Vite
deploy/k8s/ manifests, rendered by the pipeline
.gitea/workflows/ test → build → deploy
Tests
cd backend && python tests/test_runbooks.py && python tests/test_screening.py