Parham Monfared
1262690276
The single-file stdlib server became the limit: no way to track what had been done about an alert, no accounts, and a UI that had to be hand-rolled in template strings. This restructures it into something deployable. Backend (FastAPI) - app/ holds config, database, auth, delivery and the routers; triagelib keeps the triage engine unchanged, so the validated screening and runbook logic is untouched. - Cases persist per alert fingerprint with a status workflow (investigating, customer contacted, escalated to Infra, waiting, remediated, resolved, won't fix, false positive), an assignee, notes and an append-only history. An alert that stops and re-fires lands back on the same case and counts as a reopen. - Suppression rules move from a JSON file into the database. Auth - Signed session cookies over PBKDF2 local accounts, plus an OIDC flow ready for Authentik: users are created on first login and admin follows a group claim. Local login can be switched off entirely once SSO is live. Zendesk and Jira - Delivery is now implemented, behind three gates: the integration must be configured, its feature flag on, and CX_FEATURE_SEND_ENABLED on. A demo instance leaves the last off and cannot mail anyone. Both search before creating, so re-diagnosing an alert updates one ticket rather than opening several, and a rolling daily cap stops a loop mailing everybody. Deployment - Multi-stage Dockerfile builds the bundle and serves it from the API origin. - docker-compose for local and single-host use; Gitea Actions runs the tests, builds the image and renders deploy/k8s with envsubst. Two fixes found while testing: assigning a case returned a null assignee, and add_event could leave an already-loaded history collection stale. Known gap: the engine reaches OpenStack via `docker exec <region>-osc`, which does not work in a pod without the CX-Tools containers alongside it. docs/DEPLOYMENT.md sets out the three ways to close that. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
72 lines
2.6 KiB
Markdown
72 lines
2.6 KiB
Markdown
# CX Triage
|
|
|
|
Turns the Infrahub error-alert firehose into a short list of things that
|
|
actually need doing — then helps you do them.
|
|
|
|
Python/FastAPI backend, React frontend, PostgreSQL for case state.
|
|
|
|
## What it does
|
|
|
|
1. **Pulls** the alert queue from Prometheus.
|
|
2. **Screens** every alert by re-checking its condition against live state, so
|
|
noise and already-resolved alerts drop out.
|
|
3. **Diagnoses** what is left using the CX runbooks, reconciling Infrahub
|
|
against OpenStack through the CX-Tools collectors.
|
|
4. **Drafts** the customer email with contacts resolved from Infrahub, and the
|
|
Jira escalation with the evidence attached.
|
|
5. **Tracks** each case — who owns it, what was done, what was sent.
|
|
|
|
On live data this takes roughly **2,650 firing alerts down to ~20** that need a
|
|
decision.
|
|
|
|
## Findings that shaped it
|
|
|
|
Validated against production, not assumed:
|
|
|
|
- **`Suspected Rogue VM` is measuring spare capacity.** `In_Use_Gpus` equals the
|
|
physical GPU count on 71 of 75 firing hosts, so the rule reduces to "this host
|
|
has a free GPU". Checked against OpenStack on 10 hosts: Infrahub and OpenStack
|
|
agreed exactly on all of them. Those alerts are flagged as a rule defect.
|
|
- **`Exists in Infrahub but does not exist in OpenStack` matches every VM**,
|
|
because `openstack_nova_server_status` returns no series. Excluded outright.
|
|
The same gap means `Suspected Orphan VM` cannot fire at all.
|
|
- **Prometheus alert ages are unreliable here.** The `Resources` metric drops
|
|
most of its series several times a day; every alert alive at the time resolves
|
|
and re-fires, resetting `activeAt`. Ages are recovered from `ALERTS` history
|
|
instead.
|
|
|
|
## Read-only by design
|
|
|
|
The app queries and advises. It never deletes, shelves, or edits a VM — those
|
|
stay copy-a-command. The only thing it can send is a Zendesk ticket or a Jira
|
|
issue, behind three gates and a confirm step. See
|
|
[docs/INTEGRATIONS.md](docs/INTEGRATIONS.md).
|
|
|
|
## Run it
|
|
|
|
```bash
|
|
cp .env.example .env
|
|
docker compose up --build
|
|
```
|
|
|
|
More: [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) ·
|
|
[docs/LINKAGE.md](docs/LINKAGE.md) · [docs/PLAN.md](docs/PLAN.md)
|
|
|
|
## Layout
|
|
|
|
```
|
|
backend/
|
|
app/ FastAPI: config, db, models, auth, delivery, routers
|
|
triagelib/ the triage engine (screening, runbooks, comms, linkage)
|
|
tests/ pure-logic tests — no network, no CX-Tools
|
|
frontend/ React + TypeScript + Vite
|
|
deploy/k8s/ manifests, rendered by the pipeline
|
|
.gitea/workflows/ test → build → deploy
|
|
```
|
|
|
|
## Tests
|
|
|
|
```bash
|
|
cd backend && python tests/test_runbooks.py && python tests/test_screening.py
|
|
```
|