parham.monfared/cx-ui
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Split into a FastAPI backend and a React frontend, add case state and SSO
Some checks failed
build-and-deploy / test (push) Has been cancelled
Details
build-and-deploy / image (push) Has been cancelled
Details
build-and-deploy / deploy (push) Has been cancelled
Details

Browse Source 
The single-file stdlib server became the limit: no way to track what had been
done about an alert, no accounts, and a UI that had to be hand-rolled in
template strings. This restructures it into something deployable.

Backend (FastAPI)
- app/ holds config, database, auth, delivery and the routers; triagelib keeps
  the triage engine unchanged, so the validated screening and runbook logic is
  untouched.
- Cases persist per alert fingerprint with a status workflow (investigating,
  customer contacted, escalated to Infra, waiting, remediated, resolved, won't
  fix, false positive), an assignee, notes and an append-only history. An alert
  that stops and re-fires lands back on the same case and counts as a reopen.
- Suppression rules move from a JSON file into the database.

Auth
- Signed session cookies over PBKDF2 local accounts, plus an OIDC flow ready for
  Authentik: users are created on first login and admin follows a group claim.
  Local login can be switched off entirely once SSO is live.

Zendesk and Jira
- Delivery is now implemented, behind three gates: the integration must be
  configured, its feature flag on, and CX_FEATURE_SEND_ENABLED on. A demo
  instance leaves the last off and cannot mail anyone. Both search before
  creating, so re-diagnosing an alert updates one ticket rather than opening
  several, and a rolling daily cap stops a loop mailing everybody.

Deployment
- Multi-stage Dockerfile builds the bundle and serves it from the API origin.
- docker-compose for local and single-host use; Gitea Actions runs the tests,
  builds the image and renders deploy/k8s with envsubst.

Two fixes found while testing: assigning a case returned a null assignee, and
add_event could leave an already-loaded history collection stale.

Known gap: the engine reaches OpenStack via `docker exec <region>-osc`, which
does not work in a pod without the CX-Tools containers alongside it.
docs/DEPLOYMENT.md sets out the three ways to close that.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Parham Monfared
2026-08-06 07:11:28 +01:00
parent a039e0b5fd
commit 1262690276
68 changed files with 3839 additions and 2223 deletions
Download Patch File Download Diff File Expand all files Collapse all files

263
README.md
View File

@@ -1,238 +1,71 @@
# CX Triage
A small local webapp that takes the Infrahub error alerts out of Prometheus,
diagnoses each one using the **unmodified** CX-Tools (`vmc`) collectors, tells you
what the runbook says to do next, and — when the next step is contacting the
customer — shows the approved wording alongside the customer's contact details.
Turns the Infrahub error-alert firehose into a short list of things that
actually need doing — then helps you do them.
**It is read-only.** It queries Infrahub, OpenStack, InfraInsight and Prometheus.
It never changes platform state, never deletes or shelves anything, and never
sends a message. Every action it identifies is presented for a human to perform.
Python/FastAPI backend, React frontend, PostgreSQL for case state.
## Separating noise from real work
## What it does
Thousands of alerts fire; only a handful are work. Before anything is shown, each
alert's condition is **re-checked against current state**, and the verdict is
displayed with its reason:
1. **Pulls** the alert queue from Prometheus.
2. **Screens** every alert by re-checking its condition against live state, so
noise and already-resolved alerts drop out.
3. **Diagnoses** what is left using the CX runbooks, reconciling Infrahub
against OpenStack through the CX-Tools collectors.
4. **Drafts** the customer email with contacts resolved from Infrahub, and the
Jira escalation with the evidence attached.
5. **Tracks** each case — who owns it, what was done, what was sent.
| Verdict | Meaning | In the queue? |
|---|---|---|
| **needs action** | The condition still holds | yes |
| **needs action (unverified)** | Couldn't be re-checked — never hidden on a guess | yes |
| **already resolved** | Infrahub has moved on / the IP is no longer duplicated / the GPU gap has closed | hidden |
| **not yet firing** | Prometheus still has it pending | hidden |
| **chronic** | Still true, but firing over 3 days — already ticketed, not new work | hidden |
| **low impact** | Still true, but owned by an internal org or a platform-owned node | hidden |
On live data this takes roughly **2,650 firing alerts down to ~20** that need a
decision.
Screening only ever demotes an alert on **positive evidence**; anything it can't
settle stays in the queue. Hidden alerts are one checkbox away, and any of them
can be force-diagnosed with **Diagnose anyway**.
## Findings that shaped it
### Alert ages are recovered, not taken from Prometheus
Validated against production, not assumed:
Prometheus' own `activeAt` is unreliable here. The Infrahub `Resources` metric
drops most of its series for ~5 minutes several times a day (4 dips in the last
24h observed; one took it from ~4,370 series to 1,359). Every alert alive during
a dip resolves and re-fires, so `activeAt` resets on all of them at once — which
is why the Prometheus UI shows dozens of unrelated alerts with the *same* age.
- **`Suspected Rogue VM` is measuring spare capacity.** `In_Use_Gpus` equals the
physical GPU count on 71 of 75 firing hosts, so the rule reduces to "this host
has a free GPU". Checked against OpenStack on 10 hosts: Infrahub and OpenStack
agreed exactly on all of them. Those alerts are flagged as a rule defect.
- **`Exists in Infrahub but does not exist in OpenStack` matches every VM**,
because `openstack_nova_server_status` returns no series. Excluded outright.
The same gap means `Suspected Orphan VM` cannot fire at all.
- **Prometheus alert ages are unreliable here.** The `Resources` metric drops
most of its series several times a day; every alert alive at the time resolves
and re-fires, resetting `activeAt`. Ages are recovered from `ALERTS` history
instead.
So the app walks the `ALERTS` series backwards over 7 days instead, bridging gaps
under 45 minutes, and reports how long each condition has **actually** held. In
practice this is the difference between "40 alerts all 7h old" and "11 that are
genuinely new, 33 that have been true for days". Both numbers are shown: the
recovered duration, with Prometheus' value in a tooltip when they disagree.
## Read-only by design
The app detects these dips and warns about them, since they also mean any alert
with a long `for:` may never reach firing state.
The app queries and advises. It never deletes, shelves, or edits a VM — those
stay copy-a-command. The only thing it can send is a Zendesk ticket or a Jira
issue, behind three gates and a confirm step. See
[docs/INTEGRATIONS.md](docs/INTEGRATIONS.md).
The re-check is cheap on purpose: it reads the same Prometheus series the rules
are built from — `Resources`, `In_Use_Gpus`, `Total_Gpus` — in one bulk snapshot
for the entire queue, rather than an Infrahub and OpenStack call per alert. Only
alerts you actually open cost a CX-Tools query.
On live data this takes **~2,670 firing alerts down to ~11** that need a decision.
Caches are warmed at startup (~20s, mostly the 7-day history read), so page loads
are instant afterwards. They refresh on a 30s/60s/5min cadence.
**Excluded outright:** `Exists in Infrahub but does not exist in OpenStack`. It is
built as `Resources unless on(openstack_id) openstack_nova_server_status`, and
that second metric is currently returning **zero series** — so nothing gets
excluded by the `unless` and every Infrahub VM alerts. It is a broken exporter,
not a queue of work. The app detects this class of failure and shows a banner,
because the same gap also means `Suspected Orphan VM` cannot fire at all.
## Two tabs
- **CX runbooks** — the alerts below, grouped into collapsible sections in
working order (rogue VMs, duplicate IPs, total GPUs, hibernating, creating,
shutoff, deleting, error), each showing how long it has been firing, **newest
first** so long-running alerts sink to the bottom.
- **Infrastructure** — everything else, so it stays out of the triage queue:
node-exporter host alerts in their own section, then Ceph, MySQL, Galera,
OpenStack services, blackbox. Listed and counted, not diagnosed. Routing is
keyed off the **rule file**, not the alert name, because two different rule
files both use the group name "Imported Rules".
## What it covers
One runbook per alert type, from *Infrahub Errors Remediation*:
| Alert | Priority | What the app works out for you |
|---|---|---|
| Instance in ERROR state | LOW–HIGH | Matches the fault against the runbook fault table; decides whether the VM was ever ACTIVE (which changes both the urgency and the comms template); for the NUMA/PCI fault it sums GPUs on the host to check whether the host is full before you escalate |
| Instance in DELETING state | LOW | Confirms the delete request in Infrahub events, and whether the OpenStack server is still there or already gone |
| Instance in SHUTOFF state | LOW | Confirms SHUTOFF and drafts the billing-awareness note |
| Instance in HIBERNATING state | HIGH | Runs the host signals (Nova state/status, disabled reason, OVS liveness) and escalates when they're bad |
| Instance in CREATING state | MEDIUM | Determines whether the VM ever got an OpenStack ID |
| Instance in RESTORING state | HIGH | Host signals plus the most recent *failed* OpenStack event to escalate |
| Instance in REBOOTING state | HIGH | Confirms `InstanceRebootRequest` and the expected `HARD_REBOOT` state |
| Instance in BUILD state | MEDIUM | Distinguishes "large flavor, still transient" from "stuck, escalate" |
| Suspected Rogue VM | HIGH | Quantifies the per-host GPU accounting gap the rule actually fires on, then reconciles every instance on the host and maps each mismatch to its row in the Mismatch Remediation table; tempest instances are ignored |
| Duplicated IPs | HIGH | Classifies each claimant of the IP as Scenario #1 / #2 / rightful owner, and pulls the cross-environment claimant list from `Resources{floating_ip=...}` |
| Problem with Total GPUs | HIGH | Lists which customers are on the affected host |
| Suspected Orphan VM | HIGH | Same host reconciliation as Rogue VM (cannot currently fire — see the exporter note above) |
| Openstack status=X / Infrahub status!=X | HIGH | Single-VM mismatch, taken through the same remediation table |
### A note on Suspected Rogue VM
The rule is not a status comparison — it is
`sum by(instance)(In_Use_Gpus) - sum by(instance)(Resources{status=~"ACTIVE|SHUTOFF|PRE_ACTIVE"}) >= 1`,
a **per-host GPU accounting gap**. Two different faults produce that gap:
1. instances running on the host that Infrahub has no record of (a true rogue VM), or
2. Infrahub VMs that are ACTIVE but have **no host recorded**, so they are never
counted against the host that is actually running them.
Prometheus cannot tell these apart, so the app states both and lets the
per-instance host reconciliation settle it — a genuine rogue VM shows up as
`Infrahub Missing`. It also reports how many unattributed ACTIVE VMs exist
platform-wide, because that number alone can be large enough to explain the gaps
without any rogue VM existing.
## Requirements
Whatever `vmc` already needs, plus nothing:
- The CX-Tools checkout (`cxlib/` + `vmc`), unmodified
- Python 3 (standard library only — no pip install)
- Docker with the `ca1-osc` / `ca2-osc` / `us1-osc` / `no1-osc` containers running
- A signed-in 1Password CLI session
## Run
## Run it
```bash
op signin
cp .env.example .env
docker compose up --build
```
```bash
./cx-triage
```
It opens <http://127.0.0.1:8765>. Bound to localhost only.
```bash
./cx-triage --check
```
Runs preflight (CX-Tools located, credentials loaded, containers up, Prometheus
reachable) and exits.
Useful flags: `--port`, `--prometheus <url>`, `--no-open`.
If CX-Tools isn't found automatically, point at it:
```bash
CX_TOOLS_PATH=~/scripts/CX-Tools ./cx-triage
```
## How it reaches things
- **CX-Tools** is imported as a library. `cxbridge.py` calls `collect_vm`,
`collect_host` and the query helpers — the same code paths as `vmc --json` —
and guards every OpenStack subcommand against a read-only allowlist, so a bug
here cannot mutate an instance.
- **Prometheus** at `10.11.254.250:9090` is on the internal network, which the
laptop has no route to (10.11.* leaves via the default gateway). So queries are
relayed `docker exec ca1-osc curl ...` — the same trick CX-Tools uses for
OpenStack. A direct HTTP transport is tried first, so this still works from a
host that does have a route. Override the relay with `CX_PROMETHEUS_RELAY`.
## Working an alert
1. Pick an alert from the queue. Or paste an `ALERTS{...}` line or a Prometheus
graph URL into the box.
2. **What the platforms say** — the reconciled Infrahub/OpenStack/InfraInsight
facts, with mismatches called out in red.
3. **Next steps** — the remaining runbook steps, each tagged with its owner (CX /
Infrastructure team / DevOps). Steps the app has already verified are ticked
off, so you can see what's left rather than re-deriving it.
4. **Suggested customer comms** — only when the runbook calls for it. Verbatim
approved wording with the instance name (and floating IP) substituted, above
the organization and owner contacts CX-Tools resolved. Copy it and send it
from HubSpot.
5. **Evidence** — the raw alert labels and the raw CX-Tools output, for pasting
into a Slack thread or a Jira ticket.
## Caveats
These are real limits, not bugs:
- **Production Infrahub only.** CX-Tools queries production. When a Duplicated
IPs or Rogue VM alert points at a PreProd/Staging record, the app says so and
tells you to check the other environments — it can't query them. The
Prometheus `Resources` series does span environments, which is why the
Duplicated IPs view uses it.
- **No InfraInsight SQL.** The DELETING runbook identifies the requesting *user*
via a SQL query. The app confirms the delete request from Infrahub events but
cannot name the requester, and says so when it matters.
- **Host Health Checks is partial.** The app reports the host signals CX-Tools
exposes (Nova state/status, disabled reason, OVS liveness/heartbeat, uptime,
aggregates). The rest of that guide — disk, dmesg, GPU checks — is still
manual, and the app says which part it did.
- **Fault table coverage.** ERROR faults outside the runbook's table produce an
explicit "not in the table, escalate to a peer" verdict rather than a guess.
- **Chronic/low-impact thresholds are judgement calls**, not runbook rules:
3 days for chronic (`CHRONIC_DAYS` in `screening.py`), and "internal" means an
`@nexgencloud.com` owner. Adjust to taste.
- **Recovered ages are bounded by a 7-day window** (`TrueAgeIndex.WINDOW_DAYS`).
Anything older shows as `7d+`.
- **"Chronic" does not mean "ignore".** It means the condition has been true for
days, so it is not *new* work. Several ERROR alerts are 6–7 days old; if those
have not actually been ticketed, they are a backlog, not noise.
More: [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) ·
[docs/LINKAGE.md](docs/LINKAGE.md) · [docs/PLAN.md](docs/PLAN.md)
## Layout
```text
cx-triage entry point (preflight, then serve)
triagelib/
cxbridge.py read-only adapter over cxlib
prometheus.py alert source, rule index, bulk state snapshot, relay
alerts.py normalization, classification, exclusions, grouping
screening.py noise-vs-real verdicts
runbooks.py the decision engine
comms.py customer comms templates, verbatim from Confluence
server.py HTTP API + background triage jobs
ui.py the single-page UI
tests/test_runbooks.py runbook decisions against fixture payloads
tests/test_screening.py screening, exclusion, tab routing, ordering
```
backend/
app/ FastAPI: config, db, models, auth, delivery, routers
triagelib/ the triage engine (screening, runbooks, comms, linkage)
tests/ pure-logic tests — no network, no CX-Tools
frontend/ React + TypeScript + Vite
deploy/k8s/ manifests, rendered by the pipeline
.gitea/workflows/ test → build → deploy
```
## Tests
```bash
python3 tests/test_runbooks.py && python3 tests/test_screening.py
cd backend && python tests/test_runbooks.py && python tests/test_screening.py
```
## Updating a runbook
The decision logic is meant to be edited by whoever owns the runbook:
- Fault table → `FAULT_TABLE` in `runbooks.py`
- Rogue VM mismatch table → `MISMATCH_TABLE` in `runbooks.py`
- Priority / ETTR → `KIND_META` in `alerts.py`
- Queue order → `FOCUS_ORDER` in `alerts.py`
- Alerts to suppress → `EXCLUDED_ALERTNAMES` in `alerts.py`
- Noise rules → `screening.py` (`CHRONIC_DAYS`, `_KIND_SCREENS`)
- Customer wording → `_TEMPLATES` in `comms.py`
If a wording change lands in Confluence, change it in `comms.py` and nowhere
else.