Parham Monfared
1262690276
The single-file stdlib server became the limit: no way to track what had been done about an alert, no accounts, and a UI that had to be hand-rolled in template strings. This restructures it into something deployable. Backend (FastAPI) - app/ holds config, database, auth, delivery and the routers; triagelib keeps the triage engine unchanged, so the validated screening and runbook logic is untouched. - Cases persist per alert fingerprint with a status workflow (investigating, customer contacted, escalated to Infra, waiting, remediated, resolved, won't fix, false positive), an assignee, notes and an append-only history. An alert that stops and re-fires lands back on the same case and counts as a reopen. - Suppression rules move from a JSON file into the database. Auth - Signed session cookies over PBKDF2 local accounts, plus an OIDC flow ready for Authentik: users are created on first login and admin follows a group claim. Local login can be switched off entirely once SSO is live. Zendesk and Jira - Delivery is now implemented, behind three gates: the integration must be configured, its feature flag on, and CX_FEATURE_SEND_ENABLED on. A demo instance leaves the last off and cannot mail anyone. Both search before creating, so re-diagnosing an alert updates one ticket rather than opening several, and a rolling daily cap stops a loop mailing everybody. Deployment - Multi-stage Dockerfile builds the bundle and serves it from the API origin. - docker-compose for local and single-host use; Gitea Actions runs the tests, builds the image and renders deploy/k8s with envsubst. Two fixes found while testing: assigning a case returned a null assignee, and add_event could leave an already-loaded history collection stale. Known gap: the engine reaches OpenStack via `docker exec <region>-osc`, which does not work in a pod without the CX-Tools containers alongside it. docs/DEPLOYMENT.md sets out the three ways to close that. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
131 lines
6.5 KiB
Markdown
131 lines
6.5 KiB
Markdown
# From "here's what to do" to "do it"
|
||
|
||
## 1. Validation against live data
|
||
|
||
Every recommendation was checked against OpenStack directly, not against the
|
||
app's own view. Results:
|
||
|
||
| Case | App said | Ground truth | Verdict |
|
||
|---|---|---|---|
|
||
| **Rogue VM `CA1-ESC8-057`** | "No mismatch — go check InfraInsight for HIBERNATED VMs carrying a host" | OpenStack 7 GPUs across 5 VMs; Infrahub 7 GPUs across the same 5 VMs. Perfect agreement. No hibernated VMs on the host at all. | **Wrong** |
|
||
| **Rogue VM, 10 hosts sampled** | 50 alerts "needs action" | OpenStack and Infrahub agreed **exactly** on all 10 | **Wrong** |
|
||
| `SHUTOFF` sn56-week6-transfer-e1 | Billing notice to `shettyatulya@gmail.com` | OpenStack `SHUTOFF`, no fault | Correct |
|
||
| `DELETING` hs-gaussian-splatting | Intent verified from events; delete in OpenStack | OpenStack `ACTIVE`, delete requested 17:00 | Correct |
|
||
| `DELETING` c3-pool-1785947188798 | Intent verified; delete in OpenStack | OpenStack `ERROR`, host `None`, "No valid host was found" | Correct action, **incomplete** — it never built, so the customer is owed the creation-failure notice too |
|
||
| `ERROR` hyperstack-minion-lydhjev | `no_valid_host` fault → never ACTIVE → insufficient-stock template → Lars Vagnes | Fault is exactly "No valid host was found. There are not enough hosts available."; host `None` | Correct — but was **hidden as "chronic"** |
|
||
|
||
### The Rogue VM rule is broken at source
|
||
|
||
```promql
|
||
sum by (instance) (In_Use_Gpus)
|
||
- sum by (instance) (Resources{status=~"ACTIVE|SHUTOFF|PRE_ACTIVE"}) >= 1
|
||
```
|
||
|
||
`In_Use_Gpus` equals `Total_Gpus` — the **physical** GPU count — on **71 of the 75**
|
||
firing hosts. So the expression reduces to *physical GPUs minus allocated GPUs*,
|
||
i.e. **"this host has at least one free GPU."** That is spare capacity, not a
|
||
rogue VM.
|
||
|
||
`CA1-ESC8-057` is the clean example: 8 physical, 7 allocated across 5 VMs that
|
||
both systems agree on, so it fires with a gap of 1.
|
||
|
||
Two consequences:
|
||
- 65 of the 72 rogue-VM alerts are now classified **invalid — alert rule defect**
|
||
and screened out, with the reason stated.
|
||
- The 2 that survive are genuine: `CA1-ESC812-252` (In_Use 2, one 1-GPU VM) and
|
||
`CA1-ESC812-289` (In_Use 7, 5 GPUs allocated). Both now read *"N GPU(s) in use
|
||
belong to no instance on either side"* and route to Infrastructure, because no
|
||
CX action can fix a leaked hypervisor allocation.
|
||
|
||
**This needs fixing in the Prometheus rule, not just filtered here.** Whoever owns
|
||
`infrahub-rules.yml` should either repair `In_Use_Gpus` or rewrite the expression.
|
||
|
||
### "Chronic" was hiding overdue customer contact
|
||
|
||
13 ERROR alerts have held for 6–7 days. The ERROR runbook says to contact the
|
||
customer if a stock-failure instance is not deleted **within a day**. Filing those
|
||
as "chronic, probably already ticketed" was wrong — they are overdue.
|
||
|
||
New `overdue` verdict: for kinds with a runbook time commitment, age makes a case
|
||
*more* urgent and it can never be demoted to chronic.
|
||
|
||
### Net effect
|
||
|
||
| | before validation | after |
|
||
|---|---|---|
|
||
| To action | 54 | **19** |
|
||
| of which overdue customer contact | 0 (hidden) | **13** |
|
||
| Rogue VM noise | 50 "needs action" | 65 flagged as a rule defect |
|
||
|
||
---
|
||
|
||
## 2. Integration plan
|
||
|
||
The principle: the app already knows *who* to contact and *what* to say. Wiring
|
||
delivery turns a 5-minute copy-paste into one reviewed click — without ever
|
||
sending on its own.
|
||
|
||
### Phase 1 — Zendesk (the main win)
|
||
|
||
`integrations.build_zendesk()` already produces a complete `POST /api/v2/tickets`
|
||
body: requester resolved from Infrahub owners, subject, the approved runbook
|
||
wording with placeholders filled, priority derived from the verdict, tags, and
|
||
`external_id = cx-triage-<fingerprint>` for idempotency.
|
||
|
||
To finish it:
|
||
|
||
1. **Credentials** — `CX_ZENDESK_SUBDOMAIN`, `CX_ZENDESK_EMAIL`, `CX_ZENDESK_TOKEN`
|
||
(API token, Basic auth as `email/token:token`). Read from 1Password via the
|
||
existing CX-Tools loader rather than env vars, so nothing lands on disk.
|
||
2. **Search before create** — `GET /api/v2/search?query=external_id:<fp>` so a
|
||
re-diagnosed alert updates the existing ticket instead of opening a duplicate.
|
||
3. **Send** — implement `App.send_zendesk`, which currently refuses by design.
|
||
4. **Guard rails** (all already scaffolded in the UI):
|
||
- two-step confirm naming the recipient
|
||
- never auto-send; no bulk send in v1
|
||
- an `--allow-send` startup flag, so a demo instance physically cannot email
|
||
- append the ticket URL back onto the case and log it to `vmc-audit.log`
|
||
5. **Requester matching** — Infrahub gives owner name + email; Zendesk may already
|
||
have that user. Search by email, fall back to creating the requester inline.
|
||
|
||
### Phase 2 — Jira
|
||
|
||
`integrations.build_jira()` produces the `POST /rest/api/3/issue` body with the
|
||
evidence block already assembled. Needs: project key confirmation (`INFRA`?),
|
||
issue type, and the same search-before-create against a `cx-triage-<fp>` label.
|
||
The GPU-leak escalations above are the immediate use case.
|
||
|
||
### Phase 3 — closing the loop
|
||
|
||
- **Slack** — react to the alert in `#infrahub-errors` and thread the findings,
|
||
which the runbook asks for manually today.
|
||
- **Case state** — persist handled/snoozed/ticketed per fingerprint (SQLite) so
|
||
the queue reflects work already done and survives a restart.
|
||
- **Bulk actions** — the 13 overdue ERROR alerts are one org-grouped mail-merge;
|
||
worth doing only once single-send is trusted.
|
||
|
||
### What stays manual, deliberately
|
||
|
||
Deleting, shelving and InfraInsight edits stay copy-a-command. The read-only
|
||
guarantee is what makes this safe to run against production, and the destructive
|
||
steps are exactly where a wrong verdict would be expensive.
|
||
|
||
---
|
||
|
||
## 3. UI
|
||
|
||
Live at `http://127.0.0.1:8765/` (previous version kept at `/classic`).
|
||
|
||
- **Left rail** — one line per case: subject, org, region, how long the condition
|
||
has actually held. Grouped and collapsible. Verdict filter chips across the top.
|
||
- **Case card** — verdict as a sentence, then a *picture* of the problem:
|
||
- state alerts: `Infrahub says X` ≠ `OpenStack says Y`
|
||
- GPU cases: a segmented allocation bar (green allocated / red unaccounted)
|
||
- duplicate IPs: one row per claimant
|
||
- **"Do this"** — the customer's name and address, then the action buttons.
|
||
Nothing else competes with them.
|
||
- **Everything else collapsed** — Caveats, Why, Runbook steps, Raw evidence.
|
||
- **Zendesk drawer** — recipient, subject, editable body, priority, tags, and a
|
||
Send button that is disabled and labelled *"Preview only. Nothing will be sent."*
|
||
until credentials exist and sending is explicitly enabled.
|