Split into a FastAPI backend and a React frontend, add case state and SSO
Some checks failed
build-and-deploy / test (push) Has been cancelled
build-and-deploy / image (push) Has been cancelled
build-and-deploy / deploy (push) Has been cancelled

The single-file stdlib server became the limit: no way to track what had been
done about an alert, no accounts, and a UI that had to be hand-rolled in
template strings. This restructures it into something deployable.

Backend (FastAPI)
- app/ holds config, database, auth, delivery and the routers; triagelib keeps
  the triage engine unchanged, so the validated screening and runbook logic is
  untouched.
- Cases persist per alert fingerprint with a status workflow (investigating,
  customer contacted, escalated to Infra, waiting, remediated, resolved, won't
  fix, false positive), an assignee, notes and an append-only history. An alert
  that stops and re-fires lands back on the same case and counts as a reopen.
- Suppression rules move from a JSON file into the database.

Auth
- Signed session cookies over PBKDF2 local accounts, plus an OIDC flow ready for
  Authentik: users are created on first login and admin follows a group claim.
  Local login can be switched off entirely once SSO is live.

Zendesk and Jira
- Delivery is now implemented, behind three gates: the integration must be
  configured, its feature flag on, and CX_FEATURE_SEND_ENABLED on. A demo
  instance leaves the last off and cannot mail anyone. Both search before
  creating, so re-diagnosing an alert updates one ticket rather than opening
  several, and a rolling daily cap stops a loop mailing everybody.

Deployment
- Multi-stage Dockerfile builds the bundle and serves it from the API origin.
- docker-compose for local and single-host use; Gitea Actions runs the tests,
  builds the image and renders deploy/k8s with envsubst.

Two fixes found while testing: assigning a case returned a null assignee, and
add_event could leave an already-loaded history collection stale.

Known gap: the engine reaches OpenStack via `docker exec <region>-osc`, which
does not work in a pod without the CX-Tools containers alongside it.
docs/DEPLOYMENT.md sets out the three ways to close that.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-06 07:11:28 +01:00
parent a039e0b5fd
commit 1262690276
68 changed files with 3839 additions and 2223 deletions

120
docs/LINKAGE.md Normal file
View File

@@ -0,0 +1,120 @@
# How the linkage scan works
## The problem it solves
Infrahub and OpenStack are two databases that are supposed to agree about which
VMs exist. The link between them is a single field: `openstack_id` on the Infrahub
record.
When a VM is created, roughly: Infrahub writes a record → asks OpenStack to build
a server → OpenStack returns a UUID → Infrahub stores that UUID. If the last step
fails, you get a record stuck in `CREATING` or `ERROR` with no `openstack_id`,
while the server itself is running perfectly.
On an alert dashboard that is indistinguishable from a genuine build failure. They
need opposite responses:
| | Genuine failure | Linkage failure |
|---|---|---|
| Server exists? | no | **yes, and running** |
| Fix | customer recreates | repair the Infrahub record |
| Billing | nothing to bill | **running, billing nobody** |
| Customer sees | "my VM failed" | "my VM failed" |
Tell a customer to recreate a VM that is actually running and you have doubled
their spend on a machine they think is broken.
## The scan, step by step
### 1. Pull everything OpenStack has
For each region, one call:
```
openstack server list --all-projects --long -f json
```
~2,500 rows for ca1, ~500 for no1, ~80 for us1. Each row gives ID, name, status,
task state, host, project. About 45 seconds per region, so ~2.5 minutes total —
which is why it is a button, not something that runs on page load.
Two indexes are built: **by UUID** and **by lowercased name**.
### 2. Pull everything Infrahub has
Free — it is already in the `Resources` metric snapshot the queue uses (~4,300
records, one Prometheus query). Each carries `openstack_id`, `instance_name`,
`status`, `region`, `organization`, and GPU count.
### 3. Only judge regions that actually answered
If a region's listing failed, every VM in it would look "missing from OpenStack".
So records in unscanned regions are **excluded entirely** and counted separately.
This is not hypothetical. On the first real run `ca2` failed
(`Could not find versioned identity endpoints`) and produced **317 false
positives**. With the filter, the same scan returns **6**.
### 4. Find Infrahub records with a broken link
For each Infrahub record whose status is one where a live server is expected —
`ERROR`, `CREATING`, `BUILD`, `ACTIVE`, `REBOOTING`, `RESTORING` — flag it if:
- it has **no** `openstack_id`, or
- it has one that **is not in the OpenStack index**
`HIBERNATED` is deliberately excluded: a shelved instance legitimately has no
running server, and including it would drown the result (2,400 records).
### 5. Pair them up by name
For each flagged record, look for OpenStack servers with the **same name**. In
Hyperstack the OpenStack server name matches the Infrahub instance name, which
makes this a strong signal.
Confidence is then graded:
| Confidence | Meaning |
|---|---|
| **high** | A same-named server exists, **no other Infrahub record claims it**, and it is not itself in ERROR. This is a linkage failure — the server is real and orphaned. |
| **medium** | A same-named server exists but another Infrahub record already owns that UUID, or it is SHELVED_OFFLOADED. Could be a name collision; needs a human. |
| **none** | No same-named server. The record's server genuinely does not exist — a real failure, or it was deleted. |
**Details** on a row runs `server show` for the candidate to fetch its creation
time, launch time and fault, so you can confirm the timing lines up with when the
Infrahub record was created.
### 6. The reverse: servers nobody claims
Every OpenStack server whose UUID appears in **no** Infrahub record. These are
running, consuming GPUs, and billing nobody. The scan also notes whether the
*name* is known to Infrahub, which separates "record exists but the link is
broken" from "completely unknown".
This is what `:pirate_flag:Suspected Orphan VM` is supposed to catch — but that
rule is built on `openstack_nova_server_status`, which currently returns zero
series, so it cannot fire at all. Until that exporter is fixed, this scan is the
only thing looking.
## What it found on the first live run
- **`daring-rutherford`** — Infrahub `CREATING`, no `openstack_id` recorded;
OpenStack had an **ACTIVE** server of exactly that name in ca1 that no other
record claimed. Textbook linkage failure. (It had linked itself by the next
scan — caught mid-flight.)
- **177 OpenStack servers** with no Infrahub record at all.
- **ca2 unreachable**, so that region is excluded and reported rather than
silently producing garbage.
## Known limits
- **Name matching is a heuristic.** Two VMs can share a name across orgs. That is
what the confidence grade and the "claimed by another record" flag are for —
nothing here should be actioned without opening the candidate.
- **No creation-time matching yet.** The bulk list does not include creation
timestamps, so time correlation is per-candidate, on demand. If you want
time-window matching as a primary signal rather than a confirmation, that needs
a `server show` per candidate — fine for tens, not for thousands.
- **Point in time.** A VM mid-build will look unlinked. Two scans a few minutes
apart separate "still settling" from "actually stuck".
- **ca2 is currently unscannable** — an environment problem, not a scan problem.