Split into a FastAPI backend and a React frontend, add case state and SSO
The single-file stdlib server became the limit: no way to track what had been done about an alert, no accounts, and a UI that had to be hand-rolled in template strings. This restructures it into something deployable. Backend (FastAPI) - app/ holds config, database, auth, delivery and the routers; triagelib keeps the triage engine unchanged, so the validated screening and runbook logic is untouched. - Cases persist per alert fingerprint with a status workflow (investigating, customer contacted, escalated to Infra, waiting, remediated, resolved, won't fix, false positive), an assignee, notes and an append-only history. An alert that stops and re-fires lands back on the same case and counts as a reopen. - Suppression rules move from a JSON file into the database. Auth - Signed session cookies over PBKDF2 local accounts, plus an OIDC flow ready for Authentik: users are created on first login and admin follows a group claim. Local login can be switched off entirely once SSO is live. Zendesk and Jira - Delivery is now implemented, behind three gates: the integration must be configured, its feature flag on, and CX_FEATURE_SEND_ENABLED on. A demo instance leaves the last off and cannot mail anyone. Both search before creating, so re-diagnosing an alert updates one ticket rather than opening several, and a rolling daily cap stops a loop mailing everybody. Deployment - Multi-stage Dockerfile builds the bundle and serves it from the API origin. - docker-compose for local and single-host use; Gitea Actions runs the tests, builds the image and renders deploy/k8s with envsubst. Two fixes found while testing: assigning a case returned a null assignee, and add_event could leave an already-loaded history collection stale. Known gap: the engine reaches OpenStack via `docker exec <region>-osc`, which does not work in a pod without the CX-Tools containers alongside it. docs/DEPLOYMENT.md sets out the three ways to close that. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
120
docs/LINKAGE.md
Normal file
120
docs/LINKAGE.md
Normal file
@@ -0,0 +1,120 @@
|
||||
# How the linkage scan works
|
||||
|
||||
## The problem it solves
|
||||
|
||||
Infrahub and OpenStack are two databases that are supposed to agree about which
|
||||
VMs exist. The link between them is a single field: `openstack_id` on the Infrahub
|
||||
record.
|
||||
|
||||
When a VM is created, roughly: Infrahub writes a record → asks OpenStack to build
|
||||
a server → OpenStack returns a UUID → Infrahub stores that UUID. If the last step
|
||||
fails, you get a record stuck in `CREATING` or `ERROR` with no `openstack_id`,
|
||||
while the server itself is running perfectly.
|
||||
|
||||
On an alert dashboard that is indistinguishable from a genuine build failure. They
|
||||
need opposite responses:
|
||||
|
||||
| | Genuine failure | Linkage failure |
|
||||
|---|---|---|
|
||||
| Server exists? | no | **yes, and running** |
|
||||
| Fix | customer recreates | repair the Infrahub record |
|
||||
| Billing | nothing to bill | **running, billing nobody** |
|
||||
| Customer sees | "my VM failed" | "my VM failed" |
|
||||
|
||||
Tell a customer to recreate a VM that is actually running and you have doubled
|
||||
their spend on a machine they think is broken.
|
||||
|
||||
## The scan, step by step
|
||||
|
||||
### 1. Pull everything OpenStack has
|
||||
|
||||
For each region, one call:
|
||||
|
||||
```
|
||||
openstack server list --all-projects --long -f json
|
||||
```
|
||||
|
||||
~2,500 rows for ca1, ~500 for no1, ~80 for us1. Each row gives ID, name, status,
|
||||
task state, host, project. About 45 seconds per region, so ~2.5 minutes total —
|
||||
which is why it is a button, not something that runs on page load.
|
||||
|
||||
Two indexes are built: **by UUID** and **by lowercased name**.
|
||||
|
||||
### 2. Pull everything Infrahub has
|
||||
|
||||
Free — it is already in the `Resources` metric snapshot the queue uses (~4,300
|
||||
records, one Prometheus query). Each carries `openstack_id`, `instance_name`,
|
||||
`status`, `region`, `organization`, and GPU count.
|
||||
|
||||
### 3. Only judge regions that actually answered
|
||||
|
||||
If a region's listing failed, every VM in it would look "missing from OpenStack".
|
||||
So records in unscanned regions are **excluded entirely** and counted separately.
|
||||
|
||||
This is not hypothetical. On the first real run `ca2` failed
|
||||
(`Could not find versioned identity endpoints`) and produced **317 false
|
||||
positives**. With the filter, the same scan returns **6**.
|
||||
|
||||
### 4. Find Infrahub records with a broken link
|
||||
|
||||
For each Infrahub record whose status is one where a live server is expected —
|
||||
`ERROR`, `CREATING`, `BUILD`, `ACTIVE`, `REBOOTING`, `RESTORING` — flag it if:
|
||||
|
||||
- it has **no** `openstack_id`, or
|
||||
- it has one that **is not in the OpenStack index**
|
||||
|
||||
`HIBERNATED` is deliberately excluded: a shelved instance legitimately has no
|
||||
running server, and including it would drown the result (2,400 records).
|
||||
|
||||
### 5. Pair them up by name
|
||||
|
||||
For each flagged record, look for OpenStack servers with the **same name**. In
|
||||
Hyperstack the OpenStack server name matches the Infrahub instance name, which
|
||||
makes this a strong signal.
|
||||
|
||||
Confidence is then graded:
|
||||
|
||||
| Confidence | Meaning |
|
||||
|---|---|
|
||||
| **high** | A same-named server exists, **no other Infrahub record claims it**, and it is not itself in ERROR. This is a linkage failure — the server is real and orphaned. |
|
||||
| **medium** | A same-named server exists but another Infrahub record already owns that UUID, or it is SHELVED_OFFLOADED. Could be a name collision; needs a human. |
|
||||
| **none** | No same-named server. The record's server genuinely does not exist — a real failure, or it was deleted. |
|
||||
|
||||
**Details** on a row runs `server show` for the candidate to fetch its creation
|
||||
time, launch time and fault, so you can confirm the timing lines up with when the
|
||||
Infrahub record was created.
|
||||
|
||||
### 6. The reverse: servers nobody claims
|
||||
|
||||
Every OpenStack server whose UUID appears in **no** Infrahub record. These are
|
||||
running, consuming GPUs, and billing nobody. The scan also notes whether the
|
||||
*name* is known to Infrahub, which separates "record exists but the link is
|
||||
broken" from "completely unknown".
|
||||
|
||||
This is what `:pirate_flag:Suspected Orphan VM` is supposed to catch — but that
|
||||
rule is built on `openstack_nova_server_status`, which currently returns zero
|
||||
series, so it cannot fire at all. Until that exporter is fixed, this scan is the
|
||||
only thing looking.
|
||||
|
||||
## What it found on the first live run
|
||||
|
||||
- **`daring-rutherford`** — Infrahub `CREATING`, no `openstack_id` recorded;
|
||||
OpenStack had an **ACTIVE** server of exactly that name in ca1 that no other
|
||||
record claimed. Textbook linkage failure. (It had linked itself by the next
|
||||
scan — caught mid-flight.)
|
||||
- **177 OpenStack servers** with no Infrahub record at all.
|
||||
- **ca2 unreachable**, so that region is excluded and reported rather than
|
||||
silently producing garbage.
|
||||
|
||||
## Known limits
|
||||
|
||||
- **Name matching is a heuristic.** Two VMs can share a name across orgs. That is
|
||||
what the confidence grade and the "claimed by another record" flag are for —
|
||||
nothing here should be actioned without opening the candidate.
|
||||
- **No creation-time matching yet.** The bulk list does not include creation
|
||||
timestamps, so time correlation is per-candidate, on demand. If you want
|
||||
time-window matching as a primary signal rather than a confirmation, that needs
|
||||
a `server show` per candidate — fine for tens, not for thousands.
|
||||
- **Point in time.** A VM mid-build will look unlinked. Two scans a few minutes
|
||||
apart separate "still settling" from "actually stuck".
|
||||
- **ca2 is currently unscannable** — an environment problem, not a scan problem.
|
||||
Reference in New Issue
Block a user