Parham Monfared
1262690276
The single-file stdlib server became the limit: no way to track what had been done about an alert, no accounts, and a UI that had to be hand-rolled in template strings. This restructures it into something deployable. Backend (FastAPI) - app/ holds config, database, auth, delivery and the routers; triagelib keeps the triage engine unchanged, so the validated screening and runbook logic is untouched. - Cases persist per alert fingerprint with a status workflow (investigating, customer contacted, escalated to Infra, waiting, remediated, resolved, won't fix, false positive), an assignee, notes and an append-only history. An alert that stops and re-fires lands back on the same case and counts as a reopen. - Suppression rules move from a JSON file into the database. Auth - Signed session cookies over PBKDF2 local accounts, plus an OIDC flow ready for Authentik: users are created on first login and admin follows a group claim. Local login can be switched off entirely once SSO is live. Zendesk and Jira - Delivery is now implemented, behind three gates: the integration must be configured, its feature flag on, and CX_FEATURE_SEND_ENABLED on. A demo instance leaves the last off and cannot mail anyone. Both search before creating, so re-diagnosing an alert updates one ticket rather than opening several, and a rolling daily cap stops a loop mailing everybody. Deployment - Multi-stage Dockerfile builds the bundle and serves it from the API origin. - docker-compose for local and single-host use; Gitea Actions runs the tests, builds the image and renders deploy/k8s with envsubst. Two fixes found while testing: assigning a case returned a null assignee, and add_event could leave an already-loaded history collection stale. Known gap: the engine reaches OpenStack via `docker exec <region>-osc`, which does not work in a pod without the CX-Tools containers alongside it. docs/DEPLOYMENT.md sets out the three ways to close that. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
5.2 KiB
How the linkage scan works
The problem it solves
Infrahub and OpenStack are two databases that are supposed to agree about which
VMs exist. The link between them is a single field: openstack_id on the Infrahub
record.
When a VM is created, roughly: Infrahub writes a record → asks OpenStack to build
a server → OpenStack returns a UUID → Infrahub stores that UUID. If the last step
fails, you get a record stuck in CREATING or ERROR with no openstack_id,
while the server itself is running perfectly.
On an alert dashboard that is indistinguishable from a genuine build failure. They need opposite responses:
| Genuine failure | Linkage failure | |
|---|---|---|
| Server exists? | no | yes, and running |
| Fix | customer recreates | repair the Infrahub record |
| Billing | nothing to bill | running, billing nobody |
| Customer sees | "my VM failed" | "my VM failed" |
Tell a customer to recreate a VM that is actually running and you have doubled their spend on a machine they think is broken.
The scan, step by step
1. Pull everything OpenStack has
For each region, one call:
openstack server list --all-projects --long -f json
~2,500 rows for ca1, ~500 for no1, ~80 for us1. Each row gives ID, name, status, task state, host, project. About 45 seconds per region, so ~2.5 minutes total — which is why it is a button, not something that runs on page load.
Two indexes are built: by UUID and by lowercased name.
2. Pull everything Infrahub has
Free — it is already in the Resources metric snapshot the queue uses (~4,300
records, one Prometheus query). Each carries openstack_id, instance_name,
status, region, organization, and GPU count.
3. Only judge regions that actually answered
If a region's listing failed, every VM in it would look "missing from OpenStack". So records in unscanned regions are excluded entirely and counted separately.
This is not hypothetical. On the first real run ca2 failed
(Could not find versioned identity endpoints) and produced 317 false
positives. With the filter, the same scan returns 6.
4. Find Infrahub records with a broken link
For each Infrahub record whose status is one where a live server is expected —
ERROR, CREATING, BUILD, ACTIVE, REBOOTING, RESTORING — flag it if:
- it has no
openstack_id, or - it has one that is not in the OpenStack index
HIBERNATED is deliberately excluded: a shelved instance legitimately has no
running server, and including it would drown the result (2,400 records).
5. Pair them up by name
For each flagged record, look for OpenStack servers with the same name. In Hyperstack the OpenStack server name matches the Infrahub instance name, which makes this a strong signal.
Confidence is then graded:
| Confidence | Meaning |
|---|---|
| high | A same-named server exists, no other Infrahub record claims it, and it is not itself in ERROR. This is a linkage failure — the server is real and orphaned. |
| medium | A same-named server exists but another Infrahub record already owns that UUID, or it is SHELVED_OFFLOADED. Could be a name collision; needs a human. |
| none | No same-named server. The record's server genuinely does not exist — a real failure, or it was deleted. |
Details on a row runs server show for the candidate to fetch its creation
time, launch time and fault, so you can confirm the timing lines up with when the
Infrahub record was created.
6. The reverse: servers nobody claims
Every OpenStack server whose UUID appears in no Infrahub record. These are running, consuming GPUs, and billing nobody. The scan also notes whether the name is known to Infrahub, which separates "record exists but the link is broken" from "completely unknown".
This is what :pirate_flag:Suspected Orphan VM is supposed to catch — but that
rule is built on openstack_nova_server_status, which currently returns zero
series, so it cannot fire at all. Until that exporter is fixed, this scan is the
only thing looking.
What it found on the first live run
daring-rutherford— InfrahubCREATING, noopenstack_idrecorded; OpenStack had an ACTIVE server of exactly that name in ca1 that no other record claimed. Textbook linkage failure. (It had linked itself by the next scan — caught mid-flight.)- 177 OpenStack servers with no Infrahub record at all.
- ca2 unreachable, so that region is excluded and reported rather than silently producing garbage.
Known limits
- Name matching is a heuristic. Two VMs can share a name across orgs. That is what the confidence grade and the "claimed by another record" flag are for — nothing here should be actioned without opening the candidate.
- No creation-time matching yet. The bulk list does not include creation
timestamps, so time correlation is per-candidate, on demand. If you want
time-window matching as a primary signal rather than a confirmation, that needs
a
server showper candidate — fine for tens, not for thousands. - Point in time. A VM mid-build will look unlinked. Two scans a few minutes apart separate "still settling" from "actually stuck".
- ca2 is currently unscannable — an environment problem, not a scan problem.