Parham Monfared
1262690276
The single-file stdlib server became the limit: no way to track what had been done about an alert, no accounts, and a UI that had to be hand-rolled in template strings. This restructures it into something deployable. Backend (FastAPI) - app/ holds config, database, auth, delivery and the routers; triagelib keeps the triage engine unchanged, so the validated screening and runbook logic is untouched. - Cases persist per alert fingerprint with a status workflow (investigating, customer contacted, escalated to Infra, waiting, remediated, resolved, won't fix, false positive), an assignee, notes and an append-only history. An alert that stops and re-fires lands back on the same case and counts as a reopen. - Suppression rules move from a JSON file into the database. Auth - Signed session cookies over PBKDF2 local accounts, plus an OIDC flow ready for Authentik: users are created on first login and admin follows a group claim. Local login can be switched off entirely once SSO is live. Zendesk and Jira - Delivery is now implemented, behind three gates: the integration must be configured, its feature flag on, and CX_FEATURE_SEND_ENABLED on. A demo instance leaves the last off and cannot mail anyone. Both search before creating, so re-diagnosing an alert updates one ticket rather than opening several, and a rolling daily cap stops a loop mailing everybody. Deployment - Multi-stage Dockerfile builds the bundle and serves it from the API origin. - docker-compose for local and single-host use; Gitea Actions runs the tests, builds the image and renders deploy/k8s with envsubst. Two fixes found while testing: assigning a case returned a null assignee, and add_event could leave an already-loaded history collection stale. Known gap: the engine reaches OpenStack via `docker exec <region>-osc`, which does not work in a pod without the CX-Tools containers alongside it. docs/DEPLOYMENT.md sets out the three ways to close that. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
121 lines
5.2 KiB
Markdown
121 lines
5.2 KiB
Markdown
# How the linkage scan works
|
|
|
|
## The problem it solves
|
|
|
|
Infrahub and OpenStack are two databases that are supposed to agree about which
|
|
VMs exist. The link between them is a single field: `openstack_id` on the Infrahub
|
|
record.
|
|
|
|
When a VM is created, roughly: Infrahub writes a record → asks OpenStack to build
|
|
a server → OpenStack returns a UUID → Infrahub stores that UUID. If the last step
|
|
fails, you get a record stuck in `CREATING` or `ERROR` with no `openstack_id`,
|
|
while the server itself is running perfectly.
|
|
|
|
On an alert dashboard that is indistinguishable from a genuine build failure. They
|
|
need opposite responses:
|
|
|
|
| | Genuine failure | Linkage failure |
|
|
|---|---|---|
|
|
| Server exists? | no | **yes, and running** |
|
|
| Fix | customer recreates | repair the Infrahub record |
|
|
| Billing | nothing to bill | **running, billing nobody** |
|
|
| Customer sees | "my VM failed" | "my VM failed" |
|
|
|
|
Tell a customer to recreate a VM that is actually running and you have doubled
|
|
their spend on a machine they think is broken.
|
|
|
|
## The scan, step by step
|
|
|
|
### 1. Pull everything OpenStack has
|
|
|
|
For each region, one call:
|
|
|
|
```
|
|
openstack server list --all-projects --long -f json
|
|
```
|
|
|
|
~2,500 rows for ca1, ~500 for no1, ~80 for us1. Each row gives ID, name, status,
|
|
task state, host, project. About 45 seconds per region, so ~2.5 minutes total —
|
|
which is why it is a button, not something that runs on page load.
|
|
|
|
Two indexes are built: **by UUID** and **by lowercased name**.
|
|
|
|
### 2. Pull everything Infrahub has
|
|
|
|
Free — it is already in the `Resources` metric snapshot the queue uses (~4,300
|
|
records, one Prometheus query). Each carries `openstack_id`, `instance_name`,
|
|
`status`, `region`, `organization`, and GPU count.
|
|
|
|
### 3. Only judge regions that actually answered
|
|
|
|
If a region's listing failed, every VM in it would look "missing from OpenStack".
|
|
So records in unscanned regions are **excluded entirely** and counted separately.
|
|
|
|
This is not hypothetical. On the first real run `ca2` failed
|
|
(`Could not find versioned identity endpoints`) and produced **317 false
|
|
positives**. With the filter, the same scan returns **6**.
|
|
|
|
### 4. Find Infrahub records with a broken link
|
|
|
|
For each Infrahub record whose status is one where a live server is expected —
|
|
`ERROR`, `CREATING`, `BUILD`, `ACTIVE`, `REBOOTING`, `RESTORING` — flag it if:
|
|
|
|
- it has **no** `openstack_id`, or
|
|
- it has one that **is not in the OpenStack index**
|
|
|
|
`HIBERNATED` is deliberately excluded: a shelved instance legitimately has no
|
|
running server, and including it would drown the result (2,400 records).
|
|
|
|
### 5. Pair them up by name
|
|
|
|
For each flagged record, look for OpenStack servers with the **same name**. In
|
|
Hyperstack the OpenStack server name matches the Infrahub instance name, which
|
|
makes this a strong signal.
|
|
|
|
Confidence is then graded:
|
|
|
|
| Confidence | Meaning |
|
|
|---|---|
|
|
| **high** | A same-named server exists, **no other Infrahub record claims it**, and it is not itself in ERROR. This is a linkage failure — the server is real and orphaned. |
|
|
| **medium** | A same-named server exists but another Infrahub record already owns that UUID, or it is SHELVED_OFFLOADED. Could be a name collision; needs a human. |
|
|
| **none** | No same-named server. The record's server genuinely does not exist — a real failure, or it was deleted. |
|
|
|
|
**Details** on a row runs `server show` for the candidate to fetch its creation
|
|
time, launch time and fault, so you can confirm the timing lines up with when the
|
|
Infrahub record was created.
|
|
|
|
### 6. The reverse: servers nobody claims
|
|
|
|
Every OpenStack server whose UUID appears in **no** Infrahub record. These are
|
|
running, consuming GPUs, and billing nobody. The scan also notes whether the
|
|
*name* is known to Infrahub, which separates "record exists but the link is
|
|
broken" from "completely unknown".
|
|
|
|
This is what `:pirate_flag:Suspected Orphan VM` is supposed to catch — but that
|
|
rule is built on `openstack_nova_server_status`, which currently returns zero
|
|
series, so it cannot fire at all. Until that exporter is fixed, this scan is the
|
|
only thing looking.
|
|
|
|
## What it found on the first live run
|
|
|
|
- **`daring-rutherford`** — Infrahub `CREATING`, no `openstack_id` recorded;
|
|
OpenStack had an **ACTIVE** server of exactly that name in ca1 that no other
|
|
record claimed. Textbook linkage failure. (It had linked itself by the next
|
|
scan — caught mid-flight.)
|
|
- **177 OpenStack servers** with no Infrahub record at all.
|
|
- **ca2 unreachable**, so that region is excluded and reported rather than
|
|
silently producing garbage.
|
|
|
|
## Known limits
|
|
|
|
- **Name matching is a heuristic.** Two VMs can share a name across orgs. That is
|
|
what the confidence grade and the "claimed by another record" flag are for —
|
|
nothing here should be actioned without opening the candidate.
|
|
- **No creation-time matching yet.** The bulk list does not include creation
|
|
timestamps, so time correlation is per-candidate, on demand. If you want
|
|
time-window matching as a primary signal rather than a confirmation, that needs
|
|
a `server show` per candidate — fine for tens, not for thousands.
|
|
- **Point in time.** A VM mid-build will look unlinked. Two scans a few minutes
|
|
apart separate "still settling" from "actually stuck".
|
|
- **ca2 is currently unscannable** — an environment problem, not a scan problem.
|