CX Triage: alert diagnosis over the CX-Tools collectors
Read-only triage for the Infrahub error alerts. Pulls the Prometheus alert queue, re-checks each alert's condition against live state to separate real work from noise, diagnoses it using the CX runbooks, and drafts the customer comms with contacts resolved from Infrahub. Findings from validating against production: - "Suspected Rogue VM" fires on spare GPU capacity, not rogue VMs: In_Use_Gpus equals the physical count on 71 of 75 firing hosts, so the rule reduces to "this host has a free GPU". Verified against OpenStack on 10 hosts. - "Exists in Infrahub but does not exist in OpenStack" matches every VM because openstack_nova_server_status returns no series; excluded as a rule defect. - Prometheus activeAt is reset several times a day by dips in the Resources metric, so alert ages are recovered from ALERTS history instead. Takes ~2,650 firing alerts down to ~20 that need a decision. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
120
LINKAGE.md
Normal file
120
LINKAGE.md
Normal file
@@ -0,0 +1,120 @@
|
||||
# How the linkage scan works
|
||||
|
||||
## The problem it solves
|
||||
|
||||
Infrahub and OpenStack are two databases that are supposed to agree about which
|
||||
VMs exist. The link between them is a single field: `openstack_id` on the Infrahub
|
||||
record.
|
||||
|
||||
When a VM is created, roughly: Infrahub writes a record → asks OpenStack to build
|
||||
a server → OpenStack returns a UUID → Infrahub stores that UUID. If the last step
|
||||
fails, you get a record stuck in `CREATING` or `ERROR` with no `openstack_id`,
|
||||
while the server itself is running perfectly.
|
||||
|
||||
On an alert dashboard that is indistinguishable from a genuine build failure. They
|
||||
need opposite responses:
|
||||
|
||||
| | Genuine failure | Linkage failure |
|
||||
|---|---|---|
|
||||
| Server exists? | no | **yes, and running** |
|
||||
| Fix | customer recreates | repair the Infrahub record |
|
||||
| Billing | nothing to bill | **running, billing nobody** |
|
||||
| Customer sees | "my VM failed" | "my VM failed" |
|
||||
|
||||
Tell a customer to recreate a VM that is actually running and you have doubled
|
||||
their spend on a machine they think is broken.
|
||||
|
||||
## The scan, step by step
|
||||
|
||||
### 1. Pull everything OpenStack has
|
||||
|
||||
For each region, one call:
|
||||
|
||||
```
|
||||
openstack server list --all-projects --long -f json
|
||||
```
|
||||
|
||||
~2,500 rows for ca1, ~500 for no1, ~80 for us1. Each row gives ID, name, status,
|
||||
task state, host, project. About 45 seconds per region, so ~2.5 minutes total —
|
||||
which is why it is a button, not something that runs on page load.
|
||||
|
||||
Two indexes are built: **by UUID** and **by lowercased name**.
|
||||
|
||||
### 2. Pull everything Infrahub has
|
||||
|
||||
Free — it is already in the `Resources` metric snapshot the queue uses (~4,300
|
||||
records, one Prometheus query). Each carries `openstack_id`, `instance_name`,
|
||||
`status`, `region`, `organization`, and GPU count.
|
||||
|
||||
### 3. Only judge regions that actually answered
|
||||
|
||||
If a region's listing failed, every VM in it would look "missing from OpenStack".
|
||||
So records in unscanned regions are **excluded entirely** and counted separately.
|
||||
|
||||
This is not hypothetical. On the first real run `ca2` failed
|
||||
(`Could not find versioned identity endpoints`) and produced **317 false
|
||||
positives**. With the filter, the same scan returns **6**.
|
||||
|
||||
### 4. Find Infrahub records with a broken link
|
||||
|
||||
For each Infrahub record whose status is one where a live server is expected —
|
||||
`ERROR`, `CREATING`, `BUILD`, `ACTIVE`, `REBOOTING`, `RESTORING` — flag it if:
|
||||
|
||||
- it has **no** `openstack_id`, or
|
||||
- it has one that **is not in the OpenStack index**
|
||||
|
||||
`HIBERNATED` is deliberately excluded: a shelved instance legitimately has no
|
||||
running server, and including it would drown the result (2,400 records).
|
||||
|
||||
### 5. Pair them up by name
|
||||
|
||||
For each flagged record, look for OpenStack servers with the **same name**. In
|
||||
Hyperstack the OpenStack server name matches the Infrahub instance name, which
|
||||
makes this a strong signal.
|
||||
|
||||
Confidence is then graded:
|
||||
|
||||
| Confidence | Meaning |
|
||||
|---|---|
|
||||
| **high** | A same-named server exists, **no other Infrahub record claims it**, and it is not itself in ERROR. This is a linkage failure — the server is real and orphaned. |
|
||||
| **medium** | A same-named server exists but another Infrahub record already owns that UUID, or it is SHELVED_OFFLOADED. Could be a name collision; needs a human. |
|
||||
| **none** | No same-named server. The record's server genuinely does not exist — a real failure, or it was deleted. |
|
||||
|
||||
**Details** on a row runs `server show` for the candidate to fetch its creation
|
||||
time, launch time and fault, so you can confirm the timing lines up with when the
|
||||
Infrahub record was created.
|
||||
|
||||
### 6. The reverse: servers nobody claims
|
||||
|
||||
Every OpenStack server whose UUID appears in **no** Infrahub record. These are
|
||||
running, consuming GPUs, and billing nobody. The scan also notes whether the
|
||||
*name* is known to Infrahub, which separates "record exists but the link is
|
||||
broken" from "completely unknown".
|
||||
|
||||
This is what `:pirate_flag:Suspected Orphan VM` is supposed to catch — but that
|
||||
rule is built on `openstack_nova_server_status`, which currently returns zero
|
||||
series, so it cannot fire at all. Until that exporter is fixed, this scan is the
|
||||
only thing looking.
|
||||
|
||||
## What it found on the first live run
|
||||
|
||||
- **`daring-rutherford`** — Infrahub `CREATING`, no `openstack_id` recorded;
|
||||
OpenStack had an **ACTIVE** server of exactly that name in ca1 that no other
|
||||
record claimed. Textbook linkage failure. (It had linked itself by the next
|
||||
scan — caught mid-flight.)
|
||||
- **177 OpenStack servers** with no Infrahub record at all.
|
||||
- **ca2 unreachable**, so that region is excluded and reported rather than
|
||||
silently producing garbage.
|
||||
|
||||
## Known limits
|
||||
|
||||
- **Name matching is a heuristic.** Two VMs can share a name across orgs. That is
|
||||
what the confidence grade and the "claimed by another record" flag are for —
|
||||
nothing here should be actioned without opening the candidate.
|
||||
- **No creation-time matching yet.** The bulk list does not include creation
|
||||
timestamps, so time correlation is per-candidate, on demand. If you want
|
||||
time-window matching as a primary signal rather than a confirmation, that needs
|
||||
a `server show` per candidate — fine for tens, not for thousands.
|
||||
- **Point in time.** A VM mid-build will look unlinked. Two scans a few minutes
|
||||
apart separate "still settling" from "actually stuck".
|
||||
- **ca2 is currently unscannable** — an environment problem, not a scan problem.
|
||||
Reference in New Issue
Block a user