Files

CX Triage: alert diagnosis over the CX-Tools collectors

Read-only triage for the Infrahub error alerts. Pulls the Prometheus alert
queue, re-checks each alert's condition against live state to separate real
work from noise, diagnoses it using the CX runbooks, and drafts the customer
comms with contacts resolved from Infrahub.

Findings from validating against production:
- "Suspected Rogue VM" fires on spare GPU capacity, not rogue VMs: In_Use_Gpus
  equals the physical count on 71 of 75 firing hosts, so the rule reduces to
  "this host has a free GPU". Verified against OpenStack on 10 hosts.
- "Exists in Infrahub but does not exist in OpenStack" matches every VM because
  openstack_nova_server_status returns no series; excluded as a rule defect.
- Prometheus activeAt is reset several times a day by dips in the Resources
  metric, so alert ages are recovered from ALERTS history instead.

Takes ~2,650 firing alerts down to ~20 that need a decision.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

2026-08-06 06:48:34 +01:00

5.2 KiB

Raw Blame History

How the linkage scan works

The problem it solves

Infrahub and OpenStack are two databases that are supposed to agree about which VMs exist. The link between them is a single field: openstack_id on the Infrahub record.

When a VM is created, roughly: Infrahub writes a record → asks OpenStack to build a server → OpenStack returns a UUID → Infrahub stores that UUID. If the last step fails, you get a record stuck in CREATING or ERROR with no openstack_id, while the server itself is running perfectly.

On an alert dashboard that is indistinguishable from a genuine build failure. They need opposite responses:

	Genuine failure	Linkage failure
Server exists?	no	yes, and running
Fix	customer recreates	repair the Infrahub record
Billing	nothing to bill	running, billing nobody
Customer sees	"my VM failed"	"my VM failed"

Tell a customer to recreate a VM that is actually running and you have doubled their spend on a machine they think is broken.

The scan, step by step

1. Pull everything OpenStack has

For each region, one call:

openstack server list --all-projects --long -f json

~2,500 rows for ca1, ~500 for no1, ~80 for us1. Each row gives ID, name, status, task state, host, project. About 45 seconds per region, so ~2.5 minutes total — which is why it is a button, not something that runs on page load.

Two indexes are built: by UUID and by lowercased name.

2. Pull everything Infrahub has

Free — it is already in the Resources metric snapshot the queue uses (~4,300 records, one Prometheus query). Each carries openstack_id, instance_name, status, region, organization, and GPU count.

3. Only judge regions that actually answered

If a region's listing failed, every VM in it would look "missing from OpenStack". So records in unscanned regions are excluded entirely and counted separately.

This is not hypothetical. On the first real run ca2 failed (Could not find versioned identity endpoints) and produced 317 false positives. With the filter, the same scan returns 6.

4. Find Infrahub records with a broken link

For each Infrahub record whose status is one where a live server is expected — ERROR, CREATING, BUILD, ACTIVE, REBOOTING, RESTORING — flag it if:

it has no openstack_id, or
it has one that is not in the OpenStack index

HIBERNATED is deliberately excluded: a shelved instance legitimately has no running server, and including it would drown the result (2,400 records).

5. Pair them up by name

For each flagged record, look for OpenStack servers with the same name. In Hyperstack the OpenStack server name matches the Infrahub instance name, which makes this a strong signal.

Confidence is then graded:

Confidence	Meaning
high	A same-named server exists, no other Infrahub record claims it, and it is not itself in ERROR. This is a linkage failure — the server is real and orphaned.
medium	A same-named server exists but another Infrahub record already owns that UUID, or it is SHELVED_OFFLOADED. Could be a name collision; needs a human.
none	No same-named server. The record's server genuinely does not exist — a real failure, or it was deleted.

Details on a row runs server show for the candidate to fetch its creation time, launch time and fault, so you can confirm the timing lines up with when the Infrahub record was created.

6. The reverse: servers nobody claims

Every OpenStack server whose UUID appears in no Infrahub record. These are running, consuming GPUs, and billing nobody. The scan also notes whether the name is known to Infrahub, which separates "record exists but the link is broken" from "completely unknown".

This is what :pirate_flag:Suspected Orphan VM is supposed to catch — but that rule is built on openstack_nova_server_status, which currently returns zero series, so it cannot fire at all. Until that exporter is fixed, this scan is the only thing looking.

What it found on the first live run

daring-rutherford — Infrahub CREATING, no openstack_id recorded; OpenStack had an ACTIVE server of exactly that name in ca1 that no other record claimed. Textbook linkage failure. (It had linked itself by the next scan — caught mid-flight.)
177 OpenStack servers with no Infrahub record at all.
ca2 unreachable, so that region is excluded and reported rather than silently producing garbage.

Known limits

Name matching is a heuristic. Two VMs can share a name across orgs. That is what the confidence grade and the "claimed by another record" flag are for — nothing here should be actioned without opening the candidate.
No creation-time matching yet. The bulk list does not include creation timestamps, so time correlation is per-candidate, on demand. If you want time-window matching as a primary signal rather than a confirmation, that needs a server show per candidate — fine for tens, not for thousands.
Point in time. A VM mid-build will look unlinked. Two scans a few minutes apart separate "still settling" from "actually stuck".
ca2 is currently unscannable — an environment problem, not a scan problem.

5.2 KiB Raw Blame History