# How the linkage scan works

## The problem it solves

Infrahub and OpenStack are two databases that are supposed to agree about which
VMs exist. The link between them is a single field: `openstack_id` on the Infrahub
record.

When a VM is created, roughly: Infrahub writes a record → asks OpenStack to build
a server → OpenStack returns a UUID → Infrahub stores that UUID. If the last step
fails, you get a record stuck in `CREATING` or `ERROR` with no `openstack_id`,
while the server itself is running perfectly.

On an alert dashboard that is indistinguishable from a genuine build failure. They
need opposite responses:

| | Genuine failure | Linkage failure |
|---|---|---|
| Server exists? | no | **yes, and running** |
| Fix | customer recreates | repair the Infrahub record |
| Billing | nothing to bill | **running, billing nobody** |
| Customer sees | "my VM failed" | "my VM failed" |

Tell a customer to recreate a VM that is actually running and you have doubled
their spend on a machine they think is broken.

## The scan, step by step

### 1. Pull everything OpenStack has

For each region, one call:

```
openstack server list --all-projects --long -f json
```

~2,500 rows for ca1, ~500 for no1, ~80 for us1. Each row gives ID, name, status,
task state, host, project. About 45 seconds per region, so ~2.5 minutes total —
which is why it is a button, not something that runs on page load.

Two indexes are built: **by UUID** and **by lowercased name**.

### 2. Pull everything Infrahub has

Free — it is already in the `Resources` metric snapshot the queue uses (~4,300
records, one Prometheus query). Each carries `openstack_id`, `instance_name`,
`status`, `region`, `organization`, and GPU count.

### 3. Only judge regions that actually answered

If a region's listing failed, every VM in it would look "missing from OpenStack".
So records in unscanned regions are **excluded entirely** and counted separately.

This is not hypothetical. On the first real run `ca2` failed
(`Could not find versioned identity endpoints`) and produced **317 false
positives**. With the filter, the same scan returns **6**.

### 4. Find Infrahub records with a broken link

For each Infrahub record whose status is one where a live server is expected —
`ERROR`, `CREATING`, `BUILD`, `ACTIVE`, `REBOOTING`, `RESTORING` — flag it if:

- it has **no** `openstack_id`, or
- it has one that **is not in the OpenStack index**

`HIBERNATED` is deliberately excluded: a shelved instance legitimately has no
running server, and including it would drown the result (2,400 records).

### 5. Pair them up by name

For each flagged record, look for OpenStack servers with the **same name**. In
Hyperstack the OpenStack server name matches the Infrahub instance name, which
makes this a strong signal.

Confidence is then graded:

| Confidence | Meaning |
|---|---|
| **high** | A same-named server exists, **no other Infrahub record claims it**, and it is not itself in ERROR. This is a linkage failure — the server is real and orphaned. |
| **medium** | A same-named server exists but another Infrahub record already owns that UUID, or it is SHELVED_OFFLOADED. Could be a name collision; needs a human. |
| **none** | No same-named server. The record's server genuinely does not exist — a real failure, or it was deleted. |

**Details** on a row runs `server show` for the candidate to fetch its creation
time, launch time and fault, so you can confirm the timing lines up with when the
Infrahub record was created.

### 6. The reverse: servers nobody claims

Every OpenStack server whose UUID appears in **no** Infrahub record. These are
running, consuming GPUs, and billing nobody. The scan also notes whether the
*name* is known to Infrahub, which separates "record exists but the link is
broken" from "completely unknown".

This is what `:pirate_flag:Suspected Orphan VM` is supposed to catch — but that
rule is built on `openstack_nova_server_status`, which currently returns zero
series, so it cannot fire at all. Until that exporter is fixed, this scan is the
only thing looking.

## What it found on the first live run

- **`daring-rutherford`** — Infrahub `CREATING`, no `openstack_id` recorded;
  OpenStack had an **ACTIVE** server of exactly that name in ca1 that no other
  record claimed. Textbook linkage failure. (It had linked itself by the next
  scan — caught mid-flight.)
- **177 OpenStack servers** with no Infrahub record at all.
- **ca2 unreachable**, so that region is excluded and reported rather than
  silently producing garbage.

## Known limits

- **Name matching is a heuristic.** Two VMs can share a name across orgs. That is
  what the confidence grade and the "claimed by another record" flag are for —
  nothing here should be actioned without opening the candidate.
- **No creation-time matching yet.** The bulk list does not include creation
  timestamps, so time correlation is per-candidate, on demand. If you want
  time-window matching as a primary signal rather than a confirmation, that needs
  a `server show` per candidate — fine for tens, not for thousands.
- **Point in time.** A VM mid-build will look unlinked. Two scans a few minutes
  apart separate "still settling" from "actually stuck".
- **ca2 is currently unscannable** — an environment problem, not a scan problem.