commit a039e0b5fd4b9a7914567604d63c4c42fff25679 Author: Parham Monfared Date: Thu Aug 6 06:48:34 2026 +0100 CX Triage: alert diagnosis over the CX-Tools collectors Read-only triage for the Infrahub error alerts. Pulls the Prometheus alert queue, re-checks each alert's condition against live state to separate real work from noise, diagnoses it using the CX runbooks, and drafts the customer comms with contacts resolved from Infrahub. Findings from validating against production: - "Suspected Rogue VM" fires on spare GPU capacity, not rogue VMs: In_Use_Gpus equals the physical count on 71 of 75 firing hosts, so the rule reduces to "this host has a free GPU". Verified against OpenStack on 10 hosts. - "Exists in Infrahub but does not exist in OpenStack" matches every VM because openstack_nova_server_status returns no series; excluded as a rule defect. - Prometheus activeAt is reset several times a day by dips in the Resources metric, so alert ages are recovered from ALERTS history instead. Takes ~2,650 firing alerts down to ~20 that need a decision. Co-Authored-By: Claude Opus 5 diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..1140849 --- /dev/null +++ b/.gitignore @@ -0,0 +1,15 @@ +__pycache__/ +*.py[cod] +.venv/ +venv/ +node_modules/ +dist/ +build/ +*.egg-info/ +.env +.env.* +!.env.example +*.db +*.sqlite3 +data/ +.DS_Store diff --git a/INTEGRATIONS.md b/INTEGRATIONS.md new file mode 100644 index 0000000..a03f02a --- /dev/null +++ b/INTEGRATIONS.md @@ -0,0 +1,89 @@ +# Wiring Zendesk and Jira + +"Not wired" meant: the app **builds** the full API payload and shows it to you, +but the code path that would POST it to Zendesk deliberately returns an error. +Nothing can reach a customer today, even by accident. Here is what is needed to +change that. + +--- + +## What I need from you + +### Zendesk + +| Thing | Where it comes from | Example | +|---|---|---| +| Subdomain | your Zendesk URL `https://.zendesk.com` | `nexgencloud` | +| API email | the agent account tickets are created as | `cx-bot@nexgencloud.com` | +| API token | Zendesk **Admin Center → Apps and integrations → APIs → Zendesk API → Add API token** | 40-char string | + +Plus four decisions: + +1. **Which agent account should own these tickets?** A dedicated `cx-triage` agent + is better than a person's account — the audit trail stays clear. +2. **Public reply or internal note on first send?** The runbook wording is written + for the customer, so public — but confirm. +3. **Requester matching.** Infrahub gives us the owner's name and email. If they + are not already a Zendesk user, should we create them, or fail and ask a human? +4. **Do you want a tag convention** beyond the `cx-triage`, `alert-`, + `region-` tags I currently set? + +### Jira + +| Thing | Where it comes from | Example | +|---|---|---| +| Base URL | your Atlassian site | `https://nexgencloud.atlassian.net` | +| API email | Atlassian account | `cx-bot@nexgencloud.com` | +| API token | | token string | +| Project key | the Infrastructure project | `INFRA`? | +| Issue type | must exist in that project | `Task`? `Bug`? | + +Confirm the project key and issue type — I guessed `INFRA` / `Task`. + +### Where to put the credentials + +**Don't paste them to me.** Two options: + +- **Preferred — 1Password.** Create two items in the `Employee` vault, e.g. + `Zendesk API (CX Triage)` and `Jira API (CX Triage)`, each with the token in the + `password` field and the email in a `username` field. I extend the existing + CX-Tools secrets loader to read them, exactly like the Infrahub key. Nothing + touches disk. +- **Quick and dirty** — environment variables in your shell before launching: + `CX_ZENDESK_SUBDOMAIN`, `CX_ZENDESK_EMAIL`, `CX_ZENDESK_TOKEN`, + `CX_JIRA_BASE`, `CX_JIRA_EMAIL`, `CX_JIRA_TOKEN`, `CX_JIRA_PROJECT`. + Fine for a trial, worse for a shared tool. + +--- + +## What I build once I have that + +1. **Delivery** — replace the refusal in `App.send_zendesk` with a real POST, and + add the same for Jira. +2. **Search before create** — `GET /api/v2/search?query=external_id:cx-triage-` + so re-diagnosing an alert comments on the existing ticket rather than opening a + second one. Same for Jira via a `cx-triage-` label. +3. **A send gate** — the server refuses to send unless started with `--allow-send`. + A demo instance then physically cannot email a customer, no matter what is + clicked. Default stays off. +4. **Audit** — every send appended to `vmc-audit.log` with who, what, and the + resulting ticket URL, and the URL shown back on the case. +5. **Rate limiting** — Zendesk allows 700 req/min; a simple per-minute cap plus a + refusal to send more than N tickets in one session, so a bad loop cannot mail + a hundred customers. + +## Guard rails that stay regardless + +- Every send needs a human click plus a confirm naming the recipient. +- No auto-send, ever — a verdict never triggers an email on its own. +- No bulk send in v1. The 13 overdue ERROR alerts are tempting, but one wrong + template across 13 customers is a bad first outing. +- The message stays editable before sending. +- Deleting, shelving and InfraInsight edits remain copy-a-command. The read-only + guarantee is what makes this safe against production. + +## Suggested first run + +Point it at a **Zendesk sandbox** first, or send the first real ticket to your own +address by editing the To field. Once one round-trip looks right in Zendesk, turn +it on for real. diff --git a/LINKAGE.md b/LINKAGE.md new file mode 100644 index 0000000..18368ef --- /dev/null +++ b/LINKAGE.md @@ -0,0 +1,120 @@ +# How the linkage scan works + +## The problem it solves + +Infrahub and OpenStack are two databases that are supposed to agree about which +VMs exist. The link between them is a single field: `openstack_id` on the Infrahub +record. + +When a VM is created, roughly: Infrahub writes a record → asks OpenStack to build +a server → OpenStack returns a UUID → Infrahub stores that UUID. If the last step +fails, you get a record stuck in `CREATING` or `ERROR` with no `openstack_id`, +while the server itself is running perfectly. + +On an alert dashboard that is indistinguishable from a genuine build failure. They +need opposite responses: + +| | Genuine failure | Linkage failure | +|---|---|---| +| Server exists? | no | **yes, and running** | +| Fix | customer recreates | repair the Infrahub record | +| Billing | nothing to bill | **running, billing nobody** | +| Customer sees | "my VM failed" | "my VM failed" | + +Tell a customer to recreate a VM that is actually running and you have doubled +their spend on a machine they think is broken. + +## The scan, step by step + +### 1. Pull everything OpenStack has + +For each region, one call: + +``` +openstack server list --all-projects --long -f json +``` + +~2,500 rows for ca1, ~500 for no1, ~80 for us1. Each row gives ID, name, status, +task state, host, project. About 45 seconds per region, so ~2.5 minutes total — +which is why it is a button, not something that runs on page load. + +Two indexes are built: **by UUID** and **by lowercased name**. + +### 2. Pull everything Infrahub has + +Free — it is already in the `Resources` metric snapshot the queue uses (~4,300 +records, one Prometheus query). Each carries `openstack_id`, `instance_name`, +`status`, `region`, `organization`, and GPU count. + +### 3. Only judge regions that actually answered + +If a region's listing failed, every VM in it would look "missing from OpenStack". +So records in unscanned regions are **excluded entirely** and counted separately. + +This is not hypothetical. On the first real run `ca2` failed +(`Could not find versioned identity endpoints`) and produced **317 false +positives**. With the filter, the same scan returns **6**. + +### 4. Find Infrahub records with a broken link + +For each Infrahub record whose status is one where a live server is expected — +`ERROR`, `CREATING`, `BUILD`, `ACTIVE`, `REBOOTING`, `RESTORING` — flag it if: + +- it has **no** `openstack_id`, or +- it has one that **is not in the OpenStack index** + +`HIBERNATED` is deliberately excluded: a shelved instance legitimately has no +running server, and including it would drown the result (2,400 records). + +### 5. Pair them up by name + +For each flagged record, look for OpenStack servers with the **same name**. In +Hyperstack the OpenStack server name matches the Infrahub instance name, which +makes this a strong signal. + +Confidence is then graded: + +| Confidence | Meaning | +|---|---| +| **high** | A same-named server exists, **no other Infrahub record claims it**, and it is not itself in ERROR. This is a linkage failure — the server is real and orphaned. | +| **medium** | A same-named server exists but another Infrahub record already owns that UUID, or it is SHELVED_OFFLOADED. Could be a name collision; needs a human. | +| **none** | No same-named server. The record's server genuinely does not exist — a real failure, or it was deleted. | + +**Details** on a row runs `server show` for the candidate to fetch its creation +time, launch time and fault, so you can confirm the timing lines up with when the +Infrahub record was created. + +### 6. The reverse: servers nobody claims + +Every OpenStack server whose UUID appears in **no** Infrahub record. These are +running, consuming GPUs, and billing nobody. The scan also notes whether the +*name* is known to Infrahub, which separates "record exists but the link is +broken" from "completely unknown". + +This is what `:pirate_flag:Suspected Orphan VM` is supposed to catch — but that +rule is built on `openstack_nova_server_status`, which currently returns zero +series, so it cannot fire at all. Until that exporter is fixed, this scan is the +only thing looking. + +## What it found on the first live run + +- **`daring-rutherford`** — Infrahub `CREATING`, no `openstack_id` recorded; + OpenStack had an **ACTIVE** server of exactly that name in ca1 that no other + record claimed. Textbook linkage failure. (It had linked itself by the next + scan — caught mid-flight.) +- **177 OpenStack servers** with no Infrahub record at all. +- **ca2 unreachable**, so that region is excluded and reported rather than + silently producing garbage. + +## Known limits + +- **Name matching is a heuristic.** Two VMs can share a name across orgs. That is + what the confidence grade and the "claimed by another record" flag are for — + nothing here should be actioned without opening the candidate. +- **No creation-time matching yet.** The bulk list does not include creation + timestamps, so time correlation is per-candidate, on demand. If you want + time-window matching as a primary signal rather than a confirmation, that needs + a `server show` per candidate — fine for tens, not for thousands. +- **Point in time.** A VM mid-build will look unlinked. Two scans a few minutes + apart separate "still settling" from "actually stuck". +- **ca2 is currently unscannable** — an environment problem, not a scan problem. diff --git a/PLAN.md b/PLAN.md new file mode 100644 index 0000000..ad8a4ee --- /dev/null +++ b/PLAN.md @@ -0,0 +1,130 @@ +# From "here's what to do" to "do it" + +## 1. Validation against live data + +Every recommendation was checked against OpenStack directly, not against the +app's own view. Results: + +| Case | App said | Ground truth | Verdict | +|---|---|---|---| +| **Rogue VM `CA1-ESC8-057`** | "No mismatch — go check InfraInsight for HIBERNATED VMs carrying a host" | OpenStack 7 GPUs across 5 VMs; Infrahub 7 GPUs across the same 5 VMs. Perfect agreement. No hibernated VMs on the host at all. | **Wrong** | +| **Rogue VM, 10 hosts sampled** | 50 alerts "needs action" | OpenStack and Infrahub agreed **exactly** on all 10 | **Wrong** | +| `SHUTOFF` sn56-week6-transfer-e1 | Billing notice to `shettyatulya@gmail.com` | OpenStack `SHUTOFF`, no fault | Correct | +| `DELETING` hs-gaussian-splatting | Intent verified from events; delete in OpenStack | OpenStack `ACTIVE`, delete requested 17:00 | Correct | +| `DELETING` c3-pool-1785947188798 | Intent verified; delete in OpenStack | OpenStack `ERROR`, host `None`, "No valid host was found" | Correct action, **incomplete** — it never built, so the customer is owed the creation-failure notice too | +| `ERROR` hyperstack-minion-lydhjev | `no_valid_host` fault → never ACTIVE → insufficient-stock template → Lars Vagnes | Fault is exactly "No valid host was found. There are not enough hosts available."; host `None` | Correct — but was **hidden as "chronic"** | + +### The Rogue VM rule is broken at source + +```promql +sum by (instance) (In_Use_Gpus) + - sum by (instance) (Resources{status=~"ACTIVE|SHUTOFF|PRE_ACTIVE"}) >= 1 +``` + +`In_Use_Gpus` equals `Total_Gpus` — the **physical** GPU count — on **71 of the 75** +firing hosts. So the expression reduces to *physical GPUs minus allocated GPUs*, +i.e. **"this host has at least one free GPU."** That is spare capacity, not a +rogue VM. + +`CA1-ESC8-057` is the clean example: 8 physical, 7 allocated across 5 VMs that +both systems agree on, so it fires with a gap of 1. + +Two consequences: +- 65 of the 72 rogue-VM alerts are now classified **invalid — alert rule defect** + and screened out, with the reason stated. +- The 2 that survive are genuine: `CA1-ESC812-252` (In_Use 2, one 1-GPU VM) and + `CA1-ESC812-289` (In_Use 7, 5 GPUs allocated). Both now read *"N GPU(s) in use + belong to no instance on either side"* and route to Infrastructure, because no + CX action can fix a leaked hypervisor allocation. + +**This needs fixing in the Prometheus rule, not just filtered here.** Whoever owns +`infrahub-rules.yml` should either repair `In_Use_Gpus` or rewrite the expression. + +### "Chronic" was hiding overdue customer contact + +13 ERROR alerts have held for 6–7 days. The ERROR runbook says to contact the +customer if a stock-failure instance is not deleted **within a day**. Filing those +as "chronic, probably already ticketed" was wrong — they are overdue. + +New `overdue` verdict: for kinds with a runbook time commitment, age makes a case +*more* urgent and it can never be demoted to chronic. + +### Net effect + +| | before validation | after | +|---|---|---| +| To action | 54 | **19** | +| of which overdue customer contact | 0 (hidden) | **13** | +| Rogue VM noise | 50 "needs action" | 65 flagged as a rule defect | + +--- + +## 2. Integration plan + +The principle: the app already knows *who* to contact and *what* to say. Wiring +delivery turns a 5-minute copy-paste into one reviewed click — without ever +sending on its own. + +### Phase 1 — Zendesk (the main win) + +`integrations.build_zendesk()` already produces a complete `POST /api/v2/tickets` +body: requester resolved from Infrahub owners, subject, the approved runbook +wording with placeholders filled, priority derived from the verdict, tags, and +`external_id = cx-triage-` for idempotency. + +To finish it: + +1. **Credentials** — `CX_ZENDESK_SUBDOMAIN`, `CX_ZENDESK_EMAIL`, `CX_ZENDESK_TOKEN` + (API token, Basic auth as `email/token:token`). Read from 1Password via the + existing CX-Tools loader rather than env vars, so nothing lands on disk. +2. **Search before create** — `GET /api/v2/search?query=external_id:` so a + re-diagnosed alert updates the existing ticket instead of opening a duplicate. +3. **Send** — implement `App.send_zendesk`, which currently refuses by design. +4. **Guard rails** (all already scaffolded in the UI): + - two-step confirm naming the recipient + - never auto-send; no bulk send in v1 + - an `--allow-send` startup flag, so a demo instance physically cannot email + - append the ticket URL back onto the case and log it to `vmc-audit.log` +5. **Requester matching** — Infrahub gives owner name + email; Zendesk may already + have that user. Search by email, fall back to creating the requester inline. + +### Phase 2 — Jira + +`integrations.build_jira()` produces the `POST /rest/api/3/issue` body with the +evidence block already assembled. Needs: project key confirmation (`INFRA`?), +issue type, and the same search-before-create against a `cx-triage-` label. +The GPU-leak escalations above are the immediate use case. + +### Phase 3 — closing the loop + +- **Slack** — react to the alert in `#infrahub-errors` and thread the findings, + which the runbook asks for manually today. +- **Case state** — persist handled/snoozed/ticketed per fingerprint (SQLite) so + the queue reflects work already done and survives a restart. +- **Bulk actions** — the 13 overdue ERROR alerts are one org-grouped mail-merge; + worth doing only once single-send is trusted. + +### What stays manual, deliberately + +Deleting, shelving and InfraInsight edits stay copy-a-command. The read-only +guarantee is what makes this safe to run against production, and the destructive +steps are exactly where a wrong verdict would be expensive. + +--- + +## 3. UI + +Live at `http://127.0.0.1:8765/` (previous version kept at `/classic`). + +- **Left rail** — one line per case: subject, org, region, how long the condition + has actually held. Grouped and collapsible. Verdict filter chips across the top. +- **Case card** — verdict as a sentence, then a *picture* of the problem: + - state alerts: `Infrahub says X` ≠ `OpenStack says Y` + - GPU cases: a segmented allocation bar (green allocated / red unaccounted) + - duplicate IPs: one row per claimant +- **"Do this"** — the customer's name and address, then the action buttons. + Nothing else competes with them. +- **Everything else collapsed** — Caveats, Why, Runbook steps, Raw evidence. +- **Zendesk drawer** — recipient, subject, editable body, priority, tags, and a + Send button that is disabled and labelled *"Preview only. Nothing will be sent."* + until credentials exist and sending is explicitly enabled. diff --git a/README.md b/README.md new file mode 100644 index 0000000..23c4f3c --- /dev/null +++ b/README.md @@ -0,0 +1,238 @@ +# CX Triage + +A small local webapp that takes the Infrahub error alerts out of Prometheus, +diagnoses each one using the **unmodified** CX-Tools (`vmc`) collectors, tells you +what the runbook says to do next, and — when the next step is contacting the +customer — shows the approved wording alongside the customer's contact details. + +**It is read-only.** It queries Infrahub, OpenStack, InfraInsight and Prometheus. +It never changes platform state, never deletes or shelves anything, and never +sends a message. Every action it identifies is presented for a human to perform. + +## Separating noise from real work + +Thousands of alerts fire; only a handful are work. Before anything is shown, each +alert's condition is **re-checked against current state**, and the verdict is +displayed with its reason: + +| Verdict | Meaning | In the queue? | +|---|---|---| +| **needs action** | The condition still holds | yes | +| **needs action (unverified)** | Couldn't be re-checked — never hidden on a guess | yes | +| **already resolved** | Infrahub has moved on / the IP is no longer duplicated / the GPU gap has closed | hidden | +| **not yet firing** | Prometheus still has it pending | hidden | +| **chronic** | Still true, but firing over 3 days — already ticketed, not new work | hidden | +| **low impact** | Still true, but owned by an internal org or a platform-owned node | hidden | + +Screening only ever demotes an alert on **positive evidence**; anything it can't +settle stays in the queue. Hidden alerts are one checkbox away, and any of them +can be force-diagnosed with **Diagnose anyway**. + +### Alert ages are recovered, not taken from Prometheus + +Prometheus' own `activeAt` is unreliable here. The Infrahub `Resources` metric +drops most of its series for ~5 minutes several times a day (4 dips in the last +24h observed; one took it from ~4,370 series to 1,359). Every alert alive during +a dip resolves and re-fires, so `activeAt` resets on all of them at once — which +is why the Prometheus UI shows dozens of unrelated alerts with the *same* age. + +So the app walks the `ALERTS` series backwards over 7 days instead, bridging gaps +under 45 minutes, and reports how long each condition has **actually** held. In +practice this is the difference between "40 alerts all 7h old" and "11 that are +genuinely new, 33 that have been true for days". Both numbers are shown: the +recovered duration, with Prometheus' value in a tooltip when they disagree. + +The app detects these dips and warns about them, since they also mean any alert +with a long `for:` may never reach firing state. + +The re-check is cheap on purpose: it reads the same Prometheus series the rules +are built from — `Resources`, `In_Use_Gpus`, `Total_Gpus` — in one bulk snapshot +for the entire queue, rather than an Infrahub and OpenStack call per alert. Only +alerts you actually open cost a CX-Tools query. + +On live data this takes **~2,670 firing alerts down to ~11** that need a decision. + +Caches are warmed at startup (~20s, mostly the 7-day history read), so page loads +are instant afterwards. They refresh on a 30s/60s/5min cadence. + +**Excluded outright:** `Exists in Infrahub but does not exist in OpenStack`. It is +built as `Resources unless on(openstack_id) openstack_nova_server_status`, and +that second metric is currently returning **zero series** — so nothing gets +excluded by the `unless` and every Infrahub VM alerts. It is a broken exporter, +not a queue of work. The app detects this class of failure and shows a banner, +because the same gap also means `Suspected Orphan VM` cannot fire at all. + +## Two tabs + +- **CX runbooks** — the alerts below, grouped into collapsible sections in + working order (rogue VMs, duplicate IPs, total GPUs, hibernating, creating, + shutoff, deleting, error), each showing how long it has been firing, **newest + first** so long-running alerts sink to the bottom. +- **Infrastructure** — everything else, so it stays out of the triage queue: + node-exporter host alerts in their own section, then Ceph, MySQL, Galera, + OpenStack services, blackbox. Listed and counted, not diagnosed. Routing is + keyed off the **rule file**, not the alert name, because two different rule + files both use the group name "Imported Rules". + +## What it covers + +One runbook per alert type, from *Infrahub Errors Remediation*: + +| Alert | Priority | What the app works out for you | +|---|---|---| +| Instance in ERROR state | LOW–HIGH | Matches the fault against the runbook fault table; decides whether the VM was ever ACTIVE (which changes both the urgency and the comms template); for the NUMA/PCI fault it sums GPUs on the host to check whether the host is full before you escalate | +| Instance in DELETING state | LOW | Confirms the delete request in Infrahub events, and whether the OpenStack server is still there or already gone | +| Instance in SHUTOFF state | LOW | Confirms SHUTOFF and drafts the billing-awareness note | +| Instance in HIBERNATING state | HIGH | Runs the host signals (Nova state/status, disabled reason, OVS liveness) and escalates when they're bad | +| Instance in CREATING state | MEDIUM | Determines whether the VM ever got an OpenStack ID | +| Instance in RESTORING state | HIGH | Host signals plus the most recent *failed* OpenStack event to escalate | +| Instance in REBOOTING state | HIGH | Confirms `InstanceRebootRequest` and the expected `HARD_REBOOT` state | +| Instance in BUILD state | MEDIUM | Distinguishes "large flavor, still transient" from "stuck, escalate" | +| Suspected Rogue VM | HIGH | Quantifies the per-host GPU accounting gap the rule actually fires on, then reconciles every instance on the host and maps each mismatch to its row in the Mismatch Remediation table; tempest instances are ignored | +| Duplicated IPs | HIGH | Classifies each claimant of the IP as Scenario #1 / #2 / rightful owner, and pulls the cross-environment claimant list from `Resources{floating_ip=...}` | +| Problem with Total GPUs | HIGH | Lists which customers are on the affected host | +| Suspected Orphan VM | HIGH | Same host reconciliation as Rogue VM (cannot currently fire — see the exporter note above) | +| Openstack status=X / Infrahub status!=X | HIGH | Single-VM mismatch, taken through the same remediation table | + +### A note on Suspected Rogue VM + +The rule is not a status comparison — it is +`sum by(instance)(In_Use_Gpus) - sum by(instance)(Resources{status=~"ACTIVE|SHUTOFF|PRE_ACTIVE"}) >= 1`, +a **per-host GPU accounting gap**. Two different faults produce that gap: + +1. instances running on the host that Infrahub has no record of (a true rogue VM), or +2. Infrahub VMs that are ACTIVE but have **no host recorded**, so they are never + counted against the host that is actually running them. + +Prometheus cannot tell these apart, so the app states both and lets the +per-instance host reconciliation settle it — a genuine rogue VM shows up as +`Infrahub Missing`. It also reports how many unattributed ACTIVE VMs exist +platform-wide, because that number alone can be large enough to explain the gaps +without any rogue VM existing. + +## Requirements + +Whatever `vmc` already needs, plus nothing: + +- The CX-Tools checkout (`cxlib/` + `vmc`), unmodified +- Python 3 (standard library only — no pip install) +- Docker with the `ca1-osc` / `ca2-osc` / `us1-osc` / `no1-osc` containers running +- A signed-in 1Password CLI session + +## Run + +```bash +op signin +``` + +```bash +./cx-triage +``` + +It opens . Bound to localhost only. + +```bash +./cx-triage --check +``` + +Runs preflight (CX-Tools located, credentials loaded, containers up, Prometheus +reachable) and exits. + +Useful flags: `--port`, `--prometheus `, `--no-open`. + +If CX-Tools isn't found automatically, point at it: + +```bash +CX_TOOLS_PATH=~/scripts/CX-Tools ./cx-triage +``` + +## How it reaches things + +- **CX-Tools** is imported as a library. `cxbridge.py` calls `collect_vm`, + `collect_host` and the query helpers — the same code paths as `vmc --json` — + and guards every OpenStack subcommand against a read-only allowlist, so a bug + here cannot mutate an instance. +- **Prometheus** at `10.11.254.250:9090` is on the internal network, which the + laptop has no route to (10.11.* leaves via the default gateway). So queries are + relayed `docker exec ca1-osc curl ...` — the same trick CX-Tools uses for + OpenStack. A direct HTTP transport is tried first, so this still works from a + host that does have a route. Override the relay with `CX_PROMETHEUS_RELAY`. + +## Working an alert + +1. Pick an alert from the queue. Or paste an `ALERTS{...}` line or a Prometheus + graph URL into the box. +2. **What the platforms say** — the reconciled Infrahub/OpenStack/InfraInsight + facts, with mismatches called out in red. +3. **Next steps** — the remaining runbook steps, each tagged with its owner (CX / + Infrastructure team / DevOps). Steps the app has already verified are ticked + off, so you can see what's left rather than re-deriving it. +4. **Suggested customer comms** — only when the runbook calls for it. Verbatim + approved wording with the instance name (and floating IP) substituted, above + the organization and owner contacts CX-Tools resolved. Copy it and send it + from HubSpot. +5. **Evidence** — the raw alert labels and the raw CX-Tools output, for pasting + into a Slack thread or a Jira ticket. + +## Caveats + +These are real limits, not bugs: + +- **Production Infrahub only.** CX-Tools queries production. When a Duplicated + IPs or Rogue VM alert points at a PreProd/Staging record, the app says so and + tells you to check the other environments — it can't query them. The + Prometheus `Resources` series does span environments, which is why the + Duplicated IPs view uses it. +- **No InfraInsight SQL.** The DELETING runbook identifies the requesting *user* + via a SQL query. The app confirms the delete request from Infrahub events but + cannot name the requester, and says so when it matters. +- **Host Health Checks is partial.** The app reports the host signals CX-Tools + exposes (Nova state/status, disabled reason, OVS liveness/heartbeat, uptime, + aggregates). The rest of that guide — disk, dmesg, GPU checks — is still + manual, and the app says which part it did. +- **Fault table coverage.** ERROR faults outside the runbook's table produce an + explicit "not in the table, escalate to a peer" verdict rather than a guess. +- **Chronic/low-impact thresholds are judgement calls**, not runbook rules: + 3 days for chronic (`CHRONIC_DAYS` in `screening.py`), and "internal" means an + `@nexgencloud.com` owner. Adjust to taste. +- **Recovered ages are bounded by a 7-day window** (`TrueAgeIndex.WINDOW_DAYS`). + Anything older shows as `7d+`. +- **"Chronic" does not mean "ignore".** It means the condition has been true for + days, so it is not *new* work. Several ERROR alerts are 6–7 days old; if those + have not actually been ticketed, they are a backlog, not noise. + +## Layout + +```text +cx-triage entry point (preflight, then serve) +triagelib/ + cxbridge.py read-only adapter over cxlib + prometheus.py alert source, rule index, bulk state snapshot, relay + alerts.py normalization, classification, exclusions, grouping + screening.py noise-vs-real verdicts + runbooks.py the decision engine + comms.py customer comms templates, verbatim from Confluence + server.py HTTP API + background triage jobs + ui.py the single-page UI +tests/test_runbooks.py runbook decisions against fixture payloads +tests/test_screening.py screening, exclusion, tab routing, ordering +``` + +```bash +python3 tests/test_runbooks.py && python3 tests/test_screening.py +``` + +## Updating a runbook + +The decision logic is meant to be edited by whoever owns the runbook: + +- Fault table → `FAULT_TABLE` in `runbooks.py` +- Rogue VM mismatch table → `MISMATCH_TABLE` in `runbooks.py` +- Priority / ETTR → `KIND_META` in `alerts.py` +- Queue order → `FOCUS_ORDER` in `alerts.py` +- Alerts to suppress → `EXCLUDED_ALERTNAMES` in `alerts.py` +- Noise rules → `screening.py` (`CHRONIC_DAYS`, `_KIND_SCREENS`) +- Customer wording → `_TEMPLATES` in `comms.py` + +If a wording change lands in Confluence, change it in `comms.py` and nowhere +else. diff --git a/cx-triage b/cx-triage new file mode 100755 index 0000000..64627c1 --- /dev/null +++ b/cx-triage @@ -0,0 +1,79 @@ +#!/usr/bin/env python3 +"""CX Triage - diagnose Infrahub error alerts using the CX-Tools collectors. + +Read-only: this tool queries Infrahub, OpenStack, InfraInsight and Prometheus, +and suggests what to do. It never changes platform state and never sends +customer comms. +""" +from __future__ import annotations + +import argparse +import os +import sys +import webbrowser + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +from triagelib import VERSION, cxbridge # noqa: E402 +from triagelib.prometheus import DEFAULT_BASE, PrometheusClient, PrometheusError # noqa: E402 + + +def preflight(prometheus_base: str) -> bool: + ok = True + + print("Locating CX-Tools...") + try: + path = cxbridge.locate_cx_tools() + print(f" found: {path}") + except cxbridge.BridgeError as exc: + print(f" FAIL: {exc}", file=sys.stderr) + return False + + # Done in the foreground so any 1Password prompt reaches the terminal + # rather than a background HTTP worker. + print("Loading API credentials from 1Password (CX-Tools loader)...") + try: + cxbridge.bootstrap() + print(" credentials loaded") + except cxbridge.BridgeError as exc: + print(f" FAIL: {exc}", file=sys.stderr) + ok = False + + print(f"Checking Prometheus at {prometheus_base}...") + try: + print(f" reachable ({PrometheusClient(prometheus_base).describe_transport()})") + except PrometheusError as exc: + print(f" WARN: {exc}", file=sys.stderr) + print(" The app will still start; paste alerts manually until this is fixed.", file=sys.stderr) + + return ok + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("--host", default="127.0.0.1", help="bind address (default: localhost only)") + parser.add_argument("--port", type=int, default=8765, help="port (default: 8765)") + parser.add_argument("--prometheus", default=DEFAULT_BASE, help=f"Prometheus base URL (default: {DEFAULT_BASE})") + parser.add_argument("--no-open", action="store_true", help="do not open a browser on start") + parser.add_argument("--check", action="store_true", help="run preflight checks and exit") + parser.add_argument("--version", action="version", version=VERSION) + args = parser.parse_args() + + if not preflight(args.prometheus): + return 1 + if args.check: + print("preflight OK") + return 0 + + url = f"http://{args.host}:{args.port}" + if not args.no_open: + webbrowser.open(url) + + from triagelib.server import serve + + serve(host=args.host, port=args.port, prometheus_base=args.prometheus) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/test_runbooks.py b/tests/test_runbooks.py new file mode 100644 index 0000000..3124c2f --- /dev/null +++ b/tests/test_runbooks.py @@ -0,0 +1,268 @@ +"""Runbook decision tests: fixtures shaped like real CX-Tools collector output.""" +import sys, os +sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..")) + +from triagelib import alerts as A, cxbridge, runbooks + +# --- stub the CX-Tools boundary so the decision logic can be tested alone --- +class FakeCx: + def status_pair_ok(self, ih, os_, task): + ih, os_ = (ih or "").upper(), (os_ or "").upper() + if ih == os_: return True + return (ih, os_) in {("HIBERNATED","SHELVED_OFFLOADED"), ("REBOOTING","HARD_REBOOT")} + def mismatch_parts(self, item): + return str(item.get("check","Mismatch")), str(item.get("detail","")) + def normalize_empty(self, v): return "" if v in (None,"","None") else str(v) + def first_present(self, m, *keys, default=""): + for k in keys: + if isinstance(m, dict) and k in m: return m[k] + return default + def public_ip_from_server(self, s): return (s or {}).get("_public_ip","") + def json_safe(self, o): return o + def map_region(self, r): return {"CANADA-1":"ca1","US-1":"us1","NORWAY-1":"no1"}.get(r,"") + +STUB = {"host_health": {}, "gpu_census": {}, "failed_event": {}, "vm": {}, "host": {}} +cxbridge.cx = lambda: FakeCx() +cxbridge.host_health = lambda region, host: STUB["host_health"] +cxbridge.host_gpu_census = lambda region, host: STUB["gpu_census"] +cxbridge.failed_openstack_event = lambda region, osid, scan=5: STUB["failed_event"] +cxbridge.collect_vm = lambda t, **k: STUB["vm"] +cxbridge.collect_host = lambda h, **k: STUB["host"] +cxbridge.json_safe = lambda o: o + +HEALTHY = {"ok": True, "nova_state": "up", "nova_status": "enabled", "ovs_alive": True, + "ovs_state": "UP", "uptime": "17:54", "aggregates": "agg", "bad_signals": []} +SICK = {**HEALTHY, "nova_state": "down", "ovs_state": "DOWN", "ovs_alive": False, + "bad_signals": ["Nova state is down", "OVS state is DOWN"]} + +def vm(**over): + base = {"ok": True, "exit_code": 0, "mode": "vm", "infrahub_id": "123456", + "openstack_id": "9ec7a021-d741-484f-9387-4eaf8879fd77", "name": "test-vm", + "region": "ca1", "region_display": "CANADA-1", "ih_status": "ACTIVE", "os_status": "ACTIVE", + "task_state": "None", "host": "CA1-ESC8-040", "flavor": "n3-H100x8", "gpu_count": "8", + "floating_ip": "69.19.140.110", "created": "2026-07-01 10:00:00 UTC", "ssh_text": "reachable", + "ssh_raw": "reachable", "volumes_summary": "None", "openstack_fault": "None", + "faults": [], "ih_events": [], "ih_events_all": [], "mismatches": [], "warn_reasons": [], + "info_notes": [], "org_value": "8463 - Acme Corp", "owners": ["Lars "], + "server": {"status": "ACTIVE"}, "infrahub": {"floating_ip": "69.19.140.110"}} + base.update(over); return base + +def alert(name, **labels): + return A.from_labels({"alertname": name, **labels}) + +def run(name, labels, vmdata=None, hostdata=None, health=None, census=None, fevent=None, prom=None): + STUB.update({"vm": vmdata or {}, "host": hostdata or {}, "host_health": health or HEALTHY, + "gpu_census": census or {}, "failed_event": fevent or {}}) + return runbooks.diagnose(alert(name, **labels), prom) + +def check(label, cond, extra=""): + print(f" {'PASS' if cond else 'FAIL'} {label}" + (f" <- {extra}" if not cond and extra else "")) + return cond + +fails = 0 +def expect(label, cond, extra=""): + global fails + if not check(label, cond, extra): fails += 1 + +L_ERR = dict(openstack_id="9ec7a021-d741-484f-9387-4eaf8879fd77", region="CANADA-1", + instance_name="test-vm", organization="8463 - Acme Corp", status="ERROR") + +print("\n[1] ERROR - creation failed, never reached ACTIVE, insufficient stock") +d = run("Instance in ERROR state in :flag-ca:CA-1", L_ERR, + vm(host="N/A", ih_status="ERROR", os_status="ERROR", server={"status": "ERROR"}, + openstack_fault="No valid host was found. There are not enough hosts available")) +expect("matched no_valid_host fault", "scheduler could not place" in d.verdict.lower(), d.verdict) +expect("identified as never-ACTIVE", any("never placed on a host" in f.value for f in d.findings)) +expect("chose the insufficient-stock template", any(x.template_id == "error_never_active" for x in d.drafts), + [x.template_id for x in d.drafts]) +expect("mentions 7-day outreach window", any("7 days" in x.when for x in d.drafts)) +expect("escalates to Infrastructure", any(a.owner == runbooks.INFRA for a in d.actions)) +expect("contacts resolved", d.contacts.get("resolved")) + +print("\n[2] ERROR - was ACTIVE, stale LVM on host") +d = run("Instance in ERROR state in :flag-ca:CA-1", L_ERR, + vm(ih_status="ERROR", os_status="ERROR", + faults=[["500", "Build of instance aborted: Failed to remove volume(s): lvremove -f /dev/nova-vg/x_disk", "2026-07-29"]])) +expect("matched lvremove fault", "stale LVM" in d.verdict, d.verdict) +expect("identified as previously ACTIVE", any("hypervisor is recorded" in f.value for f in d.findings)) +expect("chose the was-ACTIVE template", any(x.template_id == "error_was_active" for x in d.drafts), + [x.template_id for x in d.drafts]) +expect("assessment flags possible customer data", "customer data" in d.assessment) + +print("\n[3] ERROR - NUMA/PCI fault, host is FULL (proves host fine)") +d = run("Instance in ERROR state in :flag-ca:CA-1", L_ERR, + vm(ih_status="ERROR", os_status="ERROR", + openstack_fault="Insufficient compute resources: Requested instance NUMA topology together with requested PCI devices cannot fit the given host NUMA topology; Claim pci failed."), + census={"ok": True, "total_gpus": 8, "instances": [{"name":"a"}]*4}) +expect("ran the GPU census", any("GPUs allocated on host" in f.label for f in d.findings)) +expect("concluded host is FULL", any("FULL" in f.detail for f in d.findings)) +expect("marked the capacity check as already done", any(a.status == "done" for a in d.actions)) + +print("\n[3b] same fault, host NOT full -> must escalate") +d = run("Instance in ERROR state in :flag-ca:CA-1", L_ERR, + vm(ih_status="ERROR", os_status="ERROR", + openstack_fault="Claim pci failed."), + census={"ok": True, "total_gpus": 4, "instances": [{"name":"a"}]*2}) +expect("raises a Jira for Infra", any("Jira" in a.text and a.owner == runbooks.INFRA for a in d.actions)) + +print("\n[4] SHUTOFF - billing notice only") +d = run("Instance in SHUTOFF state in :flag-ca:CA-1 for greater than 30 min", + dict(openstack_id="dc156762-3fa8-481d-89e0-12c2fb55e046", region="CANADA-1", + instance_name="energetic-galileo", organization="27358 - Zyra", status="SHUTOFF"), + vm(name="energetic-galileo", ih_status="SHUTOFF", os_status="SHUTOFF", server={"status":"SHUTOFF"})) +expect("verdict is customer-initiated", "customer-initiated" in d.verdict.lower(), d.verdict) +expect("used the shutoff snippet", any(x.template_id == "shutoff" for x in d.drafts)) +expect("substituted the VM name", any("energetic-galileo" in x.body for x in d.drafts)) +expect("no Infra escalation", not any(a.owner == runbooks.INFRA for a in d.actions)) + +print("\n[5] DELETING - server still in OpenStack, delete request found") +d = run("Instance in DELETING state in :flag-ca:CA-1 for greater than 30 min", + dict(openstack_id="13a855a4-c563-4350-9a8e-ffa9efa27d9e", region="CANADA-1", + instance_name="external-hs-h100", organization="10420 - Inceptions AI", status="DELETING"), + vm(ih_status="DELETING", os_status="ACTIVE", server={"status": "ACTIVE"}, + ih_events_all=[["2026-07-29 21:40:00", "InstanceDeleteRequest", "Delete Instance Request Sent."]])) +expect("confirmed customer intent from events", any(a.status == "done" and "intent" in a.text for a in d.actions)) +expect("tells CX to delete in OpenStack", any("Delete the server in OpenStack" in a.text for a in d.actions)) +expect("offers both the notice and the ticket-closing reply", + sorted(x.template_id for x in d.drafts) == ["deleting", "deleting_resolved"], + [x.template_id for x in d.drafts]) +expect("tells CX to close the Infrahub record too, not just the server", + any("InfraInsight" in a.text for a in d.actions), [a.text for a in d.actions]) + +print("\n[5c] DELETING - server exists but never reached a host") +d = run("Instance in DELETING state in :flag-ca:CA-1 for greater than 30 min", + dict(openstack_id="x", region="CANADA-1", status="DELETING"), + vm(ih_status="DELETING", os_status="ERROR", host="N/A", + server={"status": "ERROR"}, openstack_fault="No valid host was found.")) +expect("flags that the build never completed", any("build never completed" in f.value for f in d.findings)) +expect("still requires the InfraInsight close-out", any("InfraInsight" in a.text for a in d.actions)) + +print("\n[5b] DELETING - already gone from OpenStack, no delete event") +d = run("Instance in DELETING state in :flag-ca:CA-1 for greater than 30 min", + dict(openstack_id="x", region="CANADA-1", status="DELETING"), + vm(ih_status="DELETING", os_status="N/A", server={}, ih_events_all=[])) +expect("notes the server is already gone", any("already gone" in f.value for f in d.findings)) +expect("routes attribution to DevOps", any(a.owner == runbooks.DEVOPS for a in d.actions)) + +print("\n[6] CREATING - never got an OpenStack ID") +d = run("Instance in CREATING state in :flag-ca:CA-1 for greater than 30min", + dict(openstack_id="None", region="CANADA-1", instance_name="vm-28070520-5d1f2", + organization="5574 - Nexgen", status="CREATING"), + vm(openstack_id="N/A", ih_status="CREATING", os_status="N/A", server={}, host="N/A")) +expect("verdict says never got an OpenStack ID", "never got an OpenStack ID" in d.verdict, d.verdict) +expect("used the creating template", any(x.template_id == "creating" for x in d.drafts)) +expect("instructs deletion", any("Delete the stuck instance" in a.text for a in d.actions)) + +print("\n[7] HIBERNATING - sick host") +d = run("Instance in HIBERNATING state in :flag-ca:CA-1 for greater than 120min", + dict(openstack_id="28171e6f", region="CANADA-1", instance="CA1-ESC812-211", + instance_name="apt25-prod", status="HIBERNATING"), + vm(ih_status="HIBERNATING", os_status="ACTIVE"), health=SICK) +expect("verdict blames the host", "host problem" in d.verdict, d.verdict) +expect("escalates to Infra with the bad signals", any(a.owner == runbooks.INFRA and "OVS" in a.text for a in d.actions)) +expect("still drives the shelve", any("shelve" in a.text.lower() for a in d.actions)) + +print("\n[8] Suspected Rogue VM - host with two different mismatches") +host_result = {"ok": True, "mode": "host", "host": "CA1-ESC8-068", "region": "ca1", "server_count": 3, + "hypervisor": {"state": "up", "status": "enabled"}, "ovs": {"alive": True, "state": "UP"}, + "instances": [ + {"idx": 1, "name": "vm-hib-shutoff", "infrahub_id": "1", "openstack_id": "a", "ih_status": "HIBERNATED", + "os_status": "SHUTOFF", "host": "CA1-ESC8-068", "mismatches": [{"check": "State", "detail": "IH HIBERNATED vs OS SHUTOFF"}], + "warn_reasons": ["mismatch detected"], "org_value": "1 - A", "owners": ["a@x.com"]}, + {"idx": 2, "name": "vm-hib-active", "infrahub_id": "2", "openstack_id": "b", "ih_status": "HIBERNATED", + "os_status": "ACTIVE", "host": "CA1-ESC8-068", "mismatches": [{"check": "State", "detail": "IH HIBERNATED vs OS ACTIVE"}], + "warn_reasons": ["mismatch detected"], "org_value": "2 - B", "owners": ["b@x.com"]}, + {"idx": 3, "name": "tempest-thing", "tempest": True, "ih_status": "N/A", "os_status": "ACTIVE", + "mismatches": [{"check": "Infrahub Missing", "detail": "not in Infrahub"}], "warn_reasons": []}, + ]} +d = run(":ninja:Suspected Rogue VM", dict(instance="CA1-ESC8-068"), hostdata=host_result) +expect("counted 2 of 3 as mismatched", "2 of 3" in d.verdict, d.verdict) +expect("ignored the tempest instance", d.evidence.get("ignored_tempest") == 1) +expect("HIBERNATED/SHUTOFF -> Windmill stale-image cleanup", any("Windmill" in a.text for a in d.actions)) +expect("HIBERNATED/ACTIVE -> sync-error comms", any(x.template_id == "sync_state" for x in d.drafts)) +expect("actions are scoped per instance", any(a.text.startswith("[vm-hib-active]") for a in d.actions)) + +print("\n[9] Suspected Rogue VM - clean host") +d = run(":ninja:Suspected Rogue VM", dict(instance="CA1-ESC8-068"), + hostdata={**host_result, "instances": [{"idx":1,"name":"ok","ih_status":"ACTIVE","os_status":"ACTIVE", + "mismatches": [], "warn_reasons": []}]}) +expect("verdict says no mismatch", "No Infrahub/OpenStack mismatch" in d.verdict, d.verdict) +expect("redirects to the InfraInsight host query", any("InfraInsight" in a.text for a in d.actions)) + +print("\n[10] Duplicated IPs - one DELETING claimant, one with a wrong Infrahub IP") +multi = {"ok": True, "mode": "multi_vm", "instances": [ + {"name": "old-vm", "infrahub_id": "1", "openstack_id": "a", "ih_status": "DELETING", "os_status": "ACTIVE", + "server": {"status": "ACTIVE", "_public_ip": "69.19.137.135"}, "infrahub": {"floating_ip": "69.19.137.135"}, + "org_value": "1 - A", "owners": ["a@x.com"]}, + {"name": "new-vm", "infrahub_id": "2", "openstack_id": "b", "ih_status": "ACTIVE", "os_status": "ACTIVE", + "server": {"status": "ACTIVE", "_public_ip": "69.19.140.9"}, "infrahub": {"floating_ip": "69.19.137.135"}, + "org_value": "2 - B", "owners": ["b@x.com"]}, +]} +class FakeProm: + def resources_by_floating_ip(self, fip): + return [{"instance_name": "preprod-vm", "status": "ACTIVE", "region": "CANADA-1", "environment": "preprod"}] +d = run(":awkward:Duplicated IPs", dict(floating_ip="69.19.137.135"), vmdata=multi, prom=FakeProm()) +expect("found 2 claimants needing correction", "2 of 2" in d.verdict, d.verdict) +expect("DELETING claimant -> delete it", any("[old-vm]" in a.text and "stuck DELETING" in a.text for a in d.actions)) +expect("wrong-IP claimant -> Scenario #2", any("Scenario #2" in f.value for f in d.findings)) +expect("Scenario #2 comms filled with the real IP", any( + x.template_id == "dupip_corrected" and "69.19.140.9" in x.body for x in d.drafts)) +expect("an unset sign-off name is reported rather than left as a placeholder", any( + "AGENT_NAME" in x.unfilled for x in d.drafts if x.template_id == "dupip_corrected")) +expect("surfaced the PreProd claimant from Prometheus", any("preprod-vm" in f.label for f in d.findings)) +expect("asks for a re-check after 5-10 min", any("5-10 minutes" in a.text for a in d.actions)) + +print("\n[11] Problem with Total GPUs - customers on host") +d = run("Problem with Total GPUs in a System", dict(instance="CA1-ESC8-111", region="CANADA-1", gpu_name="B200-SXM"), + census={"ok": True, "total_gpus": 6, "instances": [ + {"name": "cust-vm", "status": "ACTIVE", "flavor": "n3-B200x6", "gpus": "6", "openstack_id": "z"}]}) +expect("verdict names the host", "CA1-ESC8-111" in d.verdict, d.verdict) +expect("lists the affected instance", any("cust-vm" in f.label for f in d.findings)) +expect("flags host-maintenance comms", any(a.kind == "comms" for a in d.actions)) +expect("escalates a Jira to Infra", any("Jira" in a.text and a.owner == runbooks.INFRA for a in d.actions)) +expect("notes there is no approved template", any("no approved customer template" in n for n in d.notes)) +expect("drafts nothing", not d.drafts) + +print("\n[12] K8s instance name is flagged") +d = run("Instance in ERROR state in :flag-ca:CA-1", {**L_ERR, "instance_name": "hyperstack-minion-lydhjev"}, + vm(host="N/A", ih_status="ERROR", os_status="ERROR")) +expect("noted the likely K8s node", any("Kubernetes" in n for n in d.notes)) + + +print("\n[13] GPU sockets - every physical GPU accounted for") +from triagelib.runbooks import _gpu_slots +import collections as _c + +R289 = [{"name": "luminous-hubble", "gpus": "1", "linked": True, "match": True, "ih_status": "ACTIVE", "os_status": "ACTIVE"}, + {"name": "vm832adbe203242", "gpus": "1", "linked": True, "match": True, "ih_status": "ACTIVE", "os_status": "ACTIVE"}, + {"name": "noble-maxwell", "gpus": "1", "linked": True, "match": True, "ih_status": "ACTIVE", "os_status": "ACTIVE"}, + {"name": "clever-schrodinger", "gpus": "2", "linked": True, "match": True, "ih_status": "ACTIVE", "os_status": "ACTIVE"}] + +# Real numbers from CA1-ESC812-289: 8 physical, 7 reported in use, 4 VMs on 5 GPUs. +s = _gpu_slots({"physical": 8, "in_use_metric": 7, "spare_capacity_artifact": False}, R289) +c = _c.Counter(x["kind"] for x in s) +expect("8 sockets drawn for an 8-GPU host", len(s) == 8, len(s)) +expect("5 named + 2 unaccounted + 1 free", (c["vm"], c["unaccounted"], c["free"]) == (5, 2, 1), dict(c)) + +s = _gpu_slots({"physical": 8, "in_use_metric": 8, "spare_capacity_artifact": True}, R289) +c = _c.Counter(x["kind"] for x in s) +expect("artifact host shows spare sockets as free, not unaccounted", + (c["vm"], c["unaccounted"], c["free"]) == (5, 0, 3), dict(c)) + +s = _gpu_slots({"physical": None, "in_use_metric": 2, "spare_capacity_artifact": False}, + [{"name": "basilica", "gpus": "1", "linked": True, "match": True, "ih_status": "ACTIVE", "os_status": "ACTIVE"}]) +c = _c.Counter(x["kind"] for x in s) +expect("unknown socket count degrades gracefully", (c["vm"], c["unaccounted"]) == (1, 1), dict(c)) + +s = _gpu_slots({"physical": 8, "in_use_metric": 8, "spare_capacity_artifact": False}, + [{"name": "ghost", "gpus": "8", "linked": False, "match": False, + "ih_status": "not in Infrahub", "os_status": "ACTIVE"}]) +expect("a VM with no Infrahub record still fills its sockets and is flagged", + len(s) == 8 and all(x["kind"] == "vm" and not x["linked"] for x in s)) + +s = _gpu_slots({"physical": 8, "in_use_metric": 5, "spare_capacity_artifact": False}, R289) +expect("no negative slots when in_use is below what VMs claim", len(s) >= 5 and all( + x["kind"] in ("vm", "free", "unaccounted") for x in s), len(s)) + +print(f"\n{'ALL CHECKS PASSED' if not fails else str(fails) + ' CHECK(S) FAILED'}") +sys.exit(1 if fails else 0) diff --git a/tests/test_screening.py b/tests/test_screening.py new file mode 100644 index 0000000..43817b5 --- /dev/null +++ b/tests/test_screening.py @@ -0,0 +1,290 @@ +"""Screening, exclusion, categorisation and ordering tests. + +These cover the noise-vs-real decisions, which are what keeps the queue small. +No network and no CX-Tools: snapshots are synthetic. +""" +import datetime as dt +import os +import sys + +sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..")) + +from triagelib import alerts as A, screening + + +class Snap: + """Stands in for prometheus.StateSnapshot.""" + + loaded = True + + def __init__(self, **kw): + self.by_openstack_id = {} + self.by_instance_name = {} + self.fip_counts = {} + self.rogue_delta = {} + self.total_gpus = {} + self.in_use_gpus = {} + self.resources_by_host = {} + self.broken_inputs = [] + self.unattributed_active = 0 + self.unattributed_active_gpus = 0 + self.__dict__.update(kw) + + +def al(name, **labels): + return A.from_labels({"alertname": name, **labels}) + + +def aged(alert, minutes): + alert.active_at = dt.datetime.now(dt.timezone.utc) - dt.timedelta(minutes=minutes) + return alert + + +FAILS = [] + + +def expect(label, cond, got=""): + print((" PASS " if cond else " FAIL ") + label + ("" if cond else f" <- {got}")) + if not cond: + FAILS.append(label) + + +OSID = "abc-123" +ERROR_ALERT = "Instance in ERROR state in :flag-ca:CA-1" +SHUTOFF_ALERT = "Instance in SHUTOFF state in :flag-ca:CA-1 for greater than 30 min" + +print("\nSTATE ALERTS - does the condition still hold?") +a = al(ERROR_ALERT, openstack_id=OSID, status="ERROR", region="CANADA-1") +a.screen = screening.screen(a, Snap(by_openstack_id={OSID: {"status": "ERROR"}})) +expect("Infrahub still ERROR -> real", a.screen["verdict"] == screening.REAL, a.screen) + +a = al(ERROR_ALERT, openstack_id=OSID, status="ERROR", region="CANADA-1") +a.screen = screening.screen(a, Snap(by_openstack_id={OSID: {"status": "ACTIVE"}})) +expect("recovered to ACTIVE -> resolved", + a.screen["verdict"] == screening.RESOLVED and "ACTIVE" in a.screen["reason"], a.screen) + +a = al(SHUTOFF_ALERT, openstack_id=OSID, status="SHUTOFF") +a.screen = screening.screen(a, Snap()) +expect("record gone from Infrahub -> resolved", a.screen["verdict"] == screening.RESOLVED, a.screen) + +a = al("Instance in CREATING state in :flag-ca:CA-1 for greater than 30min", + openstack_id="None", instance_name="vm-x", status="CREATING") +a.screen = screening.screen(a, Snap()) +expect("CREATING with no OpenStack ID -> real, not 'resolved'", + a.screen["verdict"] == screening.REAL, a.screen) + +a = al(ERROR_ALERT, openstack_id=OSID) +a.screen = screening.screen(a, Snap(by_openstack_id={OSID: {"status": "ERROR"}})) +expect("no status label -> unverified but kept", a.screen["verdict"] == screening.UNVERIFIED, a.screen) + +print("\nDUPLICATED IPs") +a = al(":awkward:Duplicated IPs", floating_ip="1.2.3.4") +a.screen = screening.screen(a, Snap(fip_counts={"1.2.3.4": 1})) +expect("one claimant left -> resolved", a.screen["verdict"] == screening.RESOLVED, a.screen) +a.screen = screening.screen(a, Snap(fip_counts={"1.2.3.4": 3})) +expect("three claimants -> real", a.screen["verdict"] == screening.REAL and "3 VMs" in a.screen["reason"], a.screen) +a.screen = screening.screen(a, Snap(fip_counts={"9.9.9.9": 2})) +expect("IP held by nobody -> resolved", a.screen["verdict"] == screening.RESOLVED, a.screen) + +print("\nSUSPECTED ROGUE VM - per-host GPU accounting gap") +a = al(":ninja:Suspected Rogue VM", instance="CA1-ESC8-068") +a.screen = screening.screen(a, Snap(rogue_delta={"CA1-ESC8-068": 0.0})) +expect("gap closed -> resolved", a.screen["verdict"] == screening.RESOLVED, a.screen) +a.screen = screening.screen(a, Snap(rogue_delta={"CA1-ESC8-068": 4.0}, + resources_by_host={"CA1-ESC8-068": [{}] * 4})) +expect("gap of 4 GPUs -> real", a.screen["verdict"] == screening.REAL, a.screen) +expect("reason quantifies the gap", "4 GPU(s)" in a.screen["reason"], a.screen["reason"]) +a.screen = screening.screen(a, Snap(rogue_delta={"other-host": 4.0}, total_gpus={"x": 8})) +expect("no data for the host -> unverified, still actionable", + a.screen["verdict"] == screening.UNVERIFIED and a.screen["actionable"], a.screen) + +print("\nTOTAL GPUs") +a = al("Problem with Total GPUs in a System", instance="h1", gpu_name="B200-SXM") +a.screen = screening.screen(a, Snap(total_gpus={"h1": 8})) +expect("full complement of 8 -> resolved", a.screen["verdict"] == screening.RESOLVED, a.screen) +a.screen = screening.screen(a, Snap(total_gpus={"h1": 6}, in_use_gpus={"h1": 6})) +expect("6 GPUs -> real", a.screen["verdict"] == screening.REAL, a.screen) + +print("\nSUPPRESSION RULES - what used to be hardcoded is now user-editable") +from triagelib import settings as settings_mod +import tempfile, os as _os + +_tmp = _os.path.join(tempfile.mkdtemp(), "settings.json") +CFG = settings_mod.Settings(_tmp) + +a = al(SHUTOFF_ALERT, openstack_id=OSID, status="SHUTOFF", + organization="3491 - luis.sarabando+runpod@nexgencloud.coms-Organization") +a.screen = screening.screen(a, Snap(by_openstack_id={OSID: {"status": "SHUTOFF"}}), CFG) +expect("default rule hides internal nexgencloud orgs", + a.screen["verdict"] == screening.SUPPRESSED, a.screen) +expect("suppression names the rule that did it", "Internal NexGen" in a.screen["reason"], a.screen["reason"]) + +a = al("Instance in SHUTOFF state in :flag-no:NO-1 for greater than 30 min", openstack_id=OSID, + status="SHUTOFF", instance="no1-stor-runpod03", instance_name="no1-stor-runpod03", + organization="99 - Real Customer") +a.screen = screening.screen(a, Snap(by_openstack_id={OSID: {"status": "SHUTOFF"}}), CFG) +expect("default rule hides runpod storage nodes", a.screen["verdict"] == screening.SUPPRESSED, a.screen) + +# The combinational case the team asked for: type AND organisation. +CFG.upsert_rule({"name": "Modal ERROR churn", "reason": "known batch churn", + "conditions": {"kind": ["error"], "organization": ["modal"]}}) +hit = al(ERROR_ALERT, openstack_id=OSID, status="ERROR", organization="19417 - colin@modal.coms-Organization") +hit.screen = screening.screen(hit, Snap(by_openstack_id={OSID: {"status": "ERROR"}}), CFG) +expect("error + modal is suppressed", hit.screen["verdict"] == screening.SUPPRESSED, hit.screen) + +miss = al(ERROR_ALERT, openstack_id=OSID, status="ERROR", organization="123 - Someone Else") +miss.screen = screening.screen(miss, Snap(by_openstack_id={OSID: {"status": "ERROR"}}), CFG) +expect("error from another org is NOT suppressed", miss.screen["verdict"] != screening.SUPPRESSED, miss.screen) + +other = al(SHUTOFF_ALERT, openstack_id=OSID, status="SHUTOFF", + organization="19417 - colin@modal.coms-Organization") +other.screen = screening.screen(other, Snap(by_openstack_id={OSID: {"status": "SHUTOFF"}}), CFG) +expect("modal SHUTOFF is NOT suppressed - both conditions must match", + other.screen["verdict"] != screening.SUPPRESSED, other.screen) + +empty = {"name": "catch all", "conditions": {}} +expect("a rule with no conditions never matches", not settings_mod.rule_matches( + settings_mod._normalize_rule(empty), hit)) + +expect("rules survive a reload", settings_mod.Settings(_tmp).rules and any( + r["name"] == "Modal ERROR churn" for r in settings_mod.Settings(_tmp).rules)) + +print("\nAGE DEMOTIONS") +a = aged(al("Problem with Total GPUs in a System", instance="h1"), 7 * 24 * 60) +a.screen = screening.screen(a, Snap(total_gpus={"h1": 6})) +expect("firing 7 days -> chronic", a.screen["verdict"] == screening.CHRONIC, a.screen) + +a = aged(al("Problem with Total GPUs in a System", instance="h1"), 60) +a.screen = screening.screen(a, Snap(total_gpus={"h1": 6})) +expect("firing 1 hour -> stays real", a.screen["verdict"] == screening.REAL, a.screen) + +print("\nFAIL-SAFE BEHAVIOUR") +a = al(ERROR_ALERT, openstack_id=OSID, status="ERROR") +a.state = "pending" +a.for_seconds = 1800 +a.screen = screening.screen(a, Snap(by_openstack_id={OSID: {"status": "ERROR"}})) +expect("pending -> screened out", a.screen["verdict"] == screening.PENDING and not a.screen["actionable"], a.screen) + +a = al(ERROR_ALERT, openstack_id=OSID, status="ERROR") +a.screen = screening.screen(a, None) +expect("no snapshot -> unverified but NOT hidden", + a.screen["verdict"] == screening.UNVERIFIED and a.screen["actionable"], a.screen) + +warnings = screening.health_warnings(Snap(broken_inputs=["openstack_nova_server_status"])) +expect("empty nova metric raises a monitoring warning", + len(warnings) == 1 and "openstack_nova_server_status" in warnings[0], warnings) + +print("\nEXCLUSION AND TAB ROUTING") +ex = al("Exists in Infrahub but does not exist in OpenStack", openstack_id=OSID) +expect("orphan spam excluded outright", A.is_excluded(ex) and not A.cx_relevant(ex)) +expect("node-exporter -> Infrastructure tab", + A.category("HostSwapIsFillingUp", "node-exporter-rules.yml") == "node") +expect("ceph -> Infrastructure tab", A.category("CephOsdDown", "ceph-rules.yml") == "infra") +expect("regional Infrahub rule -> CX tab", + A.category(ERROR_ALERT, "infrahub-rules-CA1.yml") == "cx") +expect("main Infrahub rule -> CX tab", + A.category(":ninja:Suspected Rogue VM", "infrahub-rules.yml") == "cx") +expect("status-mismatch rule classified", + A.classify("Openstack status=ACTIVE and Infrahub status!=ACTIVE in :flag-ca:CANADA-1 for greater " + "than 30min") == "status_mismatch") +expect("hibernation-failure rule maps to HIBERNATING", + A.classify("Failure of Hibernation on Infrahub in :flag-ca:CANADA-1 for greater than 30min") == "hibernating") +expect("orphan VM rule classified", A.classify(":pirate_flag:Suspected Orphan VM") == "orphan_vm") + +print("\nORDERING AND GROUPING") + + +def real(minutes, name=ERROR_ALERT, **labels): + x = aged(al(name, openstack_id="o%d" % minutes, status="ERROR", **labels), minutes) + x.screen = {"actionable": True, "verdict": "real", "label": "needs action", "reason": ""} + return x + + +ages = [x["age_minutes"] for x in A.group_alerts([real(500), real(10), real(100), real(9331)])[0]["alerts"]] +expect("newest first, oldest at the bottom", ages == [10, 100, 500, 9331], ages) + +unknown = real(50) +unknown.active_at = None +ages = [x["age_minutes"] for x in A.group_alerts([unknown, real(200), real(5)])[0]["alerts"]] +expect("unknown start time sorts last", ages == [5, 200, None], ages) + +groups = A.group_alerts([real(5), real(6, ":ninja:Suspected Rogue VM", instance="h1")]) +expect("focus order puts rogue VM before ERROR", [g["kind"] for g in groups][0] == "rogue_vm", + [g["kind"] for g in groups]) + +noisy = real(7) +noisy.screen = {"actionable": False, "verdict": "resolved", "label": "already resolved", "reason": ""} +group = A.group_alerts([real(5), noisy])[0] +expect("group counts action vs noise separately", + group["actionable"] == 1 and group["noise"] == 1, group) + +expect("age_text renders days", real(9331).age_text == "6d 11h", real(9331).age_text) +expect("age_text renders hours", real(431).age_text == "7h 11m", real(431).age_text) +expect("age_text renders minutes", real(7).age_text == "7m", real(7).age_text) + +print("\nTRUE AGE - activeAt reset by pipeline dips") +# activeAt says 7h; ALERTS history says 7 days. The true value must win. +a = real(431) +a.true_age_minutes, a.true_age_capped = 7 * 24 * 60, False +expect("effective age prefers the recovered duration", a.effective_age_minutes == 10080, a.effective_age_minutes) +expect("reset is detected", a.age_is_reset) +expect("raw activeAt still reported", a.age_text == "7h 11m", a.age_text) +expect("effective text renders days", a.effective_age_text == "7d", a.effective_age_text) +a.screen = screening.screen(a, Snap(by_openstack_id={"o431": {"status": "ERROR"}})) +expect("7-day ERROR -> overdue (runbook says contact within 24h), not chronic", + a.screen["verdict"] == screening.OVERDUE, a.screen) +expect("overdue stays in the actionable queue", a.screen["actionable"]) + +# A kind with no runbook SLA still demotes to chronic, and explains the reset. +g = al("Problem with Total GPUs in a System", instance="h9") +g.active_at = dt.datetime.now(dt.timezone.utc) - dt.timedelta(minutes=431) +g.true_age_minutes, g.true_age_capped = 7 * 24 * 60, False +g.screen = screening.screen(g, Snap(total_gpus={"h9": 6})) +expect("no-SLA kind, 7 days -> chronic", g.screen["verdict"] == screening.CHRONIC, g.screen) +expect("chronic reason explains the activeAt reset", "pipeline dip" in g.screen["detail"], g.screen["detail"]) + +print("\nVALIDATION FINDINGS - rogue VM rule defect") +r = al(":ninja:Suspected Rogue VM", instance="CA1-ESC8-057") +r.screen = screening.screen(r, Snap(rogue_delta={"CA1-ESC8-057": 1.0}, + in_use_gpus={"CA1-ESC8-057": 8.0}, + total_gpus={"CA1-ESC8-057": 8.0}, + resources_by_host={"CA1-ESC8-057": [{}] * 5})) +expect("In_Use == Total -> rule defect, not a rogue VM", + r.screen["verdict"] == screening.RULE_DEFECT, r.screen) +expect("rule defect is screened out of the queue", not r.screen["actionable"]) +expect("reason names it as spare capacity", "free GPU" in r.screen["reason"], r.screen["reason"]) + +r2 = al(":ninja:Suspected Rogue VM", instance="CA1-ESC812-289") +r2.screen = screening.screen(r2, Snap(rogue_delta={"CA1-ESC812-289": 2.0}, + in_use_gpus={"CA1-ESC812-289": 7.0}, + resources_by_host={"CA1-ESC812-289": [{}] * 4})) +expect("In_Use with no Total reading -> still a real gap", + r2.screen["verdict"] == screening.REAL, r2.screen) + +r3 = al(":ninja:Suspected Rogue VM", instance="h3") +r3.screen = screening.screen(r3, Snap(rogue_delta={"h3": 3.0}, in_use_gpus={"h3": 9.0}, + total_gpus={"h3": 8.0}, resources_by_host={"h3": [{}]})) +expect("In_Use != Total -> real gap", r3.screen["verdict"] == screening.REAL, r3.screen) + +b = real(431) +b.true_age_minutes, b.true_age_capped = 7 * 24 * 60, True +expect("window-capped age marked with +", b.effective_age_text == "7d+", b.effective_age_text) + +c = real(120) +c.true_age_minutes, c.true_age_capped = 130, False +expect("small drift is not flagged as a reset", not c.age_is_reset) +expect("no true age -> falls back to activeAt", real(90).effective_age_minutes == 90) + +# Ordering must use the recovered duration, not activeAt. +old, new = real(431), real(430) +old.true_age_minutes = 7 * 24 * 60 +new.true_age_minutes = 30 +order = [x["true_age_minutes"] for x in A.group_alerts([old, new])[0]["alerts"]] +expect("true age drives ordering, not activeAt", order == [30, 10080], order) + +dips = screening.health_warnings(Snap(pipeline_dips=[ + {"start": 0, "end": __import__("time").time() - 600, "minutes": 6, "low": 1359, "normal": 4374}])) +expect("pipeline dip raises a warning", len(dips) == 1 and "1359 of ~4374" in dips[0], dips) + +print("\n" + ("ALL CHECKS PASSED" if not FAILS else f"{len(FAILS)} CHECK(S) FAILED: {FAILS}")) +sys.exit(1 if FAILS else 0) diff --git a/triagelib/__init__.py b/triagelib/__init__.py new file mode 100644 index 0000000..bf0604e --- /dev/null +++ b/triagelib/__init__.py @@ -0,0 +1,4 @@ +"""CX Triage: read-only alert diagnosis on top of the CX-Tools (vmc) collectors.""" +from __future__ import annotations + +VERSION = "cx-triage 0.1" diff --git a/triagelib/alerts.py b/triagelib/alerts.py new file mode 100644 index 0000000..911be30 --- /dev/null +++ b/triagelib/alerts.py @@ -0,0 +1,421 @@ +"""Normalizes Prometheus alerts into the alert kinds the CX runbooks cover.""" +from __future__ import annotations + +import datetime as dt +import hashlib +import re +from dataclasses import dataclass, field +from typing import Any, Optional + +EMOJI_RE = re.compile(r":[a-z0-9_+\-]+:") +THRESHOLD_RE = re.compile(r"greater than\s*(\d+)\s*min", re.I) +ORG_RE = re.compile(r"^\s*(\d+)\s*-\s*(.*)$") +NONE_VALUES = {"", "none", "null", "unknown", "n/a"} + +# Alerts excluded outright. "Exists in Infrahub but does not exist in OpenStack" +# is built as `Resources unless on(openstack_id) openstack_nova_server_status`, +# and that right-hand metric is currently empty - so every Infrahub VM matches +# and the alert fires thousands of times. It is a monitoring fault, not a queue +# of work, so it never reaches the UI. +EXCLUDED_ALERTNAMES = frozenset({ + "Exists in Infrahub but does not exist in OpenStack", +}) + +# Rule files whose alerts belong to CX. Everything else is infrastructure. +CX_RULE_FILES = ("infrahub-rules",) +NODE_RULE_FILE = "node-exporter-rules.yml" + +# The order CX wants to work the queue in. +FOCUS_ORDER = ( + "rogue_vm", "duplicate_ip", "total_gpus", "hibernating", + "creating", "shutoff", "deleting", "error", + "restoring", "rebooting", "build", "orphan_vm", "status_mismatch", +) + +# Priority and estimated time to resolve, from the "Infrahub Errors +# Remediation" alert-conditions and runbook tables. +KIND_META: dict[str, dict[str, str]] = { + "error": {"title": "Instance in ERROR state", "priority": "LOW-HIGH", "ettr": "5-30 min", "delay": "none"}, + "deleting": {"title": "Instance in DELETING state", "priority": "LOW", "ettr": "5-15 min", "delay": "30 min"}, + "shutoff": {"title": "Instance in SHUTOFF state", "priority": "LOW", "ettr": "5-10 min", "delay": "30 min"}, + "hibernating": {"title": "Instance in HIBERNATING state", "priority": "HIGH", "ettr": "5-30 min", "delay": "30 min"}, + "creating": {"title": "Instance in CREATING state", "priority": "MEDIUM", "ettr": "5-15 min", "delay": "30 min"}, + "restoring": {"title": "Instance in RESTORING state", "priority": "HIGH", "ettr": "5-15 min", "delay": "30 min"}, + "rebooting": {"title": "Instance in REBOOTING state", "priority": "HIGH", "ettr": "5-15 min", "delay": "30 min"}, + "build": {"title": "Instance in BUILD state", "priority": "MEDIUM", "ettr": "5-15 min", "delay": "30 min"}, + "rogue_vm": {"title": "Suspected Rogue VM", "priority": "HIGH", "ettr": "5-30 min", "delay": "4 hours"}, + "duplicate_ip": {"title": "Duplicated IPs", "priority": "HIGH", "ettr": "5-15 min", "delay": "10 min"}, + "total_gpus": {"title": "Problem with Total GPUs in a System", "priority": "HIGH", "ettr": "5-15 min", "delay": "5 min"}, + "orphan_vm": {"title": "Suspected Orphan VM", "priority": "HIGH", "ettr": "5-30 min", "delay": "4 hours"}, + "status_mismatch": {"title": "Infrahub/OpenStack status mismatch", "priority": "HIGH", "ettr": "5-30 min", "delay": "30 min"}, +} + +STATE_KINDS = ("error", "deleting", "shutoff", "hibernating", "creating", "restoring", "rebooting", "build") + + +def clean_alertname(name: str) -> str: + """Strip the Slack emoji shortcodes Prometheus embeds in alert names.""" + return EMOJI_RE.sub("", str(name or "")).strip() + + +def classify(alertname: str) -> str: + name = clean_alertname(alertname).lower() + if alertname in EXCLUDED_ALERTNAMES or clean_alertname(alertname) in EXCLUDED_ALERTNAMES: + return "excluded" + if "rogue vm" in name: + return "rogue_vm" + if "orphan vm" in name: + return "orphan_vm" + if "duplicated ip" in name or "duplicate ip" in name: + return "duplicate_ip" + if "total gpus" in name: + return "total_gpus" + match = re.search(r"instance in (\w+) state", name) + if match: + state = match.group(1).lower() + if state in STATE_KINDS: + return state + # The per-region cross-check rules, e.g. + # "Openstack status=SHUTOFF and Infrahub status!=SHUTOFF in CANADA-1". + if "failure of hibernation" in name: + return "hibernating" + if "openstack status=" in name and "infrahub status" in name: + return "status_mismatch" + return "other" + + +def category(alertname: str, rule_file: str = "") -> str: + """Which tab an alert belongs in: 'cx', 'node', or 'infra'.""" + if any(token in rule_file for token in CX_RULE_FILES): + return "cx" + if rule_file == NODE_RULE_FILE: + return "node" + if rule_file: + return "infra" + # No rule metadata (e.g. a pasted alert): fall back to the classifier. + return "cx" if classify(alertname) not in ("other", "excluded") else "infra" + + +def _clean(value: Any) -> str: + text = str(value or "").strip() + return "" if text.lower() in NONE_VALUES else text + + +def split_organization(value: str) -> tuple[str, str]: + """Split the `organization` label ("8463 - Some Org") into id and name.""" + match = ORG_RE.match(str(value or "")) + if match: + return match.group(1), match.group(2).strip() + return "", _clean(value) + + +def _parse_active_at(value: Any) -> Optional[dt.datetime]: + text = str(value or "").strip() + if not text: + return None + text = re.sub(r"(\.\d{1,6})\d*Z?$", r"\1", text.replace("Z", "+00:00")) + if text.endswith("+00:00") is False and "+" not in text[10:]: + text += "+00:00" + try: + return dt.datetime.fromisoformat(text) + except ValueError: + return None + + +@dataclass +class Alert: + """One normalized Prometheus alert.""" + + kind: str + alertname: str + labels: dict[str, str] = field(default_factory=dict) + annotations: dict[str, str] = field(default_factory=dict) + state: str = "firing" + active_at: Optional[dt.datetime] = None + + # Fields the runbooks key off. + openstack_id: str = "" + instance_name: str = "" + host: str = "" + region_label: str = "" + region: str = "" + status: str = "" + floating_ip: str = "" + flavor_name: str = "" + flavor_gpu: str = "" + org_id: str = "" + org_name: str = "" + contract_id: str = "" + gpu_name: str = "" + threshold_min: Optional[int] = None + + # Where the rule came from (from RuleIndex), and the screening result. + rule_file: str = "" + rule_group: str = "" + for_seconds: int = 0 + category: str = "cx" + screen: dict[str, Any] = field(default_factory=dict) + + # How long the condition has actually held, recovered from ALERTS history. + # activeAt alone is unreliable: a metric-pipeline dip resets it on every + # live alert at once, which is why raw ages cluster on one timestamp. + true_age_minutes: Optional[int] = None + true_age_capped: bool = False + + @property + def title(self) -> str: + return KIND_META.get(self.kind, {}).get("title", clean_alertname(self.alertname)) + + @property + def priority(self) -> str: + return KIND_META.get(self.kind, {}).get("priority", "UNKNOWN") + + @property + def ettr(self) -> str: + return KIND_META.get(self.kind, {}).get("ettr", "unknown") + + @property + def is_kubernetes(self) -> bool: + """Per the general process: kube* instance names are likely K8s nodes.""" + return self.instance_name.lower().startswith("kube") or "-minion-" in self.instance_name.lower() + + @property + def age_minutes(self) -> Optional[int]: + if not self.active_at: + return None + now = dt.datetime.now(dt.timezone.utc) + return max(0, int((now - self.active_at).total_seconds() // 60)) + + @staticmethod + def _duration_text(minutes: Optional[int]) -> str: + if minutes is None: + return "unknown" + if minutes < 60: + return f"{minutes}m" + hours, mins = divmod(minutes, 60) + if hours < 24: + return f"{hours}h {mins}m" if mins else f"{hours}h" + days, hours = divmod(hours, 24) + return f"{days}d {hours}h" if hours else f"{days}d" + + @property + def age_text(self) -> str: + """Raw Prometheus activeAt duration.""" + return self._duration_text(self.age_minutes) + + @property + def effective_age_minutes(self) -> Optional[int]: + """True condition duration where known, else the raw activeAt age.""" + return self.true_age_minutes if self.true_age_minutes is not None else self.age_minutes + + @property + def effective_age_text(self) -> str: + text = self._duration_text(self.effective_age_minutes) + if self.true_age_minutes is not None and self.true_age_capped: + return f"{text}+" + return text + + @property + def age_is_reset(self) -> bool: + """True when activeAt materially understates how long this has held.""" + if self.true_age_minutes is None or self.age_minutes is None: + return False + return self.true_age_minutes - self.age_minutes > 60 + + @property + def is_internal_org(self) -> bool: + """Internal/test organizations are not customer-impacting.""" + return "nexgencloud.com" in self.org_name.lower() + + @property + def is_infra_owned(self) -> bool: + """Platform-owned nodes (storage etc.) name themselves after their host.""" + return bool(self.instance_name) and self.instance_name.lower() == self.host.lower() + + def fingerprint(self) -> str: + basis = "|".join([ + self.kind, + self.openstack_id or self.instance_name or "", + self.host, + self.floating_ip, + self.region, + ]) + return hashlib.sha1(basis.encode()).hexdigest()[:16] + + def to_json(self) -> dict[str, Any]: + return { + "id": self.fingerprint(), + "kind": self.kind, + "title": self.title, + "alertname": clean_alertname(self.alertname), + "raw_alertname": self.alertname, + "state": self.state, + "priority": self.priority, + "ettr": self.ettr, + "active_at": self.active_at.isoformat() if self.active_at else "", + "age_minutes": self.age_minutes, + "age_text": self.age_text, + "true_age_minutes": self.true_age_minutes, + "true_age_capped": self.true_age_capped, + "effective_age_minutes": self.effective_age_minutes, + "effective_age_text": self.effective_age_text, + "age_is_reset": self.age_is_reset, + "threshold_min": self.threshold_min, + "rule_file": self.rule_file, + "rule_group": self.rule_group, + "for_seconds": self.for_seconds, + "category": self.category, + "screen": self.screen, + "is_internal_org": self.is_internal_org, + "is_infra_owned": self.is_infra_owned, + "openstack_id": self.openstack_id, + "instance_name": self.instance_name, + "host": self.host, + "region": self.region, + "region_label": self.region_label, + "status": self.status, + "floating_ip": self.floating_ip, + "flavor_name": self.flavor_name, + "flavor_gpu": self.flavor_gpu, + "gpu_name": self.gpu_name, + "org_id": self.org_id, + "org_name": self.org_name, + "contract_id": self.contract_id, + "is_kubernetes": self.is_kubernetes, + "labels": self.labels, + "annotations": self.annotations, + } + + +def map_region(region_label: str) -> str: + """CANADA-1 -> ca1. + + Deliberately a local table rather than a call into CX-Tools: building the + alert queue must not touch cxlib, because constructing a CX-Tools Config + loads credentials and would pop a 1Password prompt just to list alerts. + Kept in sync with SUPPORTED_REGIONS in cxlib/constants.py. + """ + fallback = { + "canada-1": "ca1", "canada-2": "ca2", "us-1": "us1", "norway-1": "no1", + "ca-1": "ca1", "ca-2": "ca2", "no-1": "no1", + "ca1": "ca1", "ca2": "ca2", "us1": "us1", "no1": "no1", + } + return fallback.get(str(region_label or "").strip().lower(), "") + + +def from_labels(labels: dict[str, str], annotations: Optional[dict[str, str]] = None, + state: str = "firing", active_at: Any = None) -> Alert: + labels = {str(k): str(v) for k, v in (labels or {}).items()} + alertname = labels.get("alertname", "") + kind = classify(alertname) + + region_label = _clean(labels.get("region")) + org_id, org_name = split_organization(labels.get("organization", "")) + threshold = THRESHOLD_RE.search(alertname) + + # For host-scoped alerts the `instance` label is the hypervisor; for + # VM-scoped alerts it is the hypervisor too, or "Unknown" when the VM never + # landed on a host. + host = _clean(labels.get("instance")) + + alert = Alert( + kind=kind, + alertname=alertname, + labels=labels, + annotations={str(k): str(v) for k, v in (annotations or {}).items()}, + state=str(state or "firing"), + active_at=_parse_active_at(active_at), + openstack_id=_clean(labels.get("openstack_id")), + instance_name=_clean(labels.get("instance_name")), + host=host, + region_label=region_label, + region=map_region(region_label) or _infer_region_from_host(host), + status=_clean(labels.get("status")), + floating_ip=_clean(labels.get("floating_ip")), + flavor_name=_clean(labels.get("flavor_name")), + flavor_gpu=_clean(labels.get("flavor_gpu")), + gpu_name=_clean(labels.get("gpu_name")), + org_id=org_id, + org_name=org_name, + contract_id=_clean(labels.get("contract_id")), + threshold_min=int(threshold.group(1)) if threshold else None, + ) + return alert + + +def _infer_region_from_host(host: str) -> str: + match = re.match(r"^(ca1|ca2|no1|us1)-", str(host or "").strip(), re.I) + return match.group(1).lower() if match else "" + + +def from_prometheus(raw: dict[str, Any], rule_index: Any = None, true_age: Any = None) -> Alert: + alert = from_labels( + raw.get("labels") or {}, + raw.get("annotations") or {}, + state=str(raw.get("state") or "firing"), + active_at=raw.get("activeAt"), + ) + meta = rule_index.get(alert.alertname) if rule_index is not None else {} + if meta: + alert.rule_file = str(meta.get("file") or "") + alert.rule_group = str(meta.get("group") or "") + alert.for_seconds = int(meta.get("for_seconds") or 0) + alert.category = category(alert.alertname, alert.rule_file) + if true_age is not None: + alert.true_age_minutes, alert.true_age_capped = true_age.lookup(alert.labels) + return alert + + +def cx_relevant(alert: Alert) -> bool: + """True for alert kinds the CX runbooks cover.""" + return alert.kind in KIND_META and alert.kind != "excluded" + + +def is_excluded(alert: Alert) -> bool: + return alert.kind == "excluded" or alert.alertname in EXCLUDED_ALERTNAMES + + +def sort_key(alert: Alert) -> tuple: + """Newest first: a fresh alert is the one that still needs a decision. + + Long-running alerts sink to the bottom - they are either chronic and + already ticketed, or noise nobody has silenced. Alerts with no known start + time sort last rather than jumping to the top. + + Sorted on the true condition duration, not activeAt: a pipeline dip resets + activeAt on every live alert at once, which would otherwise flatten the + ordering into a single tie. + """ + age = alert.effective_age_minutes + return (age if age is not None else 10**9, alert.title) + + +def focus_rank(kind: str) -> int: + try: + return FOCUS_ORDER.index(kind) + except ValueError: + return len(FOCUS_ORDER) + + +def group_alerts(items: list[Alert]) -> list[dict[str, Any]]: + """Group alerts into collapsible sections, in CX's working order.""" + buckets: dict[str, list[Alert]] = {} + for alert in items: + buckets.setdefault(alert.kind, []).append(alert) + + groups: list[dict[str, Any]] = [] + for kind, members in buckets.items(): + members.sort(key=sort_key) + actionable = [a for a in members if a.screen.get("actionable", True)] + groups.append({ + "kind": kind, + "title": KIND_META.get(kind, {}).get("title", kind), + "priority": KIND_META.get(kind, {}).get("priority", "UNKNOWN"), + "ettr": KIND_META.get(kind, {}).get("ettr", "unknown"), + "total": len(members), + "actionable": len(actionable), + "noise": len(members) - len(actionable), + "alerts": [a.to_json() for a in members], + }) + groups.sort(key=lambda g: (focus_rank(g["kind"]), -g["actionable"])) + return groups diff --git a/triagelib/comms.py b/triagelib/comms.py new file mode 100644 index 0000000..60f70d5 --- /dev/null +++ b/triagelib/comms.py @@ -0,0 +1,383 @@ +"""Customer comms templates, transcribed from the CX runbooks. + +Wording is kept verbatim from Confluence so what CX sends stays consistent with +the approved snippets; only the named placeholders are substituted. Nothing here +sends anything - the app renders the draft for a human to review and send from +HubSpot. +""" +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import Any, Optional + +INSTANCE_PLACEHOLDER = "INFRAHUB_INSTANCE_NAME" +FIP_PLACEHOLDER = "NEW_INFRAHUB_FLOATING_IP" +ID_PLACEHOLDER = "INFRAHUB_ID" +OSID_PLACEHOLDER = "OPENSTACK_ID" +NAME_PLACEHOLDER = "GREETING_NAME" +AGENT_PLACEHOLDER = "AGENT_NAME" + + +def first_name(owner: str) -> str: + """'Bojan Jovanovic ' -> 'Bojan'. + + Falls back to an empty greeting rather than guessing: the examples show + 'Hello,' is acceptable, but 'Hello shettyatulya@gmail.com,' is not. + """ + text = str(owner or "").split("<", 1)[0].strip() + if not text or "@" in text: + return "" + first = text.split()[0] + return first if first[:1].isalpha() else "" + + +@dataclass +class Draft: + template_id: str + label: str + subject: str + body: str + channel: str = "HubSpot ticket" + when: str = "" + unfilled: list[str] = field(default_factory=list) + source: str = "" + + def to_json(self) -> dict[str, Any]: + return { + "template_id": self.template_id, + "label": self.label, + "subject": self.subject, + "body": self.body, + "channel": self.channel, + "when": self.when, + "unfilled": self.unfilled, + "source": self.source, + } + + +# Wording follows the house style CX actually sends: first-name greeting, the VM +# named with its Infrahub ID, an explicit "you will not be charged" line, the +# billing-states link, and a personal sign-off. Placeholders are substituted; +# everything else is left alone so what goes out stays consistent. +BILLING_DOC = ("Here is our documentation on VM states and their cost:\n" + "Which virtual machine states incur billing costs?") +STOCK_DOC = ("You can use our Stock API to check availability at the time of deploying a VM here - " + "Stock Availability") + +_TEMPLATES: dict[str, dict[str, str]] = { + "error_never_active": { + "label": "ERROR - never deployed (transient stock issue)", + "subject": "VM in Error state", + "when": "The instance never reached a host, so nothing was built. Recommend delete and retry.", + "source": "Instance in ERROR state", + "body": f"""Hello GREETING_NAME, + +We hope you are well. + +We are emailing you in regards to VM: INFRAHUB_INSTANCE_NAME (INFRAHUB_ID) + +Due to a transient stock issue this VM never fully deployed and is currently showing in Error. + +Our recommendation would be to delete the VM and try recreating a new one. Please be aware that whilst the VM is in an Error state you will not be charged for its usage. + +{BILLING_DOC} + +{STOCK_DOC} + +Just so you are aware, if the VM is not deleted after 14 calendar days we will proceed with deleting the VM on your behalf. + +Kind Regards, + +AGENT_NAME""", + }, + "error_was_active": { + "label": "ERROR - VM had been running, escalated", + "subject": "VM in Error state", + "when": "The instance had reached ACTIVE, so customer data may be involved. Escalate first, then send.", + "source": "Instance in ERROR state", + "body": f"""Hello GREETING_NAME, + +We hope you are well. + +We are emailing you in regards to VM: INFRAHUB_INSTANCE_NAME (INFRAHUB_ID) + +We can see this VM has gone into an Error state. We have escalated this to the appropriate team on your behalf and will come back to you as soon as we have more information. + +Please be assured that whilst a VM is in an Error state you will not be charged for its usage. + +{BILLING_DOC} + +Kind Regards, + +AGENT_NAME""", + }, + "creating": { + "label": "CREATING - stuck on deploy, VM deleted for the customer", + "subject": "VM stuck in creating state", + "when": "Send after the stuck instance has been deleted.", + "source": "Instance in CREATING state", + "body": f"""Hello GREETING_NAME, + +We hope you are well. + +We are emailing you in regards to VM INFRAHUB_INSTANCE_NAME (INFRAHUB_ID). + +We can see you tried to deploy this VM but due to a transient error the VM got stuck in a creating state. + +Unfortunately as the VM was not able to fully deploy, the safest option was to delete the VM which we have actioned for you. + +Please be assured that whilst a VM is in a creating state you will not be charged for it's usage. + +{BILLING_DOC} + +If you have any queries please let us know. + +Kind Regards, + +AGENT_NAME""", + }, + "deleting": { + "label": "DELETING - stuck delete finalised for the customer", + "subject": "VM stuck in deleting state", + "when": "Send once the delete has actually been finalised.", + "source": "Instance in DELETING state", + "body": f"""Hello GREETING_NAME, + +We hope you are well. + +We are emailing you in regards to VM: INFRAHUB_INSTANCE_NAME (INFRAHUB_ID) + +We can see one of your users requested the deletion and that due to a transient error the VM got stuck in a deleting state. I wanted to make sure you are aware that we have gone and finalised the deletion for you. + +Please be assured that whilst a VM is in a deleting state you will not be charged for it's usage. + +{BILLING_DOC} + +Kind Regards, + +AGENT_NAME""", + }, + "deleting_resolved": { + "label": "DELETING - confirming resolution and closing the ticket", + "subject": "VM deleted - closing this ticket", + "when": "Use when replying on an existing ticket that can now be closed.", + "source": "Instance in DELETING state", + "body": """Hello GREETING_NAME, + +Upon reviewing this ticket, we found that the VM below, which was previously stuck in a DELETING state, has now been deleted: + +* OPENSTACK_ID (INFRAHUB_ID) + +As the VM has been deleted, we are marking the issue as resolved and closing this ticket. + +If you require further assistance, please feel free to contact us at support@hyperstack.cloud or open a new Live Chat via the Hyperstack Console. + +Have a great rest of your day and thank you for using Hyperstack. + +Kind regards, + +AGENT_NAME""", + }, + "build": { + "label": "BUILD - failed to build, escalated", + "subject": "VM stuck in build state", + "when": "Send once escalated to Infrastructure. The instance must be recreated.", + "source": "Instance in BUILD state", + "body": f"""Hello GREETING_NAME, + +We hope you are well. + +We are emailing you in regards to VM INFRAHUB_INSTANCE_NAME (INFRAHUB_ID). + +We can see you tried to deploy this VM but due to a transient error it did not finish building. We have escalated this on your behalf and ask that you retry creating the instance at your convenience. + +Please be assured that whilst a VM is in this state you will not be charged for it's usage. + +{BILLING_DOC} + +If you have any queries please let us know. + +Kind Regards, + +AGENT_NAME""", + }, + "rebooting": { + "label": "REBOOTING - reboot failed, now resolved", + "subject": "VM stuck rebooting", + "when": "Send only once the instance is confirmed ACTIVE in both Infrahub and OpenStack.", + "source": "Instance in REBOOTING state", + "body": f"""Hello GREETING_NAME, + +We hope you are well. + +We are emailing you in regards to VM INFRAHUB_INSTANCE_NAME (INFRAHUB_ID). + +We can see a reboot was requested but due to a transient error the VM got stuck. This has now been resolved and you may retry rebooting at your convenience. + +{BILLING_DOC} + +If you have any queries please let us know. + +Kind Regards, + +AGENT_NAME""", + }, + "restoring": { + "label": "RESTORING - restore failed, now resolved", + "subject": "VM stuck restoring", + "when": "Send after the instance is back to SHELVED_OFFLOADED in OpenStack and HIBERNATED in Infrahub.", + "source": "Instance in RESTORING state", + "body": f"""Hello GREETING_NAME, + +We hope you are well. + +We are emailing you in regards to VM INFRAHUB_INSTANCE_NAME (INFRAHUB_ID). + +We can see you tried to restore this VM but due to a transient error it got stuck. This has now been resolved and you may retry restoring at your convenience. + +{BILLING_DOC} + +If you have any queries please let us know. + +Kind Regards, + +AGENT_NAME""", + }, + "shutoff": { + "label": "SHUTOFF - billing awareness notice", + "subject": "VM in SHUT-OFF state is still accruing costs", + "when": "Send as-is. A HubSpot snippet also exists: type #shutoff.", + "source": "Instance in SHUTOFF state", + "body": f"""Hello GREETING_NAME, + +We hope you are well. + +We are emailing you in regards to VM: INFRAHUB_INSTANCE_NAME (INFRAHUB_ID) + +We can see that this is in a SHUT-OFF state and wanted to make sure you are aware that in this state the VM is still accruing full costs. + +{BILLING_DOC} + +Kind Regards, + +AGENT_NAME""", + }, + "dupip_removed": { + "label": "Duplicated IP - incorrect IP removed, customer must attach a new one", + "subject": "Instance assigned an incorrect public IP", + "when": "The VM has no floating IP in OpenStack and the stale IP was removed in InfraInsight.", + "source": "Duplicated IPs", + "body": """Hello GREETING_NAME, + +We hope you are well. + +We are emailing you in regards to VM: INFRAHUB_INSTANCE_NAME (INFRAHUB_ID) + +Due to a transient synchronisation issue this instance was showing an incorrect public IP. This has now been corrected. To restore external connectivity the instance will need a new public IP attached, which you can do at your convenience using either the Hyperstack UI or API. + +We apologise for any inconvenience this may have caused. + +Kind Regards, + +AGENT_NAME""", + }, + "dupip_corrected": { + "label": "Duplicated IP - Infrahub corrected to match OpenStack", + "subject": "Instance assigned an incorrect public IP", + "when": "The VM does have a floating IP in OpenStack and Infrahub was corrected to match.", + "source": "Duplicated IPs", + "body": """Hello GREETING_NAME, + +We hope you are well. + +We are emailing you in regards to VM: INFRAHUB_INSTANCE_NAME (INFRAHUB_ID) + +Due to a transient synchronisation issue this instance was showing an incorrect public IP. This has now been resolved and your instance is reachable at NEW_INFRAHUB_FLOATING_IP. + +We apologise for any inconvenience this may have caused. + +Kind Regards, + +AGENT_NAME""", + }, + "sync_state": { + "label": "Rogue VM - instance was in the incorrect state", + "subject": "VM was showing an incorrect state", + "when": "Send after the state mismatch has been remediated.", + "source": "Suspected Rogue VM", + "body": """Hello GREETING_NAME, + +We hope you are well. + +We are emailing you in regards to VM: INFRAHUB_INSTANCE_NAME (INFRAHUB_ID) + +Due to a transient sync error this instance was showing an incorrect state. This has since been resolved, and we ask that you retry any operations that failed as a result. + +We apologise for any delay this may have caused. + +Kind Regards, + +AGENT_NAME""", + }, +} + + +def draft(template_id: str, *, instance_name: str = "", floating_ip: str = "", + infrahub_id: str = "", openstack_id: str = "", greeting_name: str = "", + agent_name: str = "", note: Optional[str] = None) -> Optional[Draft]: + spec = _TEMPLATES.get(template_id) + if not spec: + return None + body = spec["body"] + unfilled: list[str] = [] + + # "Hello Bojan," when we know the name, "Hello," when we do not - never + # "Hello ,". + body = body.replace(f"Hello {NAME_PLACEHOLDER},", f"Hello {greeting_name}," if greeting_name else "Hello,") + + substitutions = { + INSTANCE_PLACEHOLDER: instance_name, + ID_PLACEHOLDER: infrahub_id, + OSID_PLACEHOLDER: openstack_id, + FIP_PLACEHOLDER: floating_ip, + AGENT_PLACEHOLDER: agent_name, + } + for placeholder, value in substitutions.items(): + if placeholder not in body: + continue + if value and value not in ("N/A", "None"): + body = body.replace(placeholder, str(value)) + else: + unfilled.append(placeholder) + + # "(INFRAHUB_ID)" with nothing to put in it reads worse than no bracket. + if ID_PLACEHOLDER in unfilled: + body = body.replace(f" ({ID_PLACEHOLDER})", "").replace(f"({ID_PLACEHOLDER})", "") + unfilled.remove(ID_PLACEHOLDER) + + when = spec["when"] + if note: + when = f"{when} {note}".strip() + + return Draft( + template_id=template_id, + label=spec["label"], + subject=spec["subject"], + body=body, + when=when, + unfilled=unfilled, + source=spec["source"], + ) + + +def contacts_from_result(result: dict[str, Any]) -> dict[str, Any]: + """Pull the organization and owner contacts CX-Tools resolved for a VM.""" + if not isinstance(result, dict): + return {"organization": "", "owners": [], "resolved": False} + org = str(result.get("org_value") or "").strip() + owners = [str(x) for x in (result.get("owners") or []) if str(x).strip()] + return { + "organization": org if org and org != "N/A" else "", + "owners": owners, + "resolved": bool(owners), + } diff --git a/triagelib/cxbridge.py b/triagelib/cxbridge.py new file mode 100644 index 0000000..36d98b7 --- /dev/null +++ b/triagelib/cxbridge.py @@ -0,0 +1,286 @@ +"""Read-only adapter over the CX-Tools (vmc) collectors. + +CX-Tools is imported as an unmodified library: this module never writes to the +CX-Tools tree and only calls collectors and query helpers that read. Every +OpenStack subcommand this module can reach is checked against READ_ONLY_VERBS +before it runs, so a bug here cannot mutate a live instance. +""" +from __future__ import annotations + +import os +import sys +import threading +from typing import Any, Optional + +DEFAULT_CX_TOOLS_PATHS = ( + os.environ.get("CX_TOOLS_PATH", ""), + os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "CX-Tools"), + os.path.expanduser("~/scripts/CX-Tools"), + os.path.expanduser("~/scripts/cx-tooling"), +) + +# OpenStack verbs the diagnosis path is allowed to reach. Anything that could +# change state (create/delete/set/unset/shelve/reboot/...) is absent on purpose. +READ_ONLY_VERBS = frozenset({"show", "list"}) + + +class BridgeError(RuntimeError): + """Raised when CX-Tools cannot be located, imported, or authenticated.""" + + +def locate_cx_tools() -> str: + for candidate in DEFAULT_CX_TOOLS_PATHS: + if not candidate: + continue + path = os.path.abspath(os.path.expanduser(candidate)) + if os.path.isfile(os.path.join(path, "cxlib", "__init__.py")): + return path + raise BridgeError( + "Could not find the CX-Tools checkout. Set CX_TOOLS_PATH to the directory " + "that contains cxlib/ and the vmc entry point." + ) + + +_lock = threading.Lock() +_state: dict[str, Any] = {"path": "", "cx": None, "config": None} + + +def _import_cxlib(path: str): + if path not in sys.path: + sys.path.insert(0, path) + try: + import cxlib # noqa: PLC0415 (import is intentionally deferred) + except Exception as exc: # pragma: no cover - depends on the local checkout + raise BridgeError(f"Failed to import cxlib from {path}: {exc}") from exc + return cxlib + + +def bootstrap() -> tuple[Any, Any]: + """Import cxlib and build the shared Config, loading secrets exactly once. + + Call this from the foreground at startup: constructing Config triggers the + CX-Tools 1Password loader, which may need an interactive sign-in. + """ + with _lock: + if _state["config"] is not None: + return _state["cx"], _state["config"] + path = locate_cx_tools() + cx = _import_cxlib(path) + config = cx.Config(no_color=True, debug=bool(os.environ.get("CX_DEBUG"))) + if not config.api_key or config.api_key in {"REDACT", "REPLACE_WITH_API_KEY"}: + raise BridgeError( + "CX-Tools could not load the Infrahub API key from 1Password. " + "Run `op signin` in this shell, then restart cx-triage." + ) + _state.update({"path": path, "cx": cx, "config": config}) + return cx, config + + +def cx() -> Any: + return bootstrap()[0] + + +def config() -> Any: + return bootstrap()[1] + + +def cx_tools_path() -> str: + bootstrap() + return str(_state["path"]) + + +def quiet_progress() -> Any: + """A Progress object that renders nothing, for use off the terminal.""" + c = cx() + return c.Progress(c.C(False), 1, enabled=False) + + +def _guard_openstack(args: list[str]) -> None: + verbs = [a for a in args if not str(a).startswith("-")] + if not any(v in READ_ONLY_VERBS for v in verbs): + raise BridgeError(f"Refusing to run a non-read-only OpenStack command: {' '.join(args)}") + + +def os_json(region: str, args: list[str], timeout: int = 90) -> tuple[bool, Any, str]: + """Run a read-only `openstack ... -f json` command through CX-Tools.""" + _guard_openstack(args) + return cx().os_json(config(), region, args, timeout=timeout) + + +# --- collectors ------------------------------------------------------------- + +def collect_vm(target: str, *, region: str = "", org_id: Optional[str] = None, ssh_timeout: int = 3) -> dict[str, Any]: + """Full VM reconciliation: the same payload `vmc --json ` emits.""" + return cx().collect_vm( + config(), + target, + region_arg=region or "", + org_id=org_id, + ssh_timeout=ssh_timeout, + include_ih_events=True, + include_volumes=True, + include_all_ih_events=True, + progress=quiet_progress(), + ) + + +def collect_host(host: str, *, ssh_timeout: int = 3) -> dict[str, Any]: + """Host reconciliation: the same payload `vmc --json --host ` emits.""" + return cx().collect_host( + config(), + host, + ssh_timeout=ssh_timeout, + include_ih_events=True, + include_volumes=True, + progress=quiet_progress(), + ) + + +def collect_vm_contacts(target: str, *, region: str = "", org_id: Optional[str] = None) -> dict[str, Any]: + return cx().collect_vm_contacts( + config(), + target, + region_arg=region or "", + org_id=org_id, + progress=quiet_progress(), + ) + + +# --- targeted queries used by individual runbooks -------------------------- + +def openstack_events(region: str, openstack_id: str, limit: Optional[int] = 5) -> list[dict[str, Any]]: + ok, events, _raw = cx().server_event_list(config(), region, openstack_id) + if not ok: + return [] + return events[:limit] if limit else events + + +def openstack_event_detail(region: str, openstack_id: str, request_id: str) -> dict[str, Any]: + ok, detail, _raw = cx().server_event_show(config(), region, openstack_id, request_id) + return detail if ok else {} + + +def failed_openstack_event(region: str, openstack_id: str, scan: int = 5) -> dict[str, Any]: + """Return the most recent OpenStack event whose detail reports a failure. + + The state runbooks all say "the most recent failed event is the thing to + escalate", so this walks recent events newest-first and returns the first + one whose result is not Success, together with its detail rows. + """ + c = cx() + for event in openstack_events(region, openstack_id, limit=scan): + request_id = c.event_request_id(event) + if not request_id: + continue + detail = openstack_event_detail(region, openstack_id, request_id) + if not detail: + continue + rows = dict((str(k), str(v)) for k, v in c.event_detail_rows(detail)) + result = rows.get("Result", "") + if result and result.lower() != "success": + return {"request_id": request_id, "action": rows.get("Action", ""), "rows": rows} + return {} + + +def infrahub_events(infrahub_id: str, limit: Optional[int] = None) -> list[list[str]]: + c = cx() + ok, data, _raw = c.query_vm_events(config(), str(infrahub_id)) + if not ok: + return [] + return c.infrahub_event_rows(data, limit) + + +def host_health(region: str, host: str) -> dict[str, Any]: + """Hypervisor, Nova service and OVS agent signals for one host. + + This is the cheap subset of `vmc --host` - the runbooks' "Host Health + Checks" entry point - without collecting every instance on the host. + """ + c = cx() + cfg = config() + ok_hv, hv, raw_hv, hv_name = c.hypervisor_show_host(cfg, region, host) + if not ok_hv: + return {"ok": False, "error": raw_hv, "host": host, "region": region} + + state = c.normalize_empty(c.first_present(hv, "state", "State", default="")) + status = c.normalize_empty(c.first_present(hv, "status", "Status", default="")) + + disabled_reason = "" + if status.lower() == "disabled": + for candidate in dict.fromkeys([x for x in (c.normalize_empty(hv_name), host) if x]): + ok_svc, services, _raw = c.compute_service_list_host(cfg, region, candidate) + if ok_svc: + disabled_reason = c.disabled_reason_from_services(services) + if disabled_reason: + break + + ovs: dict[str, Any] = {} + ok_agents, agents, _raw_agents = c.network_agent_list_host(cfg, region, host) + if ok_agents: + ovs = c.ovs_agent_summary(agents) + if ovs.get("agent_id"): + ok_show, detail, _raw_show = c.network_agent_show(cfg, region, str(ovs["agent_id"])) + if ok_show: + ovs["last_heartbeat_at"] = c.normalize_empty( + detail.get("last_heartbeat_at") or detail.get("Last Heartbeat At") or ovs.get("last_heartbeat_at") + ) + + return { + "ok": True, + "error": "", + "host": host, + "hypervisor_name": hv_name, + "region": region, + "nova_state": state or "N/A", + "nova_status": status or "N/A", + "disabled_reason": disabled_reason, + "uptime": c.host_uptime_summary(hv), + "aggregates": c.host_aggregates_summary(hv), + "ovs_alive": ovs.get("alive"), + "ovs_state": ovs.get("state"), + "ovs_last_heartbeat": ovs.get("last_heartbeat_at") or "", + "running_vms": c.normalize_empty(c.first_present(hv, "running_vms", "Running VMs", default="")) or "N/A", + "free_disk_gb": c.normalize_empty(c.first_present(hv, "free_disk_gb", "Free Disk GB", default="")) or "N/A", + "local_disk_free": c.normalize_empty(c.first_present(hv, "disk_available_least", "Disk Available Least", default="")) or "N/A", + } + + +def host_gpu_census(region: str, host: str) -> dict[str, Any]: + """Sum GPU counts of every instance on a host. + + Implements the ERROR-runbook check for the NUMA/PCI fault: "check if the + host is full prior to escalation - add the values after the x, if it = 8 + then it is FULL". + """ + c = cx() + ok, rows, raw = c.server_list_on_host(config(), region, host) + if not ok: + return {"ok": False, "error": raw, "total_gpus": None, "instances": []} + total = 0 + unknown = 0 + instances: list[dict[str, str]] = [] + for row in rows: + flavor = c.get_row_field(row, "Flavor", "flavor") or c.flavor_name_from_any(row) + count = c.gpu_count_from_flavor_name(flavor) + if count.isdigit(): + total += int(count) + else: + unknown += 1 + instances.append({ + "name": c.get_row_field(row, "Name", "name") or "N/A", + "openstack_id": c.openstack_id_from_row(row) or "N/A", + "status": c.get_row_field(row, "Status", "status") or "N/A", + "flavor": flavor or "N/A", + "gpus": count, + }) + return { + "ok": True, + "error": "", + "total_gpus": total, + "unknown_flavors": unknown, + "instances": instances, + } + + +def json_safe(obj: Any) -> Any: + return cx().json_safe(obj) diff --git a/triagelib/integrations.py b/triagelib/integrations.py new file mode 100644 index 0000000..c69470c --- /dev/null +++ b/triagelib/integrations.py @@ -0,0 +1,196 @@ +"""Outbound action payloads: Zendesk tickets and Jira issues. + +This module *builds* payloads and never sends them. Delivery is a separate, +explicitly configured step - see `outbox.py` - so that a diagnosis can never +contact a customer as a side effect of being viewed. + +Every payload carries the evidence that justified it, so the ticket a customer +or the Infrastructure team receives is self-contained. +""" +from __future__ import annotations + +import os +from dataclasses import dataclass, field +from typing import Any, Optional + +# Set these to enable the Send buttons. Absent = preview only. +ZENDESK_SUBDOMAIN = os.environ.get("CX_ZENDESK_SUBDOMAIN", "") +ZENDESK_EMAIL = os.environ.get("CX_ZENDESK_EMAIL", "") +ZENDESK_TOKEN = os.environ.get("CX_ZENDESK_TOKEN", "") +JIRA_BASE = os.environ.get("CX_JIRA_BASE", "https://nexgencloud.atlassian.net") +JIRA_EMAIL = os.environ.get("CX_JIRA_EMAIL", "") +JIRA_TOKEN = os.environ.get("CX_JIRA_TOKEN", "") +JIRA_PROJECT = os.environ.get("CX_JIRA_PROJECT", "INFRA") + +PRIORITY_BY_VERDICT = {"overdue": "high", "real": "normal", "unverified": "low"} + + +def zendesk_configured() -> bool: + return bool(ZENDESK_SUBDOMAIN and ZENDESK_EMAIL and ZENDESK_TOKEN) + + +def jira_configured() -> bool: + return bool(JIRA_BASE and JIRA_EMAIL and JIRA_TOKEN) + + +@dataclass +class Action: + """One proposed outbound action, ready to send once a human confirms.""" + + id: str + kind: str # zendesk | jira | manual + label: str + summary: str # one line: what this does + payload: dict[str, Any] = field(default_factory=dict) + recipients: list[str] = field(default_factory=list) + enabled: bool = False # is the integration configured? + blocked_reason: str = "" + requires_confirmation: bool = True + + def to_json(self) -> dict[str, Any]: + return { + "id": self.id, "kind": self.kind, "label": self.label, "summary": self.summary, + "payload": self.payload, "recipients": self.recipients, "enabled": self.enabled, + "blocked_reason": self.blocked_reason, "requires_confirmation": self.requires_confirmation, + } + + +def _parse_owner(owner: str) -> tuple[str, str]: + """'Name ' -> ('Name', 'email@x').""" + text = str(owner or "").strip() + if "<" in text and ">" in text: + name = text.split("<", 1)[0].strip() + email = text.split("<", 1)[1].split(">", 1)[0].strip() + return name, email + return ("", text) if "@" in text else (text, "") + + +def _evidence_block(diagnosis: Any) -> str: + lines = [f"Alert: {diagnosis.alert.title}", f"Verdict: {diagnosis.verdict}", ""] + for finding in diagnosis.findings[:16]: + lines.append(f"- {finding.label}: {finding.value}") + return "\n".join(lines) + + +def build_zendesk(diagnosis: Any) -> Optional[Action]: + """A customer ticket, only when the runbook actually calls for contact.""" + if not diagnosis.drafts: + return None + draft = diagnosis.drafts[0] + alert = diagnosis.alert + contacts = diagnosis.contacts or {} + owners = contacts.get("owners") or [] + if not owners: + return Action( + id="zendesk", kind="zendesk", label="Contact customer (Zendesk)", + summary="No owner contact resolved from Infrahub - look the organization up first.", + enabled=False, blocked_reason="No customer contact could be resolved.", + ) + + name, email = _parse_owner(owners[0]) + verdict = (alert.screen or {}).get("verdict", "real") + payload = { + "ticket": { + "subject": draft.subject, + "comment": {"body": draft.body, "public": True}, + "requester": {"name": name or email, "email": email}, + "priority": PRIORITY_BY_VERDICT.get(verdict, "normal"), + "type": "incident", + "tags": ["cx-triage", f"alert-{alert.kind}", f"region-{alert.region or 'unknown'}"], + "external_id": f"cx-triage-{alert.fingerprint()}", + "custom_fields_note": { + "instance_name": alert.instance_name, + "openstack_id": alert.openstack_id, + "organization": alert.org_name, + "infrahub_org_id": alert.org_id, + }, + }, + "_template": draft.template_id, + "_when": draft.when, + "_unfilled": draft.unfilled, + } + blocked = "" + if draft.unfilled: + blocked = f"Template still has placeholders: {', '.join(draft.unfilled)}" + return Action( + id="zendesk", kind="zendesk", + label="Contact customer (Zendesk)", + summary=f"Public reply to {name or email} - {draft.label}", + payload=payload, + recipients=[o for o in owners], + enabled=zendesk_configured() and not blocked, + blocked_reason=blocked or ("" if zendesk_configured() else "Zendesk is not configured."), + ) + + +def build_jira(diagnosis: Any) -> Optional[Action]: + """An Infrastructure escalation, only when a step is owned by Infra.""" + infra_steps = [a for a in diagnosis.actions if a.owner != "CX" and a.kind == "escalate"] + if not infra_steps: + return None + alert = diagnosis.alert + subject = alert.host or alert.instance_name or alert.floating_ip or "unknown" + + description = "\n".join([ + _evidence_block(diagnosis), + "", + "Requested of Infrastructure:", + *[f"- {s.text}" for s in infra_steps], + "", + f"Raised from CX Triage. Alert has held for {alert.effective_age_text}.", + ]) + payload = { + "fields": { + "project": {"key": JIRA_PROJECT}, + "summary": f"{subject}: {diagnosis.verdict}"[:250], + "description": description, + "issuetype": {"name": "Task"}, + "labels": ["cx-triage", f"alert-{alert.kind}", f"region-{alert.region or 'unknown'}"], + } + } + return Action( + id="jira", kind="jira", + label="Escalate to Infrastructure (Jira)", + summary=f"Create a {JIRA_PROJECT} issue for {subject}", + payload=payload, + enabled=jira_configured(), + blocked_reason="" if jira_configured() else "Jira is not configured.", + ) + + +def build_manual(diagnosis: Any) -> list[Action]: + """Steps a human must perform; surfaced as copyable commands, not buttons.""" + out: list[Action] = [] + alert = diagnosis.alert + osid = alert.openstack_id + region = alert.region + for step in diagnosis.actions: + if step.kind != "remediate" or step.status == "done": + continue + command = "" + low = step.text.lower() + if "delete the server in openstack" in low and osid and region: + command = f"{region} server delete {osid}" + elif "shelve" in low and osid and region: + command = f"{region} server shelve {osid}" + out.append(Action( + id=f"manual-{len(out)}", kind="manual", label=step.text, + summary=step.guide or "", payload={"command": command} if command else {}, + enabled=False, blocked_reason="Perform manually - this tool is read-only.", + requires_confirmation=False, + )) + return out + + +def build_all(diagnosis: Any) -> dict[str, Any]: + actions: list[Action] = [] + for builder in (build_zendesk, build_jira): + action = builder(diagnosis) + if action: + actions.append(action) + actions.extend(build_manual(diagnosis)) + return { + "actions": [a.to_json() for a in actions], + "zendesk_configured": zendesk_configured(), + "jira_configured": jira_configured(), + } diff --git a/triagelib/linkage.py b/triagelib/linkage.py new file mode 100644 index 0000000..252e8f9 --- /dev/null +++ b/triagelib/linkage.py @@ -0,0 +1,233 @@ +"""Linkage analysis: Infrahub records that lost their OpenStack server, and +OpenStack servers that no Infrahub record claims. + +A VM in ERROR is not always a failed build. Sometimes the build succeeded and +only the *link* between Infrahub and OpenStack was never written - so Infrahub +reports ERROR (or CREATING) with no usable openstack_id while a perfectly good +server of the same name is running. Those look identical on an alert dashboard +and are opposite problems: one needs a rebuild, the other needs a record fixed +and is quietly billing nobody. + +This scans both sides in bulk and pairs them up by name. + +It also finds the reverse - OpenStack servers with no Infrahub record at all - +which is what `Suspected Orphan VM` was meant to catch before its input metric +went empty. +""" +from __future__ import annotations + +import difflib +import threading +import time +from typing import Any, Optional + +from . import cxbridge + +REGIONS = ("ca1", "ca2", "us1", "no1") + +# Infrahub states where a missing OpenStack link is suspicious rather than normal. +# HIBERNATED is excluded: shelved instances legitimately have no running server. +UNLINKED_SUSPECT_STATES = {"ERROR", "CREATING", "BUILD", "ACTIVE", "REBOOTING", "RESTORING"} + +# Projects to ignore, matching the CX-Tools tempest suppression. +def _is_ignorable(name: str) -> bool: + low = str(name or "").lower() + return "tempest" in low + + +_REGION_ALIAS = {"canada-1": "ca1", "canada-2": "ca2", "us-1": "us1", "norway-1": "no1", + "ca1": "ca1", "ca2": "ca2", "us1": "us1", "no1": "no1"} + + +def _region_alias(region: str) -> str: + return _REGION_ALIAS.get(str(region or "").strip().lower(), "") + + +def _norm(name: str) -> str: + return str(name or "").strip().lower() + + +class Scan: + """One full cross-region scan. Slow (a server list per region), so cached.""" + + def __init__(self): + self.started = 0.0 + self.finished = 0.0 + self.state = "idle" # idle | running | done | error + self.error = "" + self.progress = "" + self.result: dict[str, Any] = {} + self._lock = threading.Lock() + + def to_json(self) -> dict[str, Any]: + return { + "state": self.state, "error": self.error, "progress": self.progress, + "started": self.started, "finished": self.finished, + "age_seconds": int(time.time() - self.finished) if self.finished else None, + "result": self.result, + } + + def run(self, snapshot: Any, regions: tuple[str, ...] = REGIONS) -> None: + with self._lock: + if self.state == "running": + return + self.state = "running" + self.started = time.time() + self.error = "" + self.progress = "starting" + try: + self.result = self._scan(snapshot, regions) + self.state = "done" + except Exception as exc: # surfaced in the UI rather than crashing the server + self.state = "error" + self.error = f"{type(exc).__name__}: {exc}" + finally: + self.finished = time.time() + self.progress = "" + + # --- the analysis ------------------------------------------------------ + + def _scan(self, snapshot: Any, regions: tuple[str, ...]) -> dict[str, Any]: + # 1. Everything OpenStack has, per region. + os_by_id: dict[str, dict[str, str]] = {} + os_by_name: dict[str, list[dict[str, str]]] = {} + region_counts: dict[str, int] = {} + failures: dict[str, str] = {} + + for region in regions: + self.progress = f"listing OpenStack servers in {region}" + ok, rows, raw = _server_list(region) + if not ok: + failures[region] = raw[:200] + continue + region_counts[region] = len(rows) + for row in rows: + sid = str(row.get("ID") or "") + if not sid: + continue + rec = { + "id": sid, + "name": str(row.get("Name") or ""), + "status": str(row.get("Status") or ""), + "task": str(row.get("Task State") or ""), + "host": str(row.get("Host") or ""), + "project_id": str(row.get("Project ID") or ""), + "flavor": str(row.get("Flavor") or ""), + "region": region, + } + os_by_id[sid] = rec + os_by_name.setdefault(_norm(rec["name"]), []).append(rec) + + # 2. Everything Infrahub has, from the bulk metric snapshot. + self.progress = "comparing against Infrahub" + infrahub = list(snapshot.by_openstack_id.values()) + list(snapshot.by_instance_name.values()) + seen: set[str] = set() + ih_records: list[dict[str, str]] = [] + for row in infrahub: + key = f"{row.get('openstack_id','')}|{row.get('instance_name','')}" + if key in seen: + continue + seen.add(key) + ih_records.append(row) + + ih_osids = {str(r.get("openstack_id") or "") for r in ih_records if r.get("openstack_id")} + ih_osids.discard("") + ih_osids.discard("None") + + # 3a. Infrahub records whose OpenStack server is missing or never linked. + # + # Only records in a region that was actually listed can be judged: if a + # region failed, every VM in it would look "missing from OpenStack". + scanned = set(region_counts) + skipped_unscanned = 0 + + broken_links: list[dict[str, Any]] = [] + for row in ih_records: + status = str(row.get("status") or "").upper() + if status not in UNLINKED_SUSPECT_STATES: + continue + if _region_alias(str(row.get("region") or "")) not in scanned: + skipped_unscanned += 1 + continue + osid = str(row.get("openstack_id") or "") + has_link = bool(osid) and osid != "None" + if has_link and osid in os_by_id: + continue # properly linked, nothing to see + + name = str(row.get("instance_name") or "") + if _is_ignorable(name): + continue + + candidates = os_by_name.get(_norm(name), []) + # An exact-name server that nothing else claims is a very strong + # candidate for the link that was never written. + unclaimed = [c for c in candidates if c["id"] not in ih_osids] + match = unclaimed[0] if unclaimed else (candidates[0] if candidates else None) + + broken_links.append({ + "instance_name": name, + "infrahub_status": status, + "infrahub_openstack_id": osid or "(none)", + "organization": str(row.get("organization") or ""), + "region": str(row.get("region") or ""), + "flavor": str(row.get("flavor_name") or ""), + "gpus": str(row.get("_gpus") or ""), + "reason": ( + "Infrahub holds an OpenStack ID that OpenStack does not have" + if has_link else "Infrahub never recorded an OpenStack ID" + ), + "candidate": match, + "candidate_claimed_by_other": bool(match and match["id"] in ih_osids), + "confidence": ( + "high" if match and not match["id"] in ih_osids and match["status"] not in ("", "ERROR") + else "medium" if match else "none" + ), + }) + + # 3b. OpenStack servers no Infrahub record claims. + orphans: list[dict[str, Any]] = [] + ih_names = {_norm(str(r.get("instance_name") or "")) for r in ih_records} + for sid, rec in os_by_id.items(): + if sid in ih_osids or _is_ignorable(rec["name"]): + continue + orphans.append({**rec, "name_known_to_infrahub": _norm(rec["name"]) in ih_names}) + + linkable = [b for b in broken_links if b["candidate"] and not b["candidate_claimed_by_other"]] + return { + "scanned_regions": region_counts, + "region_failures": failures, + "openstack_servers": len(os_by_id), + "infrahub_records": len(ih_records), + "broken_links": sorted(broken_links, key=lambda b: (b["confidence"] != "high", b["instance_name"])), + "likely_linkage_failures": len(linkable), + "skipped_unscanned_regions": skipped_unscanned, + "orphans": sorted(orphans, key=lambda o: (not o["name_known_to_infrahub"], o["name"]))[:400], + "orphan_total": len(orphans), + } + + +def _server_list(region: str) -> tuple[bool, list[dict[str, Any]], str]: + ok, data, raw = cxbridge.os_json( + region, ["server", "list", "--all-projects", "--long", "-f", "json"], timeout=180 + ) + if ok and isinstance(data, list): + return True, [x for x in data if isinstance(x, dict)], "" + return False, [], str(raw) + + +def enrich(region: str, openstack_id: str) -> dict[str, Any]: + """Fetch created time and fault for one candidate, on demand.""" + c = cxbridge.cx() + ok, srv, raw = c.server_show(cxbridge.config(), region, openstack_id) + if not ok: + return {"ok": False, "error": str(raw)[:200]} + fault = srv.get("fault") + return { + "ok": True, + "created": str(srv.get("created") or srv.get("Created") or ""), + "launched": str(srv.get("OS-SRV-USG:launched_at") or ""), + "status": str(srv.get("status") or ""), + "host": str(srv.get("OS-EXT-SRV-ATTR:host") or ""), + "project_id": str(srv.get("project_id") or ""), + "fault": (fault.get("message") if isinstance(fault, dict) else str(fault or "")) or "None", + } diff --git a/triagelib/prometheus.py b/triagelib/prometheus.py new file mode 100644 index 0000000..c2b6a1e --- /dev/null +++ b/triagelib/prometheus.py @@ -0,0 +1,545 @@ +"""Prometheus access. + +The alert Prometheus lives on the internal 10.11/8 network, which is reachable +only from inside the CX-Tools VPN containers - the laptop itself routes 10.11.* +out of its default gateway. So queries go the same way CX-Tools reaches +OpenStack: `docker exec -osc curl ...`. A direct HTTP transport is tried +first so this still works from a host that does have a route. +""" +from __future__ import annotations + +import json +import os +import re +import shlex +import subprocess +import threading +import time +import urllib.parse +import urllib.request +from typing import Any, Optional + +DEFAULT_BASE = os.environ.get("CX_PROMETHEUS_BASE", "http://10.11.254.250:9090") + +# Containers to try as an HTTP relay, in order. These are the CX-Tools +# OpenStack client containers, which share the regional VPN network namespace. +RELAY_CONTAINERS = ("ca1-osc", "us1-osc", "no1-osc", "ca2-osc") + + +class PrometheusError(RuntimeError): + pass + + +class PrometheusClient: + def __init__(self, base: str = DEFAULT_BASE, timeout: int = 20): + self.base = base.rstrip("/") + self.timeout = timeout + self._transport: Optional[tuple[str, str]] = None + self._lock = threading.Lock() + + # --- transport selection ------------------------------------------------ + + def _try_direct(self) -> bool: + try: + req = urllib.request.Request(f"{self.base}/api/v1/status/buildinfo", headers={"User-Agent": "cx-triage"}) + with urllib.request.urlopen(req, timeout=5) as resp: + return 200 <= getattr(resp, "status", 200) < 300 + except Exception: + return False + + def _try_relay(self, container: str) -> bool: + rc, out, _err = _run( + ["docker", "exec", "-i", container, "curl", "-sS", "-m", "6", f"{self.base}/api/v1/status/buildinfo"], + timeout=15, + ) + return rc == 0 and '"status":"success"' in out + + def transport(self) -> tuple[str, str]: + """Return (kind, detail) where kind is 'direct' or 'relay'.""" + with self._lock: + if self._transport is not None: + return self._transport + forced = os.environ.get("CX_PROMETHEUS_RELAY", "").strip() + if forced: + self._transport = ("relay", forced) + return self._transport + if self._try_direct(): + self._transport = ("direct", "host") + return self._transport + for container in RELAY_CONTAINERS: + if self._try_relay(container): + self._transport = ("relay", container) + return self._transport + raise PrometheusError( + f"Cannot reach Prometheus at {self.base}. The host has no route to the internal " + f"network and none of {', '.join(RELAY_CONTAINERS)} answered. Start the CX-Tools " + "VPN/OSC containers, or set CX_PROMETHEUS_RELAY to a container that has a route." + ) + + def describe_transport(self) -> str: + try: + kind, detail = self.transport() + except PrometheusError as exc: + return f"unavailable ({exc})" + return "direct from host" if kind == "direct" else f"relayed through {detail}" + + # --- requests ----------------------------------------------------------- + + def _get(self, path: str, params: Optional[dict[str, str]] = None) -> Any: + url = f"{self.base}{path}" + if params: + url = f"{url}?{urllib.parse.urlencode(params)}" + kind, detail = self.transport() + if kind == "direct": + req = urllib.request.Request(url, headers={"User-Agent": "cx-triage"}) + with urllib.request.urlopen(req, timeout=self.timeout) as resp: + body = resp.read().decode("utf-8", errors="replace") + else: + rc, out, err = _run( + ["docker", "exec", "-i", detail, "curl", "-sS", "-m", str(self.timeout), url], + timeout=self.timeout + 10, + ) + if rc != 0: + raise PrometheusError(f"Prometheus relay via {detail} failed: {(err or out).strip()}") + body = out + try: + data = json.loads(body) + except json.JSONDecodeError as exc: + raise PrometheusError(f"Prometheus returned non-JSON for {path}: {body[:200]}") from exc + if data.get("status") != "success": + raise PrometheusError(f"Prometheus error for {path}: {data.get('error') or data}") + return data.get("data") + + def alerts(self) -> list[dict[str, Any]]: + data = self._get("/api/v1/alerts") or {} + alerts = data.get("alerts") + return [a for a in alerts if isinstance(a, dict)] if isinstance(alerts, list) else [] + + def query(self, expr: str) -> list[dict[str, Any]]: + data = self._get("/api/v1/query", {"query": expr}) or {} + result = data.get("result") + return [r for r in result if isinstance(r, dict)] if isinstance(result, list) else [] + + def resources_by_floating_ip(self, floating_ip: str) -> list[dict[str, Any]]: + """The `Resources{floating_ip="..."}` query the Duplicated IPs runbook uses. + + Unlike CX-Tools (production Infrahub only), this series covers every + environment, so it is how a PreProd/Staging claimant gets found. + """ + expr = 'Resources{floating_ip="%s"}' % floating_ip.replace('"', "") + return [dict(r.get("metric") or {}) for r in self.query(expr)] + + def query_range(self, expr: str, start: int, end: int, step: int) -> list[dict[str, Any]]: + data = self._get("/api/v1/query_range", { + "query": expr, "start": str(start), "end": str(end), "step": str(step), + }) or {} + result = data.get("result") + return [r for r in result if isinstance(r, dict)] if isinstance(result, list) else [] + + def rules(self) -> list[dict[str, Any]]: + data = self._get("/api/v1/rules") or {} + return [g for g in (data.get("groups") or []) if isinstance(g, dict)] + + def series_count(self, metric: str) -> int: + rows = self.query(f"count({metric})") + if not rows: + return 0 + try: + return int(float(rows[0]["value"][1])) + except (KeyError, IndexError, ValueError, TypeError): + return 0 + + +class RuleIndex: + """Maps alertname -> which rule file it came from and its `for` duration. + + Keying off the rule file (not the alert name) is what lets node-exporter + alerts be separated reliably: two different files both use the group name + "Imported Rules". + """ + + def __init__(self, client: PrometheusClient, ttl: float = 600.0): + self.client = client + self.ttl = ttl + self._at = 0.0 + self._by_name: dict[str, dict[str, Any]] = {} + self._error = "" + self._lock = threading.Lock() + + def refresh(self) -> None: + groups = self.client.rules() + index: dict[str, dict[str, Any]] = {} + for group in groups: + source = str(group.get("file") or "").rsplit("/", 1)[-1] + for rule in group.get("rules") or []: + if rule.get("type") != "alerting": + continue + index[str(rule.get("name") or "")] = { + "group": str(group.get("name") or ""), + "file": source, + "for_seconds": int(rule.get("duration") or 0), + "query": str(rule.get("query") or ""), + } + self._by_name = index + + def ensure(self) -> None: + """Refresh if the cache is empty or stale.""" + with self._lock: + if not self._by_name or time.monotonic() - self._at > self.ttl: + try: + self.refresh() + self._error = "" + except PrometheusError as exc: + self._error = str(exc) + self._at = time.monotonic() + + def get(self, alertname: str) -> dict[str, Any]: + self.ensure() + return self._by_name.get(alertname, {}) + + @property + def count(self) -> int: + return len(self._by_name) + + @property + def error(self) -> str: + return self._error + + +# Metrics the alert rules are built on. If one of these is empty, rules that +# depend on it are broken rather than quiet - see StateSnapshot.broken_inputs. +RULE_INPUT_METRICS = ("Resources", "In_Use_Gpus", "Total_Gpus", "openstack_nova_server_status") + +ROGUE_DELTA_EXPR = ( + 'sum by (instance) (In_Use_Gpus) - sum by (instance) ' + '(Resources{organization!="3491 - luis.sarabando+runpod@nexgencloud.coms-Organization",' + 'status=~"ACTIVE|SHUTOFF|PRE_ACTIVE"})' +) + + +def _episode(points: list[tuple[int, float]], median: float) -> dict[str, Any]: + return { + "start": points[0][0], + "end": points[-1][0], + "minutes": max(1, (points[-1][0] - points[0][0]) // 60 + 1), + "low": int(min(v for _, v in points)), + "normal": int(median), + } + + +# Labels that identify one alert across time, for true-age lookup. +TRUE_AGE_KEY_LABELS = ("alertname", "instance_name", "floating_ip", "instance", "openstack_id") + + +def true_age_key(labels: dict[str, Any]) -> tuple: + """Identity of an alert, built from raw label values on both sides.""" + return tuple(str((labels or {}).get(k, "")) for k in TRUE_AGE_KEY_LABELS) + + +class TrueAgeIndex: + """How long each alert's condition has *actually* held. + + Prometheus resets an alert's activeAt whenever the alert resolves, and the + Infrahub metric pipeline drops most of the `Resources` series for a few + minutes several times a day. Every alert alive during such a dip resolves and + re-fires, so activeAt collapses to "time since the last dip" and every alert + reports the same age. + + This walks the `ALERTS` series backwards instead, bridging gaps shorter than + GAP_TOLERANCE, which recovers the real duration. Both pending and firing are + counted, so rules with a long `for:` are not reported as young. + """ + + WINDOW_DAYS = 7 + STEP_SECONDS = 900 + GAP_TOLERANCE = 2700 # 45 min: bridges pipeline dips, not genuine recoveries + + def __init__(self, client: PrometheusClient, severity: str = "infrahub-critical", ttl: float = 300.0): + self.client = client + self.severity = severity + self.ttl = ttl + self._at = 0.0 + self._lock = threading.Lock() + self._starts: dict[tuple, tuple[int, bool]] = {} + self.error = "" + self.window_start = 0 + + def refresh(self) -> None: + end = int(time.time()) + start = end - self.WINDOW_DAYS * 86400 + self.window_start = start + expr = ( + "count by (%s) (ALERTS{severity=\"%s\"})" + % (", ".join(TRUE_AGE_KEY_LABELS), self.severity) + ) + starts: dict[tuple, tuple[int, bool]] = {} + for series in self.client.query_range(expr, start, end, self.STEP_SECONDS): + stamps = [] + for point in series.get("values") or []: + try: + stamps.append(int(float(point[0]))) + except (ValueError, TypeError, IndexError): + continue + if not stamps: + continue + run_start = stamps[-1] + for earlier, later in list(zip(stamps, stamps[1:]))[::-1]: + if later - earlier > self.GAP_TOLERANCE: + break + run_start = earlier + # A run that reaches the window edge is only a lower bound. + capped = run_start <= start + self.STEP_SECONDS + key = true_age_key(series.get("metric") or {}) + existing = starts.get(key) + if existing is None or run_start < existing[0]: + starts[key] = (run_start, capped) + self._starts = starts + + def get(self) -> "TrueAgeIndex": + with self._lock: + if not self._at or time.monotonic() - self._at > self.ttl: + try: + self.refresh() + self.error = "" + except PrometheusError as exc: + self.error = str(exc) + self._at = time.monotonic() + return self + + def lookup(self, labels: dict[str, Any]) -> tuple[Optional[int], bool]: + """Return (minutes the condition has held, whether that is a floor).""" + entry = self._starts.get(true_age_key(labels)) + if not entry: + return None, False + start, capped = entry + return max(0, int((time.time() - start) // 60)), capped + + @property + def loaded(self) -> bool: + return bool(self._starts) + + @property + def count(self) -> int: + return len(self._starts) + + +class StateSnapshot: + """A bulk read of current platform state, used to screen alerts cheaply. + + Re-checking whether an alert's condition still holds is what separates a + real alert from one that already self-resolved. Doing it from these few + aggregate queries costs one Prometheus round trip for the whole queue, + instead of an Infrahub and OpenStack call per alert. + """ + + def __init__(self, client: PrometheusClient, ttl: float = 60.0): + self.client = client + self.ttl = ttl + self._at = 0.0 + self._lock = threading.Lock() + self.error = "" + self.by_openstack_id: dict[str, dict[str, str]] = {} + self.by_instance_name: dict[str, dict[str, str]] = {} + self.fip_counts: dict[str, int] = {} + self.rogue_delta: dict[str, float] = {} + self.total_gpus: dict[str, float] = {} + self.in_use_gpus: dict[str, float] = {} + self.resources_by_host: dict[str, list[dict[str, str]]] = {} + self.broken_inputs: list[str] = [] + # Infrahub VMs in a GPU-counted state with no host recorded. These are + # invisible to the per-host GPU sum the Rogue VM rule uses, so they can + # manufacture a gap on whichever host is actually running them. + self.unattributed_active: int = 0 + self.unattributed_active_gpus: int = 0 + # Episodes where the Resources series partially collapsed. Each one + # resets activeAt on every alert that was live at the time. + self.pipeline_dips: list[dict[str, Any]] = [] + + def refresh(self) -> None: + by_osid: dict[str, dict[str, str]] = {} + by_name: dict[str, dict[str, str]] = {} + fips: dict[str, int] = {} + by_host: dict[str, list[dict[str, str]]] = {} + counted_states = {"ACTIVE", "SHUTOFF", "PRE_ACTIVE"} + unattributed = 0 + unattributed_gpus = 0 + + for row in self.client.query("Resources"): + metric = {str(k): str(v) for k, v in (row.get("metric") or {}).items()} + try: + metric["_gpus"] = str(int(float(row.get("value", [0, "0"])[1]))) + except (ValueError, TypeError, IndexError): + metric["_gpus"] = "0" + osid = metric.get("openstack_id", "") + if osid and osid not in ("None", ""): + by_osid[osid] = metric + name = metric.get("instance_name", "") + if name: + by_name[name] = metric + fip = metric.get("floating_ip", "") + if fip and fip not in ("None", "NULL", ""): + fips[fip] = fips.get(fip, 0) + 1 + host = metric.get("instance", "") + if host and host not in ("Unknown", "None"): + by_host.setdefault(host, []).append(metric) + elif metric.get("status", "").upper() in counted_states: + unattributed += 1 + unattributed_gpus += int(metric["_gpus"] or 0) + + self.by_openstack_id = by_osid + self.by_instance_name = by_name + self.fip_counts = fips + self.resources_by_host = by_host + self.unattributed_active = unattributed + self.unattributed_active_gpus = unattributed_gpus + # Summed by instance: these metrics are per (instance, gpu_name), so a + # host with two GPU models carries two series. Reading them unsummed + # would silently keep only one. + self.rogue_delta = self._scalar_by_instance(ROGUE_DELTA_EXPR) + self.total_gpus = self._scalar_by_instance("sum by (instance) (Total_Gpus)") + self.in_use_gpus = self._scalar_by_instance("sum by (instance) (In_Use_Gpus)") + + self.broken_inputs = [m for m in RULE_INPUT_METRICS if self.client.series_count(m) == 0] + self.pipeline_dips = self._find_pipeline_dips() + + def _find_pipeline_dips(self, hours: int = 24, drop_ratio: float = 0.8) -> list[dict[str, Any]]: + """Find episodes where most of the `Resources` series went missing.""" + end = int(time.time()) + start = end - hours * 3600 + series = self.client.query_range("count(Resources)", start, end, 60) + if not series: + return [] + points: list[tuple[int, float]] = [] + for point in series[0].get("values") or []: + try: + points.append((int(float(point[0])), float(point[1]))) + except (ValueError, TypeError, IndexError): + continue + if len(points) < 10: + return [] + ordered = sorted(v for _, v in points) + median = ordered[len(ordered) // 2] + if median <= 0: + return [] + + episodes: list[dict[str, Any]] = [] + current: list[tuple[int, float]] = [] + for stamp, value in points: + if value < median * drop_ratio: + if current and stamp - current[-1][0] > 180: + episodes.append(_episode(current, median)) + current = [] + current.append((stamp, value)) + elif current: + episodes.append(_episode(current, median)) + current = [] + if current: + episodes.append(_episode(current, median)) + return episodes + + def _scalar_by_instance(self, expr: str) -> dict[str, float]: + out: dict[str, float] = {} + for row in self.client.query(expr): + host = str((row.get("metric") or {}).get("instance") or "") + if not host: + continue + try: + out[host] = float(row.get("value", [0, "0"])[1]) + except (ValueError, TypeError, IndexError): + continue + return out + + def get(self) -> "StateSnapshot": + with self._lock: + if not self._at or time.monotonic() - self._at > self.ttl: + try: + self.refresh() + self.error = "" + except PrometheusError as exc: + self.error = str(exc) + self._at = time.monotonic() + return self + + @property + def loaded(self) -> bool: + return bool(self.by_openstack_id) or bool(self.total_gpus) + + +def _run(cmd: list[str], timeout: int) -> tuple[int, str, str]: + try: + p = subprocess.run(cmd, text=True, capture_output=True, timeout=timeout) + return p.returncode, p.stdout, p.stderr + except subprocess.TimeoutExpired as exc: + return 124, exc.stdout or "", exc.stderr or f"timed out after {timeout}s" + except FileNotFoundError as exc: + return 127, "", str(exc) + except Exception as exc: # pragma: no cover + return 1, "", str(exc) + + +# --- parsing pasted alert text / URLs -------------------------------------- + +_LABEL_RE = re.compile(r'(\w+)\s*=\s*"((?:[^"\\]|\\.)*)"') + + +def parse_alert_text(text: str) -> list[dict[str, str]]: + """Parse pasted `ALERTS{...}` lines, or a Prometheus graph URL, into labels. + + Accepts what a CX engineer actually copies: a raw series line from the + Prometheus console, or the `g0.expr=` URL from the Slack alert. + """ + text = (text or "").strip() + if not text: + return [] + + found: list[dict[str, str]] = [] + + # A pasted graph URL: pull the expression out and parse its label matchers. + if text.startswith("http://") or text.startswith("https://"): + parsed = urllib.parse.urlparse(text) + params = urllib.parse.parse_qs(parsed.query) + exprs = [v for k, vs in params.items() if k.endswith("expr") for v in vs] + for expr in exprs: + labels = {k: _unescape(v) for k, v in _LABEL_RE.findall(expr)} + if labels: + found.append(labels) + return found + + for block in re.findall(r"\{[^{}]*\}", text): + labels = {k: _unescape(v) for k, v in _LABEL_RE.findall(block)} + if labels: + found.append(labels) + if not found: + labels = {k: _unescape(v) for k, v in _LABEL_RE.findall(text)} + if labels: + found.append(labels) + return found + + +def _unescape(value: str) -> str: + return value.replace('\\"', '"').replace("\\\\", "\\").replace("\\n", "\n") + + +class AlertCache: + """Short-lived cache so the UI can poll without hammering Prometheus.""" + + def __init__(self, client: PrometheusClient, ttl: float = 30.0): + self.client = client + self.ttl = ttl + self._at = 0.0 + self._alerts: list[dict[str, Any]] = [] + self._error = "" + self._lock = threading.Lock() + + def get(self, force: bool = False) -> tuple[list[dict[str, Any]], str, float]: + with self._lock: + age = time.monotonic() - self._at + if not force and self._at and age < self.ttl: + return self._alerts, self._error, age + try: + self._alerts = self.client.alerts() + self._error = "" + except PrometheusError as exc: + self._error = str(exc) + self._at = time.monotonic() + return self._alerts, self._error, 0.0 diff --git a/triagelib/runbooks.py b/triagelib/runbooks.py new file mode 100644 index 0000000..5460e43 --- /dev/null +++ b/triagelib/runbooks.py @@ -0,0 +1,1497 @@ +"""The diagnosis engine: CX runbooks expressed as decisions over CX-Tools data. + +Each alert kind maps to one Confluence runbook. For every alert this module +gathers the evidence the runbook asks for (via the read-only CX-Tools +collectors), reaches the verdict the runbook's decision table implies, and emits +the remaining steps a human still has to perform. Nothing here mutates state. +""" +from __future__ import annotations + +import re +from dataclasses import dataclass, field +from typing import Any, Optional + +from . import comms, cxbridge +from .alerts import Alert + +CX = "CX" +INFRA = "Infrastructure team" +DEVOPS = "DevOps" +CUSTOMER = "Customer" + +TRANSITIONAL = {"BUILD", "CREATING", "DELETING", "HIBERNATING", "RESTORING", "REBOOTING"} + + +# --- result model ----------------------------------------------------------- + +@dataclass +class Finding: + label: str + value: str + tone: str = "info" # ok | warn | bad | info + detail: str = "" + + def to_json(self) -> dict[str, Any]: + return {"label": self.label, "value": self.value, "tone": self.tone, "detail": self.detail} + + +@dataclass +class Action: + text: str + owner: str = CX + kind: str = "check" # check | remediate | escalate | comms | verify + guide: str = "" + status: str = "todo" # todo | done | skipped (done = the app verified it) + detail: str = "" + + def to_json(self) -> dict[str, Any]: + return { + "text": self.text, "owner": self.owner, "kind": self.kind, + "guide": self.guide, "status": self.status, "detail": self.detail, + } + + +@dataclass +class Diagnosis: + alert: Alert + verdict: str = "" + assessment: str = "" + confidence: str = "medium" # high | medium | low + findings: list[Finding] = field(default_factory=list) + actions: list[Action] = field(default_factory=list) + drafts: list[comms.Draft] = field(default_factory=list) + contacts: dict[str, Any] = field(default_factory=dict) + evidence: dict[str, Any] = field(default_factory=dict) + notes: list[str] = field(default_factory=list) + error: str = "" + # A small structured description of the problem, so the UI can draw it + # rather than make the reader parse a table. + visual: dict[str, Any] = field(default_factory=dict) + agent_name: str = "" + + def add(self, label: str, value: Any, tone: str = "info", detail: str = "") -> None: + text = str(value if value not in (None, "") else "N/A") + self.findings.append(Finding(label, text, tone, detail)) + + def act(self, text: str, owner: str = CX, kind: str = "check", guide: str = "", + status: str = "todo", detail: str = "") -> None: + self.actions.append(Action(text, owner, kind, guide, status, detail)) + + def note(self, text: str) -> None: + if text and text not in self.notes: + self.notes.append(text) + + def to_json(self) -> dict[str, Any]: + from . import integrations + + return { + "integrations": integrations.build_all(self), + "alert": self.alert.to_json(), + "verdict": self.verdict, + "assessment": self.assessment, + "confidence": self.confidence, + "findings": [f.to_json() for f in self.findings], + "actions": [a.to_json() for a in self.actions], + "drafts": [d.to_json() for d in self.drafts], + "contacts": self.contacts, + "evidence": self.evidence, + "notes": self.notes, + "error": self.error, + "visual": self.visual, + } + + +# --- the ERROR-state fault table ------------------------------------------- +# Transcribed from the fault table in the "Instance in ERROR state" runbook. + +FAULT_TABLE: list[dict[str, Any]] = [ + { + "id": "conflict_image_pending_upload", + "pattern": r"Conflict updating instance.*task_state.*image_pending_upload", + "summary": "Instance was deleted while it was hibernating.", + "owner": f"{INFRA} / {CX}", + "guidance": "Known case: the instance was deleted mid-hibernation. It just needs deleting again in OpenStack.", + }, + { + "id": "no_valid_host", + "pattern": r"No valid host was found|not enough hosts available", + "summary": "The scheduler could not place the instance - stock shortage or a race condition.", + "owner": INFRA, + "guidance": "Judge whether there was enough stock at the time; ask a peer if unsure. If there should have been " + "enough, try reproducing with the same flavor in pre-prod - otherwise it is most likely a race " + "condition from several near-simultaneous API creates.", + "comms": "error_never_active", + "comms_note": "Contact the customer if the instance is still not deleted after a day.", + }, + { + "id": "numa_pci", + "pattern": r"NUMA topology together with requested PCI devices|Claim pci failed", + "summary": "NUMA/PCI claim failure on the host.", + "owner": INFRA, + "guidance": "Check whether the host is full BEFORE escalating. If the summed GPU count across the host's " + "instances is 8, the host is full and that proves the host itself is fine.", + "gpu_census": True, + }, + { + "id": "lvremove", + "pattern": r"Failed to remove volume\(s\)|lvremove", + "summary": "A stale LVM volume on the host blocked the build.", + "owner": INFRA, + "guidance": "Raise a Jira for Infrastructure to investigate the host and the instance error.", + }, + { + "id": "pci_header", + "pattern": r"Unknown PCI header type", + "summary": "Host PCI device is reporting an invalid header - hardware/host fault.", + "owner": INFRA, + "guidance": "Raise a Jira for Infrastructure to investigate the host and the instance error.", + }, + { + "id": "pci_in_use", + "pattern": r"PCI device \S+ is in use by driver QEMU", + "summary": "The GPU is still claimed by another domain on the host.", + "owner": INFRA, + "guidance": "Raise a Jira for Infrastructure to investigate the host and the instance error.", + }, + { + "id": "client_socket_closed", + "pattern": r"internal error: client socket is closed", + "summary": "libvirt client socket closed during the operation.", + "owner": INFRA, + "guidance": "Can be transient, but it is associated with host issues - escalate to be safe.", + }, +] + + +def match_faults(texts: list[str]) -> list[dict[str, Any]]: + """Match collected fault text against the runbook fault table.""" + blob = "\n".join(t for t in texts if t) + matched: list[dict[str, Any]] = [] + for entry in FAULT_TABLE: + if re.search(entry["pattern"], blob, re.I | re.S): + matched.append(entry) + return matched + + +# --- shared evidence gathering --------------------------------------------- + +def _vm_target(alert: Alert) -> tuple[str, Optional[str]]: + """Pick the best CX-Tools lookup target for this alert. + + OpenStack ID is preferred. CREATING alerts routinely carry + openstack_id="None" because the VM never reached OpenStack, so those fall + back to the instance name plus the org id parsed from the alert. + """ + if alert.openstack_id: + return alert.openstack_id, None + if alert.instance_name and alert.org_id: + return alert.instance_name, alert.org_id + return alert.instance_name, None + + +def _tone_for_status_pair(ih_status: str, os_status: str, task_state: str) -> str: + try: + ok = cxbridge.cx().status_pair_ok(ih_status, os_status, task_state) + except Exception: + ok = ih_status.upper() == os_status.upper() + return "ok" if ok else "bad" + + +def _add_vm_findings(d: Diagnosis, vm: dict[str, Any]) -> None: + ih_status = str(vm.get("ih_status") or "N/A") + os_status = str(vm.get("os_status") or "N/A") + task_state = str(vm.get("task_state") or "None") + host = str(vm.get("host") or "N/A") + + d.visual = { + "type": "states", + "infrahub": ih_status, + "openstack": os_status, + "task": task_state, + "match": _tone_for_status_pair(ih_status, os_status, task_state) == "ok", + "host": host, + "name": str(vm.get("name") or ""), + "flavor": str(vm.get("flavor") or ""), + "gpus": str(vm.get("gpu_count") or ""), + "fault": str(vm.get("openstack_fault") or "None"), + "never_built": host in ("N/A", "Unknown", ""), + } + + d.add("Infrahub ID", vm.get("infrahub_id")) + d.add("OpenStack ID", vm.get("openstack_id")) + d.add("Name", vm.get("name")) + d.add("Region", vm.get("region_display") or vm.get("region")) + d.add("Infrahub status", ih_status, "info") + d.add("OpenStack status", os_status, + _tone_for_status_pair(ih_status, os_status, task_state) if "N/A" not in (ih_status, os_status) else "warn") + d.add("Task state", task_state, "warn" if task_state not in ("None", "N/A") else "info") + d.add("Hypervisor", host, "warn" if host in ("N/A", "Unknown") else "info") + d.add("Flavor", vm.get("flavor")) + d.add("GPU(s)", vm.get("gpu_count")) + d.add("Floating IP", vm.get("floating_ip")) + d.add("Created", vm.get("created")) + d.add("SSH", vm.get("ssh_text"), "ok" if vm.get("ssh_raw") == "reachable" else "info") + d.add("Volumes", vm.get("volumes_summary")) + if vm.get("project_name"): + d.add("OpenStack project", vm.get("project_name")) + if vm.get("project_environment"): + d.add("Environment", vm.get("project_environment"), "warn", + "Found outside production - Infrahub production records will not match.") + + os_fault = str(vm.get("openstack_fault") or "None") + d.add("OpenStack fault", os_fault, "bad" if os_fault not in ("None", "N/A") else "ok") + faults = vm.get("faults") or [] + d.add("InfraInsight faults", f"{len(faults)} recorded" if faults else "None", "bad" if faults else "ok") + + for item in vm.get("mismatches") or []: + check, detail = cxbridge.cx().mismatch_parts(item) + d.add(f"Mismatch: {check}", detail, "bad") + + for reason in vm.get("warn_reasons") or []: + d.add("Warning", str(reason), "warn") + for note in vm.get("info_notes") or []: + d.note(str(note)) + + +def _fault_texts(vm: dict[str, Any], failed_event: dict[str, Any]) -> list[str]: + texts: list[str] = [] + os_fault = str(vm.get("openstack_fault") or "") + if os_fault and os_fault not in ("None", "N/A"): + texts.append(os_fault) + for row in vm.get("faults") or []: + texts.extend(str(x) for x in row) + server = vm.get("server") if isinstance(vm.get("server"), dict) else {} + fault = server.get("fault") + if isinstance(fault, dict): + texts.append(str(fault.get("message") or "")) + texts.append(str(fault.get("details") or "")) + elif fault: + texts.append(str(fault)) + rows = failed_event.get("rows") or {} + for key in ("Detail", "Traceback", "Result"): + if rows.get(key) and rows[key] != "N/A": + texts.append(str(rows[key])) + return texts + + +def _host_health_findings(d: Diagnosis, region: str, host: str) -> dict[str, Any]: + """Run the cheap host signals and turn them into findings + a verdict.""" + if not host or host in ("N/A", "Unknown") or not region: + d.act("Identify the hypervisor, then run Host Health Checks manually.", CX, "check", "Host Health Checks") + d.note("No hypervisor was recorded on this alert, so host health could not be checked automatically.") + return {} + + health = cxbridge.host_health(region, host) + if not health.get("ok"): + d.add("Host health", f"lookup failed: {health.get('error', '')}", "warn") + d.act(f"Check host {host} manually.", CX, "check", "Host Health Checks") + return health + + nova_state = str(health.get("nova_state") or "") + nova_status = str(health.get("nova_status") or "") + ovs_alive = health.get("ovs_alive") + ovs_state = str(health.get("ovs_state") or "") + + bad_signals: list[str] = [] + d.add("Host", host) + d.add("Nova state", nova_state, "ok" if nova_state.lower() == "up" else "bad") + if nova_state.lower() != "up": + bad_signals.append(f"Nova state is {nova_state}") + d.add("Nova status", nova_status, "ok" if nova_status.lower() == "enabled" else "bad") + if nova_status.lower() != "enabled": + bad_signals.append(f"Nova status is {nova_status}") + if health.get("disabled_reason"): + d.add("Disabled reason", health["disabled_reason"], "bad") + bad_signals.append(f"disabled: {health['disabled_reason']}") + if ovs_alive is not None: + d.add("OVS alive", str(ovs_alive), "ok" if ovs_alive in (True, "True", "true") else "bad") + if ovs_alive not in (True, "True", "true"): + bad_signals.append("OVS agent is not alive") + if ovs_state: + d.add("OVS state", ovs_state, "ok" if ovs_state.lower() == "up" else "bad") + if ovs_state.lower() != "up": + bad_signals.append(f"OVS state is {ovs_state}") + d.add("Host uptime", health.get("uptime")) + d.add("Aggregates", health.get("aggregates")) + + health["bad_signals"] = bad_signals + if bad_signals: + d.act( + f"Host {host} shows problems ({'; '.join(bad_signals)}) - escalate to the Infrastructure team.", + INFRA, "escalate", "Host Health Checks", + ) + else: + d.act( + f"Nova and OVS signals on {host} look healthy; complete the remaining Host Health Checks " + "(disk/dmesg/GPU checks the guide covers and this app does not).", + CX, "check", "Host Health Checks", + ) + return health + + +def _vm_draft(d: Diagnosis, template: str, vm: dict[str, Any], alert: Alert, + *, floating_ip: str = "", note: Optional[str] = None) -> Optional[comms.Draft]: + """Build a customer draft with the VM identity the house style expects.""" + owners = (vm.get("owners") or []) if isinstance(vm, dict) else [] + return comms.draft( + template, + instance_name=str(vm.get("name") or alert.instance_name or ""), + infrahub_id=str(vm.get("infrahub_id") or ""), + openstack_id=str(vm.get("openstack_id") or alert.openstack_id or ""), + greeting_name=comms.first_name(owners[0] if owners else ""), + agent_name=d.agent_name, + floating_ip=floating_ip, + note=note, + ) + + +def _attach_contacts(d: Diagnosis, vm: dict[str, Any]) -> None: + d.contacts = comms.contacts_from_result(vm) + if not d.contacts.get("resolved"): + d.note( + "No organization owner contacts came back from Infrahub. Look the organization up in the Admin " + "Portal before sending anything." + ) + + +def _common_preamble(d: Diagnosis, alert: Alert) -> None: + if alert.is_kubernetes: + d.note( + f"Instance name '{alert.instance_name}' looks like a Kubernetes node. Per the general process, treat it " + "as part of a K8s cluster - deleting a single node may be handled by the cluster instead." + ) + if alert.threshold_min and alert.age_minutes is not None and alert.age_minutes < alert.threshold_min: + d.note( + f"This alert has only been active {alert.age_minutes} min against a {alert.threshold_min} min threshold; " + "it may still clear on its own." + ) + + +# --- per-kind runbooks ----------------------------------------------------- + +def _diagnose_error(d: Diagnosis, alert: Alert, vm: dict[str, Any]) -> None: + """Instance in ERROR state.""" + host = str(vm.get("host") or "N/A") + os_status = str(vm.get("os_status") or "N/A") + ih_status = str(vm.get("ih_status") or "N/A") + region = str(vm.get("region") or alert.region) + osid = str(vm.get("openstack_id") or alert.openstack_id) + + failed_event: dict[str, Any] = {} + if osid and osid != "N/A" and region: + failed_event = cxbridge.failed_openstack_event(region, osid) + if failed_event: + rows = failed_event.get("rows") or {} + d.add("Failed OpenStack event", f"{failed_event.get('action', '?')} ({failed_event.get('request_id', '')})", "bad", + rows.get("Detail", "")) + d.evidence["failed_event"] = failed_event + + # "We are more concerned if a VM goes into ERROR and it was previously + # ACTIVE" - no hypervisor means it never fully created. + was_active = host not in ("N/A", "Unknown", "") + d.add("Reached ACTIVE previously", "yes - a hypervisor is recorded" if was_active else "no - never placed on a host", + "bad" if was_active else "warn", + "" if was_active else "No hostname means the instance was never created fully and could not have become ACTIVE.") + + texts = _fault_texts(vm, failed_event) + matches = match_faults(texts) + d.evidence["fault_text"] = [t for t in texts if t][:12] + + for row in vm.get("faults") or []: + d.add("InfraInsight fault", " | ".join(str(x) for x in row), "bad") + + if matches: + names = "; ".join(m["summary"] for m in matches) + d.verdict = f"ERROR with a known fault: {names}" + d.confidence = "high" + for m in matches: + d.act(m["guidance"], m["owner"], "escalate" if m["owner"] != CX else "remediate") + if m.get("gpu_census") and host not in ("N/A", "Unknown") and region: + census = cxbridge.host_gpu_census(region, host) + d.evidence["gpu_census"] = census + if census.get("ok"): + total = census.get("total_gpus") + full = total is not None and total >= 8 + d.add("GPUs allocated on host", f"{total} (from {len(census.get('instances', []))} instances)", + "ok" if full else "warn", + "Host is FULL - this proves the host itself is fine." if full + else "Host is not full, so the NUMA/PCI failure is not simple capacity. Raise a Jira for Infra.") + if full: + d.act("Host is full (8 GPUs allocated) - this proves the host is fine. No Infra escalation needed " + "for capacity; treat as a race condition.", CX, "verify", status="done") + else: + d.act(f"Raise a Jira for the Infrastructure team to investigate host {host} and the instance error.", + INFRA, "escalate") + else: + d.verdict = "ERROR with no fault matching the runbook table" + d.confidence = "low" if not texts else "medium" + d.act( + "Fault is not in the runbook table - escalate to a peer, and add the fault to the runbook table once known.", + CX, "escalate", + ) + + if ih_status.upper() == "ERROR" and os_status.upper() not in ("ERROR", "N/A"): + d.add("Note", f"Infrahub says ERROR but OpenStack says {os_status}", "warn") + + d.assessment = ( + "The instance had reached ACTIVE, so customer data may be on it - remediate rather than leaving it. " + if was_active else + "The instance never became ACTIVE, so there is no customer data to protect; it needs deleting and recreating. " + ) + if not was_active: + d.assessment += "Customers usually just delete ERROR VMs and retry." + + d.act("Record the fault message and event output in the Slack alert thread and any HubSpot ticket.", CX, "verify") + + created = str(vm.get("created") or "") + template = "error_was_active" if was_active else "error_never_active" + note = None + for m in matches: + if m.get("comms"): + template = m["comms"] + note = m.get("comms_note") + draft = _vm_draft(d, template, vm, alert, note=note) + if draft: + if not was_active: + draft.when += " Outreach is optional and only if the instance is less than 7 days old." + if created and created != "N/A": + draft.when += f" Created: {created}." + d.drafts.append(draft) + + +def _diagnose_deleting(d: Diagnosis, alert: Alert, vm: dict[str, Any]) -> None: + """Instance in DELETING state.""" + ih_status = str(vm.get("ih_status") or "N/A") + os_status = str(vm.get("os_status") or "N/A") + server_exists = bool(vm.get("server")) + + d.verdict = f"Stuck in DELETING for {alert.age_minutes or '?'} min - unlikely to finish on its own" + d.confidence = "high" + d.assessment = ( + "The customer's intent is clear (they asked for a delete), so the remediation is to finish the delete. " + ) + + delete_requests = [ + row for row in (vm.get("ih_events_all") or vm.get("ih_events") or []) + if "delete" in " ".join(str(x) for x in row).lower() + ] + d.evidence["delete_requests"] = delete_requests[:5] + if delete_requests: + d.add("Delete request found", delete_requests[0][0], "ok", " | ".join(str(x) for x in delete_requests[0][1:])) + d.act("Delete request confirmed in Infrahub events - customer intent verified.", CX, "verify", status="done") + else: + d.add("Delete request found", "none in Infrahub events", "warn") + d.act( + "Confirm who requested the deletion via the InfraInsight InstanceDeleteRequest query. If no record " + "exists, escalate to DevOps - it could be an admin action or an anomaly.", + DEVOPS, "check", + ) + d.note( + "Requester attribution (name/email) comes from the InfraInsight SQL query in the runbook; CX-Tools " + "exposes the Infrahub event but not the requesting user." + ) + + never_built = str(vm.get("host") or "N/A") in ("N/A", "Unknown", "") + + if server_exists: + d.add("OpenStack server", f"still present ({os_status})", "bad") + d.act(f"Delete the server in OpenStack ({vm.get('openstack_id')}).", CX, "remediate", "Deleting an Instance") + if never_built: + # It reached OpenStack but never landed on a host, so this is not a + # normal delete: the build failed and the record still needs closing. + d.add("Reached a host", "no - the build never completed", "warn", + "The instance failed to build, so the customer's delete could never finish normally.") + d.assessment += ( + "The instance never made it onto a host, so removing the OpenStack server is only half of it - " + "the Infrahub record has to be closed out too. " + ) + else: + d.add("OpenStack server", "already gone", "ok") + d.assessment += ( + "The server is already gone from OpenStack, so this is an Infrahub-side record that never closed out. " + ) + + # Deleting the OpenStack server does not clear the Infrahub record. Whatever + # broke the delete the first time will still leave it in DELETING, and it is + # the record that keeps the alert firing and the resource on the books. + if ih_status.upper() == "DELETING": + d.act( + "Mark the instance deleted in InfraInsight so the Infrahub record closes out - removing the OpenStack " + "server alone leaves it stuck in DELETING and the alert still firing.", + CX, "remediate", "Infrahub Insights (Infra-Insight)", + ) + else: + d.add("Note", f"Infrahub now reports {ih_status}, not DELETING - the alert may already be stale", "warn") + d.confidence = "medium" + + d.act("Confirm the record is gone from Admin Portal Production and the alert clears.", CX, "verify") + + for template in ("deleting", "deleting_resolved"): + made = _vm_draft(d, template, vm, alert) + if made: + d.drafts.append(made) + d.note( + "The customer asked for this delete, so contact is optional - but your sent examples show CX does confirm " + "it. Two drafts are offered: a proactive notice, and a reply that closes an existing ticket." + ) + + +def _diagnose_shutoff(d: Diagnosis, alert: Alert, vm: dict[str, Any]) -> None: + """Instance in SHUTOFF state.""" + os_status = str(vm.get("os_status") or "N/A") + d.verdict = "Customer-initiated SHUTOFF - billing awareness notice required" + d.confidence = "high" + d.assessment = ( + "SHUTOFF is a result of customer action, not a platform fault. The only job here is making sure the " + "customer knows a SHUTOFF VM still accrues full cost." + ) + if os_status.upper() == "SHUTOFF": + d.act("Confirmed SHUTOFF in OpenStack.", CX, "verify", status="done") + else: + d.add("Note", f"OpenStack reports {os_status}, not SHUTOFF - the alert may be stale", "warn") + d.confidence = "medium" + d.act("Create a HubSpot ticket and send the billing awareness note (snippet: #shutoff).", CX, "comms", + "HubSpot Ticket Creation") + draft = _vm_draft(d, "shutoff", vm, alert) + if draft: + d.drafts.append(draft) + + +def _diagnose_hibernating(d: Diagnosis, alert: Alert, vm: dict[str, Any]) -> None: + """Instance in HIBERNATING state.""" + region = str(vm.get("region") or alert.region) + host = str(vm.get("host") or alert.host or "N/A") + d.verdict = "Stuck HIBERNATING - almost certainly a host issue" + d.confidence = "high" + d.assessment = ( + "The runbook treats stuck HIBERNATING as almost certainly a host problem. Check the host, then drive the " + "shelve to completion." + ) + health = _host_health_findings(d, region, host) + d.evidence["host_health"] = health + d.act("Complete the shelve for the instance so it lands in SHELVED_OFFLOADED.", CX, "remediate", + "Shelving an Instance") + d.act("Confirm the instance ends up HIBERNATED in Infrahub and SHELVED_OFFLOADED in OpenStack.", CX, "verify") + if health.get("bad_signals"): + d.verdict = f"Stuck HIBERNATING due to a host problem on {host}" + d.note("This runbook has no customer comms step of its own.") + + +def _diagnose_creating(d: Diagnosis, alert: Alert, vm: dict[str, Any]) -> None: + """Instance in CREATING state.""" + osid = str(vm.get("openstack_id") or "") + has_osid = bool(osid) and osid != "N/A" + server_exists = bool(vm.get("server")) + + d.confidence = "high" + if not has_osid: + d.verdict = "Stuck CREATING and never got an OpenStack ID - the VM does not exist in OpenStack" + d.add("OpenStack ID", "never assigned", "bad", + "The instance never reached OpenStack, so it cannot be recovered and must be recreated.") + d.assessment = ( + "The instance never got an OpenStack ID, so nothing was ever built. The customer cannot delete it " + "themselves in this transitional state - CX has to clear it and tell them to retry." + ) + elif not server_exists: + d.verdict = "Stuck CREATING with an OpenStack ID, but no server exists in OpenStack" + d.assessment = "Infrahub holds an OpenStack ID that OpenStack does not know about; the record needs clearing." + else: + d.verdict = f"Stuck CREATING while OpenStack reports {vm.get('os_status')}" + d.confidence = "medium" + d.assessment = "The server does exist in OpenStack, so this is a sync failure rather than a failed build." + + d.act("Confirm the instance and its Infrahub ID in the Admin Portal (Billing -> Manage Organization -> Resources).", + CX, "check") + d.act("Delete the stuck instance - it cannot be recovered in this state.", CX, "remediate", "Deleting an Instance") + d.act("Create a HubSpot ticket and tell the customer they can retry deploying.", CX, "comms", + "HubSpot Ticket Creation") + draft = _vm_draft(d, "creating", vm, alert) + if draft: + d.drafts.append(draft) + + +def _diagnose_restoring(d: Diagnosis, alert: Alert, vm: dict[str, Any]) -> None: + """Instance in RESTORING state.""" + region = str(vm.get("region") or alert.region) + host = str(vm.get("host") or alert.host or "N/A") + osid = str(vm.get("openstack_id") or "") + + d.verdict = "Stuck RESTORING - unshelve did not complete" + d.confidence = "high" + d.assessment = ( + "A stuck restore blocks the customer from reaching their data. Larger images can be slow, but past an hour " + "this needs investigating." + ) + health = _host_health_findings(d, region, host) + d.evidence["host_health"] = health + + failed_event = cxbridge.failed_openstack_event(region, osid) if osid and osid != "N/A" and region else {} + if failed_event: + rows = failed_event.get("rows") or {} + d.add("Failed OpenStack event", f"{failed_event.get('action', '?')} ({failed_event.get('request_id', '')})", + "bad", rows.get("Detail", "")) + d.evidence["failed_event"] = failed_event + d.act("Escalate the failed unshelve event and its fault to the Infrastructure team.", INFRA, "escalate") + elif not health.get("bad_signals"): + d.add("Failed OpenStack event", "none found in recent events", "warn") + d.act("Analyze the unshelve event in OpenStack for fault information, then escalate it to Infrastructure.", + INFRA, "escalate") + + d.act("Once resolved, shelve the instance so it returns to its pre-restore state and the customer can retry.", + CX, "remediate", "Shelving an Instance") + d.act("Validate the instance is SHELVED_OFFLOADED in OpenStack.", CX, "verify") + d.act("Set the instance state back to HIBERNATED in InfraInsight.", CX, "remediate", + "Infrahub Insights (Infra-Insight)") + d.act("Create a HubSpot ticket telling the customer they can retry restoring.", CX, "comms", + "HubSpot Ticket Creation") + draft = _vm_draft(d, "restoring", vm, alert) + if draft: + d.drafts.append(draft) + + +def _diagnose_rebooting(d: Diagnosis, alert: Alert, vm: dict[str, Any]) -> None: + """Instance in REBOOTING state.""" + region = str(vm.get("region") or alert.region) + host = str(vm.get("host") or alert.host or "N/A") + os_status = str(vm.get("os_status") or "N/A") + task_state = str(vm.get("task_state") or "None") + + d.verdict = "Stuck REBOOTING - the reboot did not complete" + d.confidence = "high" + d.assessment = "Instances get stuck rebooting from host problems; confirm the reboot request, then escalate." + + health = _host_health_findings(d, region, host) + d.evidence["host_health"] = health + + reboot_events = [ + row for row in (vm.get("ih_events_all") or vm.get("ih_events") or []) + if "reboot" in " ".join(str(x) for x in row).lower() + ] + d.evidence["reboot_requests"] = reboot_events[:5] + if reboot_events: + d.add("InstanceRebootRequest", reboot_events[0][0], "ok", " | ".join(str(x) for x in reboot_events[0][1:])) + d.act("Reboot request confirmed in Infrahub events.", CX, "verify", status="done") + else: + d.add("InstanceRebootRequest", "not found in Infrahub events", "warn") + d.act("Confirm the instance received a HARD_REBOOT request (event InstanceRebootRequest) in the Admin Portal.", + CX, "check") + + expected = os_status.upper() == "HARD_REBOOT" or "REBOOT" in task_state.upper() + d.add("OpenStack reboot state", f"status={os_status}, task_state={task_state}", "ok" if expected else "warn", + "" if expected else "The runbook expects status HARD_REBOOT while a reboot is in flight.") + + failed_event = cxbridge.failed_openstack_event(region, str(vm.get("openstack_id") or "")) \ + if vm.get("openstack_id") not in (None, "", "N/A") and region else {} + if failed_event: + rows = failed_event.get("rows") or {} + d.add("Failed OpenStack event", f"{failed_event.get('action', '?')} ({failed_event.get('request_id', '')})", + "bad", rows.get("Detail", "")) + d.evidence["failed_event"] = failed_event + + d.act("Escalate the instance event to the Infrastructure team for review.", INFRA, "escalate") + d.act("Once cleared, confirm the instance is ACTIVE in both Infrahub and OpenStack.", CX, "verify") + d.act("Create a HubSpot ticket telling the customer they can retry rebooting.", CX, "comms", + "HubSpot Ticket Creation") + draft = _vm_draft(d, "rebooting", vm, alert) + if draft: + d.drafts.append(draft) + + +def _diagnose_build(d: Diagnosis, alert: Alert, vm: dict[str, Any]) -> None: + """Instance in BUILD state.""" + region = str(vm.get("region") or alert.region) + host = str(vm.get("host") or alert.host or "N/A") + os_status = str(vm.get("os_status") or "N/A") + age = alert.age_minutes + + gpus = str(vm.get("gpu_count") or "") + large = gpus.isdigit() and int(gpus) >= 4 + + if age is not None and age < 60 and large: + d.verdict = f"BUILD for {age} min on a large flavor - may still be transient" + d.confidence = "medium" + d.assessment = ( + "Larger flavors (more GPUs/storage) can take a while to build and often resolve on their own. " + "The runbook says to investigate host/infrastructure problems once it is stuck past an hour." + ) + else: + d.verdict = f"Stuck in BUILD{f' for {age} min' if age is not None else ''} - investigate host/infrastructure" + d.confidence = "high" + d.assessment = "Past roughly an hour in BUILD this is a host or infrastructure problem, not slow provisioning." + + health = _host_health_findings(d, region, host) + d.evidence["host_health"] = health + + d.add("OpenStack status", os_status, "ok" if os_status.upper() == "BUILD" else "warn", + "" if os_status.upper() == "BUILD" else "The runbook expects the instance to be in status BUILD.") + + failed_event = cxbridge.failed_openstack_event(region, str(vm.get("openstack_id") or "")) \ + if vm.get("openstack_id") not in (None, "", "N/A") and region else {} + if failed_event: + rows = failed_event.get("rows") or {} + d.add("Failed OpenStack event", f"{failed_event.get('action', '?')} ({failed_event.get('request_id', '')})", + "bad", rows.get("Detail", "")) + d.evidence["failed_event"] = failed_event + + d.act("Escalate to the Infrastructure team with the server event and server show output.", INFRA, "escalate") + d.act( + "Tell the customer the instance must be deleted and recreated. It will eventually transition to ERROR, " + "which lets them delete it themselves; otherwise InfraInsight can delete the Infrahub record (Infra may need " + "to delete it in OpenStack while it is stuck in BUILD).", + CX, "comms", "HubSpot Ticket Creation", + ) + draft = _vm_draft(d, "build", vm, alert) + if draft: + d.drafts.append(draft) + + +# --- rogue VM -------------------------------------------------------------- + +MISMATCH_TABLE: list[dict[str, Any]] = [ + { + "ih": "HIBERNATED", "os": "SHUTOFF", + "verdict": "Hibernation did not complete - commonly host disk capacity", + "steps": [ + (CX, "Run Host Health Checks on the hypervisor.", "Host Health Checks"), + (CX, "Run the Windmill workflow to clean stale images on the host and free space for hibernation.", ""), + (CX, "Complete the shelve for the instance.", "Shelving an Instance"), + ], + "comms": None, + }, + { + "ih": "HIBERNATED", "os": "ACTIVE", + "verdict": "Infrahub thinks the VM is hibernated but it is still running and billable", + "steps": [ + (CX, "Run Host Health Checks on the hypervisor.", "Host Health Checks"), + (CX, "If the host shows no visible issues, shelve the instance.", "Shelving an Instance"), + (CX, "Create a HubSpot ticket for the state mismatch.", "HubSpot Ticket Creation"), + ], + "comms": "sync_state", + }, + { + "ih": "DELETING", "os": "*", + "verdict": "Infrahub is stuck DELETING - follow the DELETING runbook", + "steps": [(CX, "Follow the Instance in DELETING state runbook.", "Instance in DELETING state")], + "comms": None, + }, + { + "ih": "RESTORING", "os": "*", + "verdict": "Infrahub is stuck RESTORING - follow the RESTORING runbook", + "steps": [(CX, "Follow the Instance in RESTORING state runbook.", "Instance in RESTORING state")], + "comms": None, + }, +] + + +def _match_mismatch_table(ih_status: str, os_status: str) -> Optional[dict[str, Any]]: + ih = (ih_status or "").upper() + os_ = (os_status or "").upper() + if ih == "ERROR" or os_ == "ERROR": + return { + "ih": ih, "os": os_, + "verdict": "ERROR on one side - follow the ERROR runbook", + "steps": [(CX, "Follow the Instance in ERROR state runbook.", "Instance in ERROR state")], + "comms": None, + } + for entry in MISMATCH_TABLE: + if entry["ih"] == ih and entry["os"] in ("*", os_): + return entry + return None + + +def _gpu_slots(visual: dict[str, Any], roster: list[dict[str, Any]]) -> list[dict[str, Any]]: + """One entry per physical GPU on the host. + + A host has a fixed number of GPU sockets. Every one of them is in exactly + one of three states, and showing all of them is the only way the arithmetic + reads as complete: + + * claimed by a named VM - one slot per GPU that VM holds + * in use but unclaimed - the host says busy, no instance says so + * free - physically present, nothing using it + + Drawing only the first two (as the earlier bar did) drops the free sockets + and leaves a total that does not match the host's GPU count. + """ + physical = visual.get("physical") + in_use = visual.get("in_use_metric") + artifact = bool(visual.get("spare_capacity_artifact")) + + slots: list[dict[str, Any]] = [] + for vm in roster: + try: + count = int(str(vm.get("gpus") or "0")) + except ValueError: + count = 0 + for _ in range(max(0, count)): + slots.append({ + "kind": "vm", + "name": vm["name"], + "linked": vm["linked"], + "match": vm["match"], + "ih_status": vm["ih_status"], + "os_status": vm["os_status"], + }) + + named = len(slots) + if in_use is None: + return slots + + # Sockets the host counts as busy that no instance accounts for. When + # In_Use_Gpus is really just the physical count, that difference is spare + # capacity rather than a missing workload, so label it honestly. + unaccounted = max(0, int(in_use) - named) + for _ in range(unaccounted): + slots.append({"kind": "free" if artifact else "unaccounted"}) + + if physical is not None: + for _ in range(max(0, int(physical) - max(int(in_use), named))): + slots.append({"kind": "free"}) + return slots + + +def _diagnose_rogue_vm(d: Diagnosis, alert: Alert) -> None: + """Suspected Rogue VM - host-wide Infrahub/OpenStack reconciliation.""" + host = alert.host or alert.instance_name + region = alert.region + + if not host: + d.error = "This alert carries no hypervisor, so there is nothing to reconcile." + return + if not region: + d.error = f"Could not derive a CX-Tools region from host '{host}'." + return + + result = cxbridge.collect_host(host) + d.evidence["host_result"] = cxbridge.json_safe(result) + if not result.get("ok"): + d.error = str(result.get("error") or f"Host collection failed for {host}.") + return + + hv = result.get("hypervisor") if isinstance(result.get("hypervisor"), dict) else {} + ovs = result.get("ovs") if isinstance(result.get("ovs"), dict) else {} + c = cxbridge.cx() + + d.add("Host", host) + d.add("Region", region) + d.add("Instances on host", result.get("server_count")) + d.add("Nova state", c.normalize_empty(c.first_present(hv, "state", "State", default="")) or "N/A", + "ok" if str(c.first_present(hv, "state", "State", default="")).lower() == "up" else "bad") + d.add("Nova status", c.normalize_empty(c.first_present(hv, "status", "Status", default="")) or "N/A", + "ok" if str(c.first_present(hv, "status", "Status", default="")).lower() == "enabled" else "bad") + if result.get("compute_service_disabled_reason"): + d.add("Disabled reason", result["compute_service_disabled_reason"], "bad") + if ovs: + d.add("OVS alive", str(ovs.get("alive")), "ok" if ovs.get("alive") in (True, "True", "true") else "bad") + d.add("OVS state", str(ovs.get("state") or "N/A"), "ok" if str(ovs.get("state") or "").lower() == "up" else "bad") + + instances = result.get("instances") or [] + problem_rows: list[dict[str, Any]] = [] + ignored = 0 + + # Per-instance reconciliation for the visual: every VM on the host, and + # whether Infrahub has a counterpart for it. + roster: list[dict[str, Any]] = [] + for vm in instances: + ih_status = str(vm.get("ih_status") or "N/A") + os_status = str(vm.get("os_status") or "N/A") + linked = bool(vm.get("infrahub")) + roster.append({ + "name": str(vm.get("name") or "N/A"), + "openstack_id": str(vm.get("openstack_id") or "N/A"), + "infrahub_id": str(vm.get("infrahub_id") or "N/A"), + "ih_status": ih_status if linked else "not in Infrahub", + "os_status": os_status, + "gpus": str(vm.get("gpu_count") or "?"), + "linked": linked, + "match": linked and _tone_for_status_pair(ih_status, os_status, str(vm.get("task_state") or "None")) == "ok", + "tempest": bool(vm.get("tempest")), + "org": str(vm.get("org_value") or ""), + }) + + for vm in instances: + mismatches = vm.get("mismatches") or [] + warn_reasons = vm.get("warn_reasons") or [] + if vm.get("tempest"): + ignored += 1 + continue + if not mismatches and not warn_reasons: + continue + + ih_status = str(vm.get("ih_status") or "N/A") + os_status = str(vm.get("os_status") or "N/A") + checks = [c.mismatch_parts(m)[0] for m in mismatches] + entry = _match_mismatch_table(ih_status, os_status) + + row: dict[str, Any] = { + "idx": vm.get("idx"), + "name": vm.get("name"), + "infrahub_id": vm.get("infrahub_id"), + "openstack_id": vm.get("openstack_id"), + "ih_status": ih_status, + "os_status": os_status, + "floating_ip": vm.get("floating_ip"), + "environment": vm.get("project_environment") or "", + "mismatches": [{"check": c.mismatch_parts(m)[0], "detail": c.mismatch_parts(m)[1]} for m in mismatches], + "warn_reasons": [str(x) for x in warn_reasons], + "contacts": comms.contacts_from_result(vm), + "verdict": "", + "steps": [], + "comms_template": None, + } + + if "Infrahub Missing" in checks: + row["verdict"] = "Exists in OpenStack with no production Infrahub record" + row["steps"] = [ + (CX, "Search for the VM in Admin Portal PreProd and Staging with 'Include Deleted' checked.", ""), + (CX, "If it is DELETED or absent everywhere, delete the server in OpenStack.", "Deleting an Instance"), + ] + if vm.get("project_environment"): + row["verdict"] = f"Lives in {vm['project_environment']}, not production" + row["steps"] = [(CX, f"Confirm in Admin Portal {vm['project_environment']} whether this VM is still needed.", "")] + elif entry: + row["verdict"] = entry["verdict"] + row["steps"] = list(entry["steps"]) + row["comms_template"] = entry.get("comms") + elif "Openstack Missing" in checks: + row["verdict"] = "Infrahub holds a record OpenStack does not have" + row["steps"] = [ + (CX, "Confirm the VM is DELETED in Admin Portal Production.", ""), + (CX, "If Infrahub still shows it live, correct the record in InfraInsight.", "Infrahub Insights (Infra-Insight)"), + ] + elif ih_status.upper() == "HIBERNATED" and str(vm.get("host") or "N/A") not in ("N/A", ""): + row["verdict"] = "Infrahub kept a stale host on a HIBERNATED VM" + row["steps"] = [(CX, "Use the InfraInsight update-resource tool to remove the host from the VM record.", + "Infrahub Insights (Infra-Insight)")] + else: + row["verdict"] = "Mismatch is not in the runbook table" + row["steps"] = [ + (CX, "Ping Kheano Martinez or John Priest for a runbook update, and escalate to Infrastructure for next steps.", ""), + ] + + problem_rows.append(row) + + d.evidence["instances"] = problem_rows + d.evidence["ignored_tempest"] = ignored + d.visual = {**(d.visual or {}), "roster": roster, "slots": _gpu_slots(d.visual or {}, roster)} + + if ignored: + d.add("Ignored (tempest/OIE testing)", str(ignored), "info", + "CX-Tools suppresses tempest-project instances, which the runbook says to ignore.") + + if not problem_rows: + gap = int((d.visual or {}).get("gap") or 0) + artifact = bool((d.visual or {}).get("spare_capacity_artifact")) + d.confidence = "high" + + if gap >= 1 and not artifact: + # Every Infrahub record matches OpenStack, yet the host reports more + # GPUs in use than the instances account for. Nothing on the CX side + # explains that - it is a host-level allocation question. + d.verdict = ( + f"{gap} GPU(s) in use on {host} belong to no instance on either side" + ) + d.assessment = ( + "Every Infrahub record reconciles with OpenStack, so this is not a stale record or a rogue VM " + "that CX can correct. The host is reporting GPUs in use beyond what any instance claims, which " + "points at a leaked allocation on the hypervisor." + ) + d.act( + f"Escalate to the Infrastructure team: {host} reports {gap} GPU(s) in use that no OpenStack " + "instance claims. Ask them to check for allocations left behind by deleted domains.", + INFRA, "escalate", "Host Health Checks", + ) + d.act( + "Confirm from the hypervisor's own PCI/GPU view before escalating, in case the exporter is at fault.", + CX, "check", + ) + return + + d.verdict = f"No Infrahub/OpenStack mismatch found on {host}" + if artifact: + d.verdict = f"Not a rogue VM - {gap} spare GPU(s) on {host} reported as a discrepancy" + d.assessment = ( + "Every instance reconciles, and the 'gap' is the host's unallocated capacity: the rule's " + "In_Use_Gpus reading equals the physical GPU count, so it subtracts allocated GPUs from total " + "GPUs. Nothing to do on this host - the alert rule itself needs fixing." + ) + d.act("No CX action. Raise the rule defect with the alert owner so these stop firing.", CX, "escalate") + return + + d.assessment = ( + "Every instance on the host reconciles. Per the runbook, when there is no mismatch the issue likely " + "exists only on the Infrahub side." + ) + d.act( + "Query InfraInsight for all instances on this host and check for HIBERNATED VMs that still carry a host " + "value; remove the host with the update-resource tool.", + CX, "check", "SQL Queries - Infra Insight", + ) + d.note("The alert may already have cleared, or the discrepancy may be Infrahub-only and invisible to OpenStack.") + return + + d.verdict = f"{len(problem_rows)} of {len(instances)} instances on {host} do not reconcile" + d.confidence = "high" + d.assessment = ( + "Each mismatched instance below is mapped to its remediation from the Suspected Rogue VM table. Work them " + "individually - they can need different runbooks." + ) + + for row in problem_rows: + label = f"#{row['idx']} {row['name']} ({row['ih_status']} / {row['os_status']})" + d.add(f"Mismatch {label}", row["verdict"], "bad", + "; ".join(m["detail"] for m in row["mismatches"])) + for owner, text, guide in row["steps"]: + d.act(f"[{row['name']}] {text}", owner, "remediate", guide) + if row["comms_template"]: + draft = comms.draft(row["comms_template"], instance_name=str(row["name"] or ""), + infrahub_id=str(row.get("infrahub_id") or ""), + openstack_id=str(row.get("openstack_id") or ""), + greeting_name=comms.first_name((row["contacts"].get("owners") or [""])[0]), + agent_name=d.agent_name) + if draft: + draft.label += f" - {row['name']}" + d.drafts.append(draft) + d.contacts = row["contacts"] if row["contacts"].get("resolved") else d.contacts + + d.act("Record the findings and the alert link in the Slack thread and any associated ticket.", CX, "verify") + d.act( + "If a customer request (reboot, hibernation, restore) has been incomplete for over 30 minutes, treat it as a " + "host issue.", + CX, "check", + ) + + +# --- duplicated IPs ------------------------------------------------------- + +def _diagnose_duplicate_ip(d: Diagnosis, alert: Alert, prom: Any = None) -> None: + """Duplicated IPs.""" + fip = alert.floating_ip + if not fip: + d.error = "This alert carries no floating_ip label." + return + + d.add("Floating IP", fip) + summary = alert.annotations.get("summary", "") + claimed = re.search(r"present on `(\d+)` VMs", summary) + if claimed: + d.add("Prometheus reports", f"{claimed.group(1)} VMs holding this IP", "bad") + + # The runbook's "EASY WAY": Resources{floating_ip="..."} covers every + # environment, which is how a PreProd/Staging claimant is found. + prom_rows: list[dict[str, Any]] = [] + if prom is not None: + try: + prom_rows = prom.resources_by_floating_ip(fip) + except Exception as exc: + d.note(f"Prometheus Resources lookup failed: {exc}") + d.evidence["prometheus_claimants"] = prom_rows + for row in prom_rows: + env = row.get("environment") or row.get("env") or "" + d.add( + f"Prometheus claimant: {row.get('instance_name', '?')}", + " / ".join(x for x in [row.get("status", ""), row.get("region", ""), str(env)] if x), + "warn", + ) + + result = cxbridge.collect_vm(fip, region=alert.region) + d.evidence["vm_result"] = cxbridge.json_safe(result) + if not result.get("ok"): + d.error = str(result.get("error") or f"CX-Tools could not resolve floating IP {fip}.") + return + + claimants = result.get("instances") if result.get("mode") == "multi_vm" else [result] + claimants = [c for c in claimants if isinstance(c, dict)] + d.add("Production claimants found by CX-Tools", str(len(claimants)), + "bad" if len(claimants) > 1 else "warn") + + c = cxbridge.cx() + scenarios: list[dict[str, Any]] = [] + for vm in claimants: + ih_status = str(vm.get("ih_status") or "N/A") + server = vm.get("server") if isinstance(vm.get("server"), dict) else {} + os_fip = c.public_ip_from_server(server) if server else "" + ih_fip = str((vm.get("infrahub") or {}).get("floating_ip") or "") if isinstance(vm.get("infrahub"), dict) else "" + + entry: dict[str, Any] = { + "name": vm.get("name"), + "infrahub_id": vm.get("infrahub_id"), + "openstack_id": vm.get("openstack_id"), + "ih_status": ih_status, + "os_status": vm.get("os_status"), + "infrahub_fip": ih_fip or "none", + "openstack_fip": os_fip or "none", + "server_present": bool(server), + "contacts": comms.contacts_from_result(vm), + } + + if ih_status.upper() == "DELETING": + entry["verdict"] = "Stuck DELETING in Infrahub while the IP has moved on" + entry["steps"] = [(CX, "Delete this instance - it is stuck DELETING per Infrahub.", "Deleting an Instance")] + entry["comms"] = None + elif not server: + entry["verdict"] = "No longer exists in OpenStack, but Infrahub still holds the IP" + entry["steps"] = [ + (CX, "Remove the stale floating IP in InfraInsight and set Floating IP Status to NO FLOATING IP.", + "Infrahub Insights (Infra-Insight)"), + ] + entry["comms"] = "dupip_removed" + elif not os_fip: + entry["verdict"] = "Scenario #1 - the VM has no floating IP in OpenStack" + entry["steps"] = [ + (CX, "Remove the incorrect floating IP in InfraInsight and set Floating IP Status to NO FLOATING IP.", + "Infrahub Insights (Infra-Insight)"), + ] + entry["comms"] = "dupip_removed" + elif ih_fip and os_fip and ih_fip != os_fip: + entry["verdict"] = f"Scenario #2 - Infrahub says {ih_fip} but OpenStack says {os_fip}" + entry["steps"] = [ + (CX, f"Update the Infrahub floating IP to {os_fip} in InfraInsight.", + "Infrahub Insights (Infra-Insight)"), + ] + entry["comms"] = "dupip_corrected" + entry["new_fip"] = os_fip + else: + entry["verdict"] = "Infrahub and OpenStack agree - this is the rightful owner of the IP" + entry["steps"] = [] + entry["comms"] = None + scenarios.append(entry) + + d.evidence["claimants"] = scenarios + d.visual = { + "type": "claimants", + "ip": fip, + "items": [{ + "name": str(e["name"]), "ih_status": str(e["ih_status"]), "os_status": str(e["os_status"]), + "verdict": e["verdict"], + } for e in scenarios], + } + + for entry in scenarios: + d.add( + f"Claimant {entry['name']} ({entry['ih_status']} / {entry['os_status']})", + entry["verdict"], + "ok" if not entry["steps"] else "bad", + f"Infrahub FIP={entry['infrahub_fip']}, OpenStack FIP={entry['openstack_fip']}", + ) + for owner, text, guide in entry["steps"]: + d.act(f"[{entry['name']}] {text}", owner, "remediate", guide) + if entry.get("comms"): + draft = comms.draft(entry["comms"], instance_name=str(entry["name"] or ""), + infrahub_id=str(entry.get("infrahub_id") or ""), + openstack_id=str(entry.get("openstack_id") or ""), + greeting_name=comms.first_name((entry["contacts"].get("owners") or [""])[0]), + agent_name=d.agent_name, + floating_ip=str(entry.get("new_fip") or "")) + if draft: + draft.label += f" - {entry['name']}" + d.drafts.append(draft) + if entry["contacts"].get("resolved"): + d.contacts = entry["contacts"] + + actionable = [e for e in scenarios if e["steps"]] + if actionable: + d.verdict = f"{len(actionable)} of {len(scenarios)} claimants of {fip} need correcting" + d.confidence = "high" + elif len(claimants) <= 1: + d.verdict = f"Only one production claimant of {fip} - the duplicate is likely outside production" + d.confidence = "medium" + d.assessment = ( + "CX-Tools reconciles production Infrahub only. When production shows a single owner, the duplicate is " + "usually a PreProd or Staging record." + ) + d.act("Check PreProd and Staging for VMs holding this floating IP (repeat with those API keys).", CX, "check") + if prom_rows: + d.note("The Prometheus Resources rows above span every environment - use them to spot the other claimant.") + else: + d.verdict = f"{len(scenarios)} claimants of {fip}, none needing a correction" + d.confidence = "low" + d.note("Both sides agree for every claimant, so the alert may already be stale. Re-run in 5-10 minutes.") + + d.assessment = d.assessment or ( + "Each claimant is classified against the Duplicated IPs scenarios. Correct the losing records, then re-run " + "the query after 5-10 minutes to confirm the alert clears." + ) + d.act("Re-run the check after ~5-10 minutes to confirm only one VM holds the IP and the alert has cleared.", + CX, "verify") + + +# --- total GPUs ------------------------------------------------------------ + +def _diagnose_total_gpus(d: Diagnosis, alert: Alert) -> None: + """Problem with Total GPUs in a System.""" + host = alert.host or alert.instance_name + region = alert.region + if not host: + d.error = "This alert carries no hypervisor label." + return + + d.verdict = f"Host {host} is reporting missing GPU(s)" + d.confidence = "high" + d.assessment = ( + "Monitoring detected a host with fewer GPUs than expected. This hits revenue and customer experience, so the " + "job is to find out who is on the host and escalate the hardware fault to Infrastructure." + ) + d.add("Host", host) + d.add("GPU model", alert.gpu_name or "N/A") + summary = alert.annotations.get("summary", "") + count = re.search(r"\*\s*" + re.escape(alert.gpu_name or "") + r":\*\s*`(\d+)`", summary) if alert.gpu_name else None + if count: + d.add("GPUs reported present", count.group(1), "bad", + "Compare against the expected count for this chassis (typically 8).") + + if region: + census = cxbridge.host_gpu_census(region, host) + d.evidence["gpu_census"] = census + if census.get("ok"): + customers = [i for i in census.get("instances", [])] + d.add("Instances on host", str(len(customers)), "warn" if customers else "ok") + d.add("GPUs allocated to instances", str(census.get("total_gpus"))) + for inst in customers: + d.add(f"Instance {inst['name']}", f"{inst['status']} - {inst['flavor']} ({inst['gpus']} GPU)", "warn") + if customers: + d.act( + "Customers are on this host - they may need contacting for host maintenance, depending on the " + "Infrastructure team's assessment.", + CX, "comms", + ) + d.act("Run List Instances on a Hypervisor (Windmill) to confirm the customer list.", CX, "check") + else: + d.act("No instances on the host - no customer impact to coordinate.", CX, "verify", status="done") + else: + d.add("Instance census", f"failed: {census.get('error', '')}", "warn") + d.act("Run List Instances on a Hypervisor (Windmill) to determine if any customers are on the host.", + CX, "check") + else: + d.act("Run List Instances on a Hypervisor (Windmill) to determine if any customers are on the host.", + CX, "check") + + d.act( + f"Search Jira for an existing OPEN issue for {host}. If one exists, update it with the alert details and ask " + "whether a new ticket is needed.", + CX, "check", + ) + d.act(f"Otherwise raise a Jira for the Infrastructure team with '{host}' in the title, describing the issue and " + "copying the alert details.", INFRA, "escalate") + d.note("There is no approved customer template for this alert - host-maintenance comms are coordinated separately.") + + +# --- single-VM status mismatch --------------------------------------------- + +def _diagnose_status_mismatch(d: Diagnosis, alert: Alert, vm: dict[str, Any]) -> None: + """The per-region "Openstack status=X and Infrahub status!=X" rules. + + Same class of problem as Suspected Rogue VM, scoped to one VM, so it takes + the same Mismatch Remediation table. + """ + ih_status = str(vm.get("ih_status") or "N/A") + os_status = str(vm.get("os_status") or "N/A") + entry = _match_mismatch_table(ih_status, os_status) + + if entry: + d.verdict = entry["verdict"] + d.confidence = "high" + for owner, text, guide in entry["steps"]: + d.act(text, owner, "remediate", guide) + if entry.get("comms"): + draft = _vm_draft(d, entry["comms"], vm, alert) + if draft: + d.drafts.append(draft) + elif not vm.get("mismatches"): + d.verdict = f"Infrahub and OpenStack now agree ({ih_status} / {os_status})" + d.confidence = "high" + d.assessment = "The mismatch has already cleared; the alert should stop on the next evaluation." + else: + d.verdict = f"Mismatch ({ih_status} / {os_status}) is not in the runbook table" + d.confidence = "low" + d.act("Ping Kheano Martinez or John Priest for a runbook update, and escalate to Infrastructure for next steps.", + CX, "escalate") + + d.assessment = d.assessment or ( + "A single-VM state mismatch. The Suspected Rogue VM remediation table covers which side to correct." + ) + + +# --- dispatch -------------------------------------------------------------- + +_VM_KINDS = { + "error": _diagnose_error, + "deleting": _diagnose_deleting, + "shutoff": _diagnose_shutoff, + "hibernating": _diagnose_hibernating, + "creating": _diagnose_creating, + "restoring": _diagnose_restoring, + "rebooting": _diagnose_rebooting, + "build": _diagnose_build, + "status_mismatch": _diagnose_status_mismatch, +} + + +def _screened_out(d: Diagnosis, alert: Alert) -> None: + """Report a screened-out alert without spending calls diagnosing it.""" + screen = alert.screen or {} + d.verdict = f"No action needed - {screen.get('label', 'screened out')}" + d.confidence = "high" + d.assessment = str(screen.get("reason") or "") + d.add("Screening verdict", screen.get("label", ""), "ok") + if screen.get("current_state"): + d.add("Current state", screen["current_state"], "ok") + if screen.get("detail"): + d.note(str(screen["detail"])) + d.act( + "Nothing to do. If the alert is still up in Prometheus it should clear on the next evaluation; " + "if it does not, the rule may need a silence or a fix.", + CX, "verify", status="done", + ) + d.note("Re-run with 'Diagnose anyway' to query Infrahub and OpenStack directly for this alert.") + + +def _add_screening_findings(d: Diagnosis, alert: Alert) -> None: + screen = alert.screen or {} + if not screen: + return + tone = {"real": "bad", "unverified": "warn", "chronic": "warn", + "low_impact": "warn", "pending": "warn", "resolved": "ok"}.get(str(screen.get("verdict")), "info") + d.add("Screening", f"{screen.get('label', '')} - {screen.get('reason', '')}", tone, + str(screen.get("detail") or "")) + + +def artifact_now(d: Diagnosis) -> bool: + return bool((d.visual or {}).get("spare_capacity_artifact")) + + +def _add_rogue_gpu_gap(d: Diagnosis, alert: Alert, snap: Any) -> None: + """Show the GPU accounting gap the Rogue VM rule actually fires on. + + The rule is `sum by(instance)(In_Use_Gpus) - sum by(instance)(Resources{ + status=~"ACTIVE|SHUTOFF|PRE_ACTIVE"}) >= 1` - a per-host GPU accounting gap, + not a status comparison. Two different faults produce that gap and Prometheus + cannot tell them apart, so both are stated and the host reconciliation below + is what settles it. + """ + host = alert.host or alert.instance_name + if snap is None or not host: + return + delta = getattr(snap, "rogue_delta", {}).get(host) + if delta is None: + return + + total = getattr(snap, "total_gpus", {}).get(host) + in_use = getattr(snap, "in_use_gpus", {}).get(host) + rows = getattr(snap, "resources_by_host", {}).get(host, []) + counted = sum(int(r.get("_gpus", "0") or 0) for r in rows + if r.get("status", "").upper() in ("ACTIVE", "SHUTOFF", "PRE_ACTIVE")) + + artifact = in_use is not None and total is not None and in_use == total + d.visual = { + "type": "gpu", + "host": host, + "physical": int(total) if total is not None else None, + "in_use_metric": int(in_use) if in_use is not None else None, + "accounted": counted, + "gap": int(delta), + "instances": len(rows), + # When the rule's "in use" reading is just the physical count, the gap + # it reports is spare capacity, not a missing VM. + "spare_capacity_artifact": artifact, + } + + if total is not None: + d.add("GPUs on host (physical)", str(int(total))) + if in_use is not None: + d.add("GPUs allocated on host", str(int(in_use))) + d.add("GPUs Infrahub accounts for", str(counted)) + d.add("Accounting gap", f"{int(delta)} GPU(s)", "bad" if delta >= 1 else "ok", + "This is the quantity the alert fired on.") + + if rows: + by_status: dict[str, int] = {} + for row in rows: + by_status[row.get("status", "?")] = by_status.get(row.get("status", "?"), 0) + 1 + d.add("Infrahub VMs on host", ", ".join(f"{v} {k}" for k, v in sorted(by_status.items()))) + else: + d.add("Infrahub VMs on host", "none recorded", "bad") + + if delta and delta >= 1 and not artifact_now(d): + d.add( + "Why there is no ID to chase", "nothing in OpenStack claims these GPUs", "warn", + "`server list --host` is the complete set of instances Nova knows on this host, and their flavours " + "account for fewer GPUs than the host reports in use. There is no instance UUID to look up because no " + "instance owns them - which is exactly why this is a host-level escalation. Identifying them means " + "looking at the host itself.", + ) + d.evidence["host_probe_commands"] = [ + f"{alert.region} server list --all-projects --host {host} -c ID -c Name -c Status -c Flavor", + f"{alert.region} hypervisor show {host} -f json", + f"ssh {host} nvidia-smi --query-gpu=index,uuid,pci.bus_id --format=csv", + f"ssh {host} 'virsh list --all'", + ] + + if delta and delta >= 1: + unattributed = getattr(snap, "unattributed_active", 0) + unattributed_gpus = getattr(snap, "unattributed_active_gpus", 0) + d.add( + "Two possible causes", "instances OpenStack has that Infrahub does not, or Infrahub VMs with no host set", + "warn", + "Either there are instances running on this host with no Infrahub record (a true rogue VM), or Infrahub " + "has ACTIVE VMs whose host field is unset, so they are not counted against this host. The per-instance " + "reconciliation below distinguishes them: a genuine rogue VM shows up as 'Infrahub Missing'.", + ) + if unattributed: + d.note( + f"Platform-wide, Infrahub currently has {unattributed} ACTIVE/SHUTOFF VM(s) ({unattributed_gpus} GPUs) " + "with no host recorded. That is enough to explain gaps like this one without any rogue VM existing, " + "so confirm against the per-instance list before escalating." + ) + + +def diagnose(alert: Alert, prom: Any = None, snap: Any = None, force: bool = False, + user_settings: Any = None) -> Diagnosis: + """Gather evidence for one alert and reach the runbook's verdict.""" + d = Diagnosis(alert=alert) + d.agent_name = getattr(user_settings, "agent_name", "") or "" + + if not force and alert.screen and not alert.screen.get("actionable", True): + _screened_out(d, alert) + return d + + _common_preamble(d, alert) + _add_screening_findings(d, alert) + + try: + if alert.kind in ("rogue_vm", "orphan_vm"): + _add_rogue_gpu_gap(d, alert, snap) + _diagnose_rogue_vm(d, alert) + elif alert.kind == "duplicate_ip": + _diagnose_duplicate_ip(d, alert, prom) + elif alert.kind == "total_gpus": + _diagnose_total_gpus(d, alert) + elif alert.kind in _VM_KINDS: + target, org_id = _vm_target(alert) + if not target: + d.error = "This alert has neither an OpenStack ID nor an instance name to look up." + return d + vm = cxbridge.collect_vm(target, region=alert.region, org_id=org_id) + d.evidence["vm_result"] = cxbridge.json_safe(vm) + if not vm.get("ok"): + d.error = str(vm.get("error") or f"CX-Tools returned no usable telemetry for {target}.") + if alert.kind == "creating": + # A CREATING VM that never reached OpenStack legitimately has + # nothing to collect; the runbook conclusion still holds. + d.error = "" + d.add("CX-Tools lookup", "no telemetry available", "bad", + str(vm.get("error") or "")) + _diagnose_creating(d, alert, vm) + _attach_contacts(d, vm) + return d + _add_vm_findings(d, vm) + _VM_KINDS[alert.kind](d, alert, vm) + _attach_contacts(d, vm) + else: + d.error = f"Alert kind '{alert.kind}' is not covered by the CX runbooks." + except cxbridge.BridgeError as exc: + d.error = str(exc) + except Exception as exc: # surface, don't crash the request + d.error = f"Diagnosis failed: {type(exc).__name__}: {exc}" + + return d diff --git a/triagelib/screening.py b/triagelib/screening.py new file mode 100644 index 0000000..26b4eb8 --- /dev/null +++ b/triagelib/screening.py @@ -0,0 +1,313 @@ +"""Noise-vs-real screening. + +An alert firing is not the same as work existing. Prometheus keeps an alert up +until its expression stops matching on the next evaluation, and the CX rules sit +on top of a metric pipeline that can go stale or empty. So before anything gets +diagnosed, each alert's condition is re-checked against current state. + +The re-check is deliberately cheap: it reads the same Prometheus series the rules +are built from (one bulk snapshot for the whole queue) rather than making an +Infrahub or OpenStack call per alert. Anything it cannot settle is treated as +real - screening only ever demotes an alert on positive evidence. +""" +from __future__ import annotations + +import time +from typing import Any, Optional + +from .alerts import Alert + +# Beyond this, an alert is chronic: it is either already ticketed or nobody has +# silenced it. Either way it is not today's queue. +CHRONIC_DAYS = 3 + +# Verdicts, most to least urgent. +REAL = "real" +OVERDUE = "overdue" +UNVERIFIED = "unverified" +CHRONIC = "chronic" +LOW_IMPACT = "low_impact" +PENDING = "pending" +RESOLVED = "resolved" +RULE_DEFECT = "rule_defect" +SUPPRESSED = "suppressed" + +VERDICT_LABELS = { + REAL: "needs action", + OVERDUE: "overdue", + UNVERIFIED: "needs action (unverified)", + CHRONIC: "chronic", + LOW_IMPACT: "low impact", + PENDING: "not yet firing", + RESOLVED: "already resolved", + RULE_DEFECT: "invalid - alert rule defect", + SUPPRESSED: "hidden by a rule", +} + +# Runbook commitments: past this age the customer contact is late, not chronic. +# From "Instance in ERROR state": if a stock-failure instance is not deleted +# within a day, contact the customer. SHUTOFF is here for a different reason - +# the VM accrues full cost the entire time it is stopped, so an old one is a +# customer who has been paying for nothing for longer, not a stale alert. +SLA_HOURS = {"error": 24, "creating": 24, "restoring": 24, "rebooting": 24, + "build": 24, "shutoff": 48} + +SLA_REASON = { + "shutoff": "a SHUTOFF VM accrues full cost the whole time, so this customer has been paying " + "for a stopped instance that long and may never have been told", +} + +# Verdicts that stay in the working queue by default. +ACTIONABLE = {REAL, OVERDUE, UNVERIFIED} + +# Kept for the SLA check: an internal owner should not make an alert *overdue*. +def _is_internal(alert) -> bool: + return bool(getattr(alert, 'is_internal_org', False)) + + +def _result(verdict: str, reason: str, *, current: str = "", detail: str = "") -> dict[str, Any]: + return { + "verdict": verdict, + "label": VERDICT_LABELS[verdict], + "reason": reason, + "actionable": verdict in ACTIONABLE, + "current_state": current, + "detail": detail, + } + + +def _resources_row(alert: Alert, snap: Any) -> Optional[dict[str, str]]: + """Find the VM's current Infrahub row in the snapshot.""" + if alert.openstack_id and alert.openstack_id in snap.by_openstack_id: + return snap.by_openstack_id[alert.openstack_id] + if alert.instance_name and alert.instance_name in snap.by_instance_name: + return snap.by_instance_name[alert.instance_name] + return None + + +def _screen_state_alert(alert: Alert, snap: Any) -> dict[str, Any]: + """State alerts: does Infrahub still report the state that fired?""" + expected = alert.status.upper() + row = _resources_row(alert, snap) + + if row is None: + if alert.kind == "creating" and not alert.openstack_id: + # A CREATING VM that never reached OpenStack has no Resources row to + # find; the alert stands on its own. + return _result(REAL, "Instance never got an OpenStack ID, so it cannot have recovered.") + return _result( + RESOLVED, + "No longer present in Infrahub's Resources series - the record has been deleted or cleaned up.", + ) + + current = row.get("status", "").upper() + if not expected: + return _result(UNVERIFIED, "Alert carried no status label, so its condition could not be re-checked.", + current=current) + if current == expected: + return _result(REAL, f"Infrahub still reports {current}.", current=current) + return _result( + RESOLVED, + f"Fired on {expected} but Infrahub now reports {current} - it resolved on its own.", + current=current, + ) + + +def _screen_duplicate_ip(alert: Alert, snap: Any) -> dict[str, Any]: + count = snap.fip_counts.get(alert.floating_ip) + if count is None: + return _result(RESOLVED, f"No Infrahub VM currently holds {alert.floating_ip}.") + if count > 1: + return _result(REAL, f"{count} VMs still hold {alert.floating_ip}.", current=f"{count} claimants") + return _result( + RESOLVED, + f"Only 1 VM holds {alert.floating_ip} now - the duplicate is gone.", + current="1 claimant", + ) + + +def _screen_rogue_vm(alert: Alert, snap: Any) -> dict[str, Any]: + """Rogue VM fires on a per-host GPU accounting gap; re-evaluate the gap. + + The rule subtracts Infrahub's allocated GPUs from `In_Use_Gpus`. On almost + every host `In_Use_Gpus` equals `Total_Gpus` - the physical GPU count - so + the expression reduces to "this host has at least one unallocated GPU" and + fires on ordinary spare capacity. Validated against OpenStack on 10 firing + hosts: Infrahub and OpenStack agreed exactly on all of them. + """ + host = alert.host or alert.instance_name + delta = snap.rogue_delta.get(host) + if delta is None: + return _result(UNVERIFIED, f"No current GPU accounting data for {host}.") + if delta >= 1: + known = len(snap.resources_by_host.get(host, [])) + in_use = snap.in_use_gpus.get(host) + total = snap.total_gpus.get(host) + if in_use is not None and total is not None and in_use == total: + return _result( + RULE_DEFECT, + f"Not a rogue VM: on {host} the rule's 'GPUs in use' reading ({int(in_use)}) is just the " + f"physical GPU count, so it is reporting {int(delta)} free GPU(s) as a discrepancy.", + current=f"{int(delta)} GPU(s) spare capacity", + detail=( + "In_Use_Gpus == Total_Gpus on this host, so the rule expression reduces to " + "'physical GPUs minus allocated GPUs', which is spare capacity rather than an " + "Infrahub/OpenStack mismatch. The rule needs fixing at source." + ), + ) + if total is None: + detail = (f"Infrahub records {known} VM(s) on this host. No Total_Gpus reading is available, so the " + "spare-capacity explanation cannot be confirmed or ruled out from metrics alone.") + else: + detail = (f"Infrahub records {known} VM(s) on this host. In_Use_Gpus ({int(in_use)}) differs from " + f"Total_Gpus ({int(total)}), so this is not simply spare capacity.") + return _result( + REAL, + f"{int(delta)} GPU(s) allocated on {host} are still unaccounted for in Infrahub.", + current=f"gap {int(delta)}", + detail=detail, + ) + return _result( + RESOLVED, + f"GPU accounting for {host} now balances (delta {int(delta)}).", + current=f"delta {int(delta)}", + ) + + +def _screen_total_gpus(alert: Alert, snap: Any) -> dict[str, Any]: + host = alert.host or alert.instance_name + total = snap.total_gpus.get(host) + if total is None: + return _result(UNVERIFIED, f"No current Total_Gpus reading for {host}.") + # The rule fires on any count in 1,2,3,5,6,7,9 - i.e. not a full complement. + if int(total) in (0, 4, 8): + return _result( + RESOLVED, + f"{host} now reports {int(total)} GPUs, a valid complement.", + current=f"{int(total)} GPUs", + ) + in_use = snap.in_use_gpus.get(host) + detail = f"{int(in_use)} GPU(s) currently allocated to instances." if in_use is not None else "" + return _result( + REAL, + f"{host} still reports {int(total)} GPUs - hardware is missing.", + current=f"{int(total)} GPUs", + detail=detail, + ) + + +_KIND_SCREENS = { + "duplicate_ip": _screen_duplicate_ip, + "rogue_vm": _screen_rogue_vm, + "orphan_vm": _screen_rogue_vm, + "total_gpus": _screen_total_gpus, +} + + +def screen(alert: Alert, snap: Any, user_settings: Any = None) -> dict[str, Any]: + """Decide whether an alert is worth a human's attention right now.""" + # A rule the team wrote wins over anything inferred here. + if user_settings is not None: + from . import settings as settings_mod + + rule = settings_mod.first_match(user_settings, alert) + if rule: + reason = rule.get("reason") or "Matched a suppression rule." + return _result( + SUPPRESSED, + f"Hidden by \u201c{rule.get('name')}\u201d - {reason}", + detail="Edit or disable this in Settings.", + ) + + # Prometheus has not committed to this alert yet. + if alert.state == "pending": + remaining = "" + if alert.for_seconds and alert.age_minutes is not None: + remaining = f" It needs {alert.for_seconds // 60} min of continuous firing; it has {alert.age_minutes} min." + return _result(PENDING, f"Prometheus still has this pending, not firing.{remaining}") + + if snap is None or not getattr(snap, "loaded", False): + return _result(UNVERIFIED, "Current-state snapshot unavailable, so the condition could not be re-checked.") + + screener = _KIND_SCREENS.get(alert.kind) + result = screener(alert, snap) if screener else _screen_state_alert(alert, snap) + + # A still-valid alert can still be the wrong thing to spend time on. + if result["actionable"]: + # Uses the recovered duration: activeAt is reset by pipeline dips, which + # would make every chronic alert look hours old. + age = alert.effective_age_minutes + sla = SLA_HOURS.get(alert.kind) + if sla and age is not None and age > sla * 60 and not alert.is_internal_org: + # The runbook commits to contacting the customer inside this window, + # so age makes it more urgent, not less. Never demote these to chronic. + why = SLA_REASON.get( + alert.kind, + f"past the {sla}h point where the runbook says to contact the customer", + ) + return _result( + OVERDUE, + f"Condition has held for {alert.effective_age_text} - {why}. Overdue, not chronic.", + current=result["current_state"], + detail=result["reason"], + ) + if age is not None and age > CHRONIC_DAYS * 24 * 60: + note = result["reason"] + if alert.age_is_reset: + note += (f" Prometheus reports only {alert.age_text} because a metric-pipeline dip reset " + "activeAt; the condition itself has held far longer.") + return _result( + CHRONIC, + f"Condition still holds, but it has held for {alert.effective_age_text} - " + "chronic, so it is likely already ticketed rather than new work.", + current=result["current_state"], + detail=note, + ) + return result + + +def screen_all(items: list[Alert], snap: Any, user_settings: Any = None) -> None: + for alert in items: + alert.screen = screen(alert, snap, user_settings) + + +def summarize(items: list[Alert]) -> dict[str, Any]: + counts: dict[str, int] = {} + for alert in items: + verdict = alert.screen.get("verdict", UNVERIFIED) + counts[verdict] = counts.get(verdict, 0) + 1 + return { + "counts": counts, + "actionable": sum(1 for a in items if a.screen.get("actionable")), + "screened_out": sum(1 for a in items if not a.screen.get("actionable")), + "labels": VERDICT_LABELS, + } + + +def health_warnings(snap: Any) -> list[str]: + """Rules whose input metrics are empty are broken, not quiet.""" + warnings: list[str] = [] + for metric in getattr(snap, "broken_inputs", []) or []: + if metric == "openstack_nova_server_status": + warnings.append( + "The OpenStack server metric (openstack_nova_server_status) is currently empty. Any rule built on " + "it is unreliable: 'Exists in Infrahub but does not exist in OpenStack' matches every VM (which is " + "why it is excluded here), and 'Suspected Orphan VM' cannot fire at all. Worth raising with whoever " + "owns the exporter." + ) + else: + warnings.append( + f"The metric '{metric}' is currently empty, so alert rules that depend on it are unreliable." + ) + + dips = getattr(snap, "pipeline_dips", []) or [] + if dips: + latest = max(dips, key=lambda d: d["end"]) + mins_ago = max(0, int((time.time() - latest["end"]) // 60)) + warnings.append( + f"The Infrahub 'Resources' metric dropped most of its series {len(dips)} time(s) in the last 24h " + f"(most recently {mins_ago} min ago: {latest['low']} of ~{latest['normal']} series for " + f"{latest['minutes']} min). Every alert live during a dip resolves and re-fires, so Prometheus' own " + "alert ages all reset together. Ages shown here are recovered from ALERTS history instead." + ) + return warnings diff --git a/triagelib/server.py b/triagelib/server.py new file mode 100644 index 0000000..3f99a40 --- /dev/null +++ b/triagelib/server.py @@ -0,0 +1,437 @@ +"""Localhost HTTP server: alert queue, background triage jobs, JSON API.""" +from __future__ import annotations + +import json +import subprocess +import threading +import time +import traceback +import urllib.parse +import uuid +from concurrent.futures import ThreadPoolExecutor +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from typing import Any, Optional + +from . import (VERSION, alerts as alertlib, comms, cxbridge, integrations, linkage, + runbooks, screening, settings as settings_mod, ui, ui_linkage, + ui_settings, ui_v2) +from .prometheus import ( + AlertCache, PrometheusClient, PrometheusError, RuleIndex, StateSnapshot, TrueAgeIndex, + parse_alert_text, +) + +TRIAGE_WORKERS = 3 +JOB_TTL_SECONDS = 30 * 60 + + +class Jobs: + """In-memory triage jobs. Diagnosis takes tens of seconds, so the UI polls.""" + + def __init__(self, workers: int = TRIAGE_WORKERS): + self._pool = ThreadPoolExecutor(max_workers=workers, thread_name_prefix="triage") + self._lock = threading.Lock() + self._jobs: dict[str, dict[str, Any]] = {} + + def submit(self, alert: alertlib.Alert, prom: PrometheusClient, + snap: Any = None, force: bool = False, user_settings: Any = None) -> str: + job_id = uuid.uuid4().hex[:12] + with self._lock: + self._reap() + self._jobs[job_id] = { + "id": job_id, + "state": "running", + "created": time.time(), + "alert": alert.to_json(), + "result": None, + "error": "", + } + self._pool.submit(self._run, job_id, alert, prom, snap, force, user_settings) + return job_id + + def _run(self, job_id: str, alert: alertlib.Alert, prom: PrometheusClient, + snap: Any = None, force: bool = False, user_settings: Any = None) -> None: + started = time.monotonic() + try: + diagnosis = runbooks.diagnose(alert, prom, snap, force, user_settings) + payload = diagnosis.to_json() + payload["elapsed_seconds"] = round(time.monotonic() - started, 1) + with self._lock: + job = self._jobs.get(job_id) + if job is not None: + job.update({"state": "done", "result": payload}) + except Exception: + with self._lock: + job = self._jobs.get(job_id) + if job is not None: + job.update({"state": "error", "error": traceback.format_exc(limit=4)}) + + def get(self, job_id: str) -> Optional[dict[str, Any]]: + with self._lock: + job = self._jobs.get(job_id) + return dict(job) if job else None + + def _reap(self) -> None: + cutoff = time.time() - JOB_TTL_SECONDS + for key in [k for k, v in self._jobs.items() if v["created"] < cutoff]: + self._jobs.pop(key, None) + + +class App: + def __init__(self, prom: PrometheusClient): + self.prom = prom + self.cache = AlertCache(prom) + self.rules = RuleIndex(prom) + self.snapshot = StateSnapshot(prom) + self.true_age = TrueAgeIndex(prom) + self.jobs = Jobs() + self.scan = linkage.Scan() + self.settings = settings_mod.Settings() + + def warm(self, log=print) -> None: + """Populate the caches before serving. + + Recovering true alert ages reads a week of ALERTS history, so doing it + lazily would make the first page load take ~20 seconds. + """ + try: + self.rules.ensure() + log(f" rule index: {self.rules.count} alerting rule(s)") + snap = self.snapshot.get() + log(f" state snapshot: {len(snap.by_openstack_id)} VMs, {len(snap.total_gpus)} hosts") + if snap.pipeline_dips: + log(f" {len(snap.pipeline_dips)} Resources metric dip(s) in the last 24h " + "- alert ages will be recovered from history") + ages = self.true_age.get() + log(f" alert history: {ages.count} alert(s) indexed over {ages.WINDOW_DAYS} days") + except PrometheusError as exc: + log(f" WARN: could not warm Prometheus caches: {exc}") + + # --- endpoints --------------------------------------------------------- + + def health(self) -> dict[str, Any]: + checks: list[dict[str, Any]] = [] + + def add(name: str, ok: bool, detail: str) -> None: + checks.append({"name": name, "ok": ok, "detail": detail}) + + try: + path = cxbridge.cx_tools_path() + add("CX-Tools", True, path) + except cxbridge.BridgeError as exc: + add("CX-Tools", False, str(exc)) + + try: + cfg = cxbridge.config() + add("API credentials", True, "Infrahub and InfraInsight keys loaded from 1Password") + add("Infrahub endpoint", True, cfg.infrahub_base) + except cxbridge.BridgeError as exc: + add("API credentials", False, str(exc)) + + try: + rc, out, _err = _run(["docker", "ps", "--format", "{{.Names}}"], 10) + running = {x.strip() for x in out.splitlines() if x.strip()} if rc == 0 else set() + expected = {"ca1-osc", "ca2-osc", "us1-osc", "no1-osc"} + missing = sorted(expected - running) + add("OpenStack CLI containers", not missing, + "all present" if not missing else f"not running: {', '.join(missing)}") + except Exception as exc: + add("OpenStack CLI containers", False, str(exc)) + + try: + add("Prometheus", True, f"{self.prom.base} ({self.prom.describe_transport()})") + except PrometheusError as exc: + add("Prometheus", False, str(exc)) + + return {"version": VERSION, "ok": all(c["ok"] for c in checks), "checks": checks} + + def alert_queue(self, force: bool = False) -> dict[str, Any]: + raw, error, age = self.cache.get(force=force) + snap = self.snapshot.get() + ages = self.true_age.get() + parsed = [alertlib.from_prometheus(a, self.rules, ages) for a in raw] + + excluded = [a for a in parsed if alertlib.is_excluded(a)] + candidates = [a for a in parsed if not alertlib.is_excluded(a)] + + cx = [a for a in candidates if a.category == "cx" and alertlib.cx_relevant(a)] + screening.screen_all(cx, snap, self.settings) + + # Everything that is not a CX runbook alert: node-exporter in its own + # section, then the rest of the platform rules. Split by identity, since + # two distinct alerts can compare equal field-for-field. + in_cx = {id(a) for a in cx} + infra = [a for a in candidates if id(a) not in in_cx] + infra.sort(key=alertlib.sort_key) + + return { + "error": error or snap.error or ages.error, + "cache_age_seconds": round(age, 1), + "warnings": screening.health_warnings(snap), + "totals": { + "prometheus": len(parsed), + "cx": len(cx), + "infrastructure": len(infra), + "excluded": len(excluded), + }, + "excluded_note": ( + f"{len(excluded)} '{', '.join(sorted({a.alertname for a in excluded}))}' alerts hidden" + if excluded else "" + ), + "integrations": {"zendesk": integrations.zendesk_configured(), + "jira": integrations.jira_configured()}, + "summary": screening.summarize(cx), + "groups": alertlib.group_alerts(cx), + "infrastructure": _infra_sections(infra), + } + + def triage(self, labels: dict[str, str], annotations: Optional[dict[str, str]] = None, + state: str = "firing", active_at: Any = None, force: bool = False) -> dict[str, Any]: + alert = alertlib.from_labels(labels, annotations, state=state, active_at=active_at) + if alert.kind == "excluded": + return {"error": f"'{alertlib.clean_alertname(alert.alertname)}' is excluded as a monitoring fault."} + if alert.kind == "other": + return {"error": f"'{alertlib.clean_alertname(alert.alertname)}' is not an alert type the CX runbooks cover."} + meta = self.rules.get(alert.alertname) + if meta: + alert.rule_file = str(meta.get("file") or "") + alert.rule_group = str(meta.get("group") or "") + alert.for_seconds = int(meta.get("for_seconds") or 0) + snap = self.snapshot.get() + alert.true_age_minutes, alert.true_age_capped = self.true_age.get().lookup(alert.labels) + alert.screen = screening.screen(alert, snap, self.settings) + job_id = self.jobs.submit(alert, self.prom, snap, force, self.settings) + return {"job_id": job_id, "alert": alert.to_json()} + + def triage_by_id(self, alert_id: str, force: bool = False) -> dict[str, Any]: + raw, error, _age = self.cache.get() + if error and not raw: + return {"error": error} + for item in raw: + alert = alertlib.from_prometheus(item, self.rules, self.true_age.get()) + if alert.fingerprint() == alert_id: + return self.triage(alert.labels, alert.annotations, alert.state, + item.get("activeAt"), force=force) + return {"error": "That alert is no longer firing. Refresh the queue."} + + def parse(self, text: str) -> dict[str, Any]: + found = parse_alert_text(text) + if not found: + return {"error": "Could not find any label set in that text. Paste an ALERTS{...} line or a Prometheus graph URL."} + out = [] + for labels in found: + alert = alertlib.from_labels(labels) + out.append({"alert": alert.to_json(), "supported": alertlib.cx_relevant(alert)}) + return {"parsed": out} + + def settings_payload(self) -> dict[str, Any]: + return self.settings.to_json() + + def _current_alerts(self) -> list[Any]: + raw, _error, _age = self.cache.get() + return [a for a in (alertlib.from_prometheus(x, self.rules) for x in raw) + if not alertlib.is_excluded(a) and alertlib.cx_relevant(a)] + + def settings_action(self, body: dict[str, Any]) -> dict[str, Any]: + action = str(body.get("action") or "") + if action == "save_rule": + rule = self.settings.upsert_rule(body.get("rule") or {}) + return {"ok": True, "rule": rule, "settings": self.settings.to_json()} + if action == "delete_rule": + self.settings.delete_rule(str(body.get("id") or "")) + return {"ok": True, "settings": self.settings.to_json()} + if action == "toggle_rule": + self.settings.toggle_rule(str(body.get("id") or ""), bool(body.get("enabled"))) + return {"ok": True, "settings": self.settings.to_json()} + if action == "general": + self.settings.set_general(body.get("agent_name"), body.get("chronic_days")) + return {"ok": True, "settings": self.settings.to_json()} + if action == "preview": + hits = settings_mod.preview(self.settings, body.get("rule") or {}, self._current_alerts()) + return {"ok": True, "matches": hits, "count": len(hits)} + return {"ok": False, "error": f"Unknown action: {action}"} + + def start_scan(self) -> dict[str, Any]: + if self.scan.state == "running": + return {"started": False, "reason": "already running"} + snap = self.snapshot.get() + threading.Thread(target=self.scan.run, args=(snap,), daemon=True).start() + return {"started": True} + + def send_zendesk(self, body: dict[str, Any]) -> dict[str, Any]: + """Deliberately refuses until Zendesk is configured AND enabled. + + Contacting a customer is the one thing this tool must never do as a + side effect, so delivery stays behind explicit configuration rather + than being reachable from the UI by default. + """ + if not integrations.zendesk_configured(): + return {"ok": False, "error": "Zendesk is not configured. Set CX_ZENDESK_SUBDOMAIN, " + "CX_ZENDESK_EMAIL and CX_ZENDESK_TOKEN, then restart."} + return {"ok": False, "error": "Sending is not enabled in this build. The payload is ready; " + "wiring delivery is a deliberate, separate step."} + + def templates(self) -> dict[str, Any]: + return {"templates": [d.to_json() for d in (comms.draft(k) for k in comms._TEMPLATES) if d]} + + +SOURCE_LABELS = { + "node-exporter-rules.yml": "Node exporter (hosts)", + "ceph-rules.yml": "Ceph", + "mysql-rules.yml": "MySQL", + "mysql-performance-rules.yml": "MySQL performance", + "galera-rules.yml": "Galera", + "openstack-rules.yml": "OpenStack services", + "blackbox.yml": "Blackbox / OOB", + "infrahub-rules.yml": "Infrahub (no CX runbook)", +} + + +def _infra_sections(items: list[alertlib.Alert]) -> list[dict[str, Any]]: + """Group infrastructure alerts by their rule file, node-exporter first.""" + buckets: dict[str, list[alertlib.Alert]] = {} + for alert in items: + buckets.setdefault(alert.rule_file or "unknown", []).append(alert) + + sections = [] + for source, members in buckets.items(): + by_name: dict[str, int] = {} + for alert in members: + name = alertlib.clean_alertname(alert.alertname) + by_name[name] = by_name.get(name, 0) + 1 + sections.append({ + "source": source, + "label": SOURCE_LABELS.get(source, source), + "total": len(members), + "by_alertname": sorted(({"name": k, "count": v} for k, v in by_name.items()), + key=lambda x: (-x["count"], x["name"])), + "alerts": [a.to_json() for a in members], + }) + sections.sort(key=lambda s: (s["source"] != alertlib.NODE_RULE_FILE, -s["total"])) + return sections + + +def _run(cmd: list[str], timeout: int) -> tuple[int, str, str]: + try: + p = subprocess.run(cmd, text=True, capture_output=True, timeout=timeout) + return p.returncode, p.stdout, p.stderr + except Exception as exc: + return 1, "", str(exc) + + +class Handler(BaseHTTPRequestHandler): + server_version = f"cx-triage/{VERSION}" + app: App + + def log_message(self, fmt: str, *args: Any) -> None: + if self.path.startswith("/api/jobs/"): + return + print(f" {self.command} {self.path}") + + # --- helpers ---------------------------------------------------------- + + def _send_json(self, payload: Any, code: int = 200) -> None: + body = json.dumps(payload, default=str).encode() + self.send_response(code) + self.send_header("Content-Type", "application/json; charset=utf-8") + self.send_header("Content-Length", str(len(body))) + self.send_header("Cache-Control", "no-store") + self.end_headers() + self.wfile.write(body) + + def _send_html(self, html: str) -> None: + body = html.encode() + self.send_response(200) + self.send_header("Content-Type", "text/html; charset=utf-8") + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def _body_json(self) -> dict[str, Any]: + length = int(self.headers.get("Content-Length") or 0) + if not length: + return {} + try: + return json.loads(self.rfile.read(length).decode("utf-8", errors="replace")) or {} + except json.JSONDecodeError: + return {} + + # --- routes ----------------------------------------------------------- + + def do_GET(self) -> None: # noqa: N802 + parsed = urllib.parse.urlparse(self.path) + path = parsed.path + query = urllib.parse.parse_qs(parsed.query) + + try: + if path in ("/", "/index.html"): + self._send_html(ui_v2.PAGE) + elif path == "/classic": + self._send_html(ui.PAGE) + elif path == "/settings": + self._send_html(ui_settings.PAGE) + elif path == "/api/settings": + self._send_json(self.app.settings_payload()) + elif path == "/linkage": + self._send_html(ui_linkage.PAGE) + elif path == "/api/linkage": + self._send_json(self.app.scan.to_json()) + elif path == "/api/health": + self._send_json(self.app.health()) + elif path == "/api/alerts": + self._send_json(self.app.alert_queue(force=query.get("force", ["0"])[0] == "1")) + elif path == "/api/templates": + self._send_json(self.app.templates()) + elif path.startswith("/api/jobs/"): + job = self.app.jobs.get(path.rsplit("/", 1)[-1]) + self._send_json(job or {"error": "Unknown job."}, 200 if job else 404) + else: + self._send_json({"error": "Not found."}, 404) + except Exception as exc: + self._send_json({"error": f"{type(exc).__name__}: {exc}"}, 500) + + def do_POST(self) -> None: # noqa: N802 + path = urllib.parse.urlparse(self.path).path + body = self._body_json() + try: + force = bool(body.get("force")) + if path == "/api/triage": + if body.get("alert_id"): + self._send_json(self.app.triage_by_id(str(body["alert_id"]), force=force)) + elif isinstance(body.get("labels"), dict): + self._send_json(self.app.triage(body["labels"], body.get("annotations"), force=force)) + else: + self._send_json({"error": "Provide alert_id or labels."}, 400) + elif path == "/api/parse": + self._send_json(self.app.parse(str(body.get("text") or ""))) + elif path == "/api/actions/zendesk": + self._send_json(self.app.send_zendesk(body)) + elif path == "/api/settings": + self._send_json(self.app.settings_action(body)) + elif path == "/api/linkage/scan": + self._send_json(self.app.start_scan()) + elif path == "/api/linkage/enrich": + self._send_json(linkage.enrich(str(body.get("region") or ""), + str(body.get("openstack_id") or ""))) + else: + self._send_json({"error": "Not found."}, 404) + except Exception as exc: + self._send_json({"error": f"{type(exc).__name__}: {exc}"}, 500) + + +def serve(host: str = "127.0.0.1", port: int = 8765, prometheus_base: Optional[str] = None) -> None: + from .prometheus import DEFAULT_BASE + + prom = PrometheusClient(prometheus_base or DEFAULT_BASE) + app = App(prom) + print("Warming caches...") + app.warm() + Handler.app = app + httpd = ThreadingHTTPServer((host, port), Handler) + httpd.daemon_threads = True + print(f"cx-triage listening on http://{host}:{port}") + try: + httpd.serve_forever() + except KeyboardInterrupt: + print("\nshutting down") + finally: + httpd.server_close() diff --git a/triagelib/settings.py b/triagelib/settings.py new file mode 100644 index 0000000..a99c308 --- /dev/null +++ b/triagelib/settings.py @@ -0,0 +1,245 @@ +"""User settings: suppression rules and comms identity. + +Suppression rules replace hardcoded judgement calls. The internal-organisation +check used to be baked into the screening code, which meant the one person who +knew about it was whoever read the source. Now it ships as an editable default +rule that says what it does and why, and anyone can add their own. + +A rule matches when *every* condition it sets matches (AND); within one +condition, any listed value matches (OR). So "type is error AND organisation +contains modal" is one rule with two conditions. +""" +from __future__ import annotations + +import json +import os +import re +import threading +import time +import uuid +from typing import Any, Optional + +SETTINGS_DIR = os.path.expanduser(os.environ.get("CX_TRIAGE_HOME", "~/.cx-triage")) +SETTINGS_PATH = os.path.join(SETTINGS_DIR, "settings.json") + +# Conditions a rule can set. Every one is a substring match, case-insensitive, +# except `kind` and `region` which are exact. +CONDITIONS = { + "kind": "Alert type", + "organization": "Organisation contains", + "instance_name": "VM name contains", + "host": "Host contains", + "region": "Region", + "status": "Status is", +} + +DEFAULT_RULES: list[dict[str, Any]] = [ + { + "id": "builtin-internal-orgs", + "name": "Internal NexGen organisations", + "enabled": True, + "reason": "Owned by an internal test or platform organisation, not a customer.", + "conditions": {"organization": ["nexgencloud.com"]}, + }, + { + "id": "builtin-runpod-storage", + "name": "Runpod storage nodes (Luis)", + "enabled": True, + "reason": "Platform-owned storage nodes; SHUTOFF on these is expected and not customer-impacting.", + "conditions": {"kind": ["shutoff"], "instance_name": ["stor-runpod"]}, + }, +] + +DEFAULTS: dict[str, Any] = { + "rules": DEFAULT_RULES, + "agent_name": "", + "chronic_days": 3, +} + + +def _blank(value: Any) -> bool: + return value is None or str(value).strip() == "" + + +class Settings: + """Loaded once, written through on every change.""" + + def __init__(self, path: str = SETTINGS_PATH): + self.path = path + self._lock = threading.Lock() + self._data: dict[str, Any] = {} + self.load() + + # --- persistence ------------------------------------------------------- + + def load(self) -> None: + data = dict(DEFAULTS) + try: + with open(self.path, encoding="utf-8") as handle: + stored = json.load(handle) + if isinstance(stored, dict): + data.update(stored) + except (OSError, json.JSONDecodeError): + pass + data["rules"] = [r for r in (data.get("rules") or []) if isinstance(r, dict)] + with self._lock: + self._data = data + + def save(self) -> None: + with self._lock: + payload = json.dumps(self._data, indent=2, sort_keys=True) + try: + os.makedirs(os.path.dirname(self.path), exist_ok=True) + tmp = f"{self.path}.tmp" + with open(tmp, "w", encoding="utf-8") as handle: + handle.write(payload) + os.replace(tmp, self.path) + except OSError: + pass + + # --- accessors --------------------------------------------------------- + + @property + def rules(self) -> list[dict[str, Any]]: + with self._lock: + return [dict(r) for r in self._data.get("rules", [])] + + @property + def agent_name(self) -> str: + with self._lock: + return str(self._data.get("agent_name") or "") + + @property + def chronic_days(self) -> int: + with self._lock: + try: + return max(1, int(self._data.get("chronic_days") or 3)) + except (TypeError, ValueError): + return 3 + + def to_json(self) -> dict[str, Any]: + with self._lock: + return { + "rules": [dict(r) for r in self._data.get("rules", [])], + "agent_name": self._data.get("agent_name") or "", + "chronic_days": self._data.get("chronic_days", 3), + "conditions": CONDITIONS, + "path": self.path, + } + + # --- mutations --------------------------------------------------------- + + def set_general(self, agent_name: Optional[str] = None, chronic_days: Optional[Any] = None) -> None: + with self._lock: + if agent_name is not None: + self._data["agent_name"] = str(agent_name).strip() + if chronic_days is not None: + try: + self._data["chronic_days"] = max(1, int(chronic_days)) + except (TypeError, ValueError): + pass + self.save() + + def upsert_rule(self, rule: dict[str, Any]) -> dict[str, Any]: + clean = _normalize_rule(rule) + with self._lock: + rules = self._data.setdefault("rules", []) + for idx, existing in enumerate(rules): + if existing.get("id") == clean["id"]: + rules[idx] = clean + break + else: + rules.append(clean) + self.save() + return clean + + def delete_rule(self, rule_id: str) -> None: + with self._lock: + self._data["rules"] = [r for r in self._data.get("rules", []) if r.get("id") != rule_id] + self.save() + + def toggle_rule(self, rule_id: str, enabled: bool) -> None: + with self._lock: + for rule in self._data.get("rules", []): + if rule.get("id") == rule_id: + rule["enabled"] = bool(enabled) + self.save() + + +def _normalize_rule(rule: dict[str, Any]) -> dict[str, Any]: + conditions: dict[str, list[str]] = {} + for field, values in (rule.get("conditions") or {}).items(): + if field not in CONDITIONS: + continue + if isinstance(values, str): + values = [v.strip() for v in values.split(",")] + cleaned = [str(v).strip() for v in (values or []) if str(v).strip()] + if cleaned: + conditions[field] = cleaned + return { + "id": str(rule.get("id") or f"rule-{uuid.uuid4().hex[:8]}"), + "name": str(rule.get("name") or "Untitled rule").strip(), + "enabled": bool(rule.get("enabled", True)), + "reason": str(rule.get("reason") or "").strip(), + "conditions": conditions, + "created": rule.get("created") or time.strftime("%Y-%m-%d"), + } + + +# --- matching --------------------------------------------------------------- + +def _alert_field(alert: Any, field: str) -> str: + if field == "kind": + return str(getattr(alert, "kind", "")) + if field == "organization": + return f"{getattr(alert, 'org_id', '')} {getattr(alert, 'org_name', '')}" + if field == "instance_name": + return str(getattr(alert, "instance_name", "")) + if field == "host": + return str(getattr(alert, "host", "")) + if field == "region": + return f"{getattr(alert, 'region', '')} {getattr(alert, 'region_label', '')}" + if field == "status": + return str(getattr(alert, "status", "")) + return "" + + +def _condition_matches(field: str, values: list[str], alert: Any) -> bool: + actual = _alert_field(alert, field).lower() + if field in ("kind", "status"): + return any(actual == str(v).strip().lower() for v in values) + if field == "region": + return any(str(v).strip().lower() in actual for v in values) + return any(str(v).strip().lower() in actual for v in values) + + +def rule_matches(rule: dict[str, Any], alert: Any) -> bool: + """Every condition in the rule must match (AND).""" + conditions = rule.get("conditions") or {} + if not conditions: + return False # an empty rule would swallow the whole queue + return all(_condition_matches(f, v, alert) for f, v in conditions.items()) + + +def first_match(settings: Settings, alert: Any) -> Optional[dict[str, Any]]: + for rule in settings.rules: + if rule.get("enabled") and rule_matches(rule, alert): + return rule + return None + + +def preview(settings: Settings, rule: dict[str, Any], alerts: list[Any]) -> list[dict[str, Any]]: + """Which currently-firing alerts a rule would hide - shown before saving.""" + clean = _normalize_rule(rule) + hits = [] + for alert in alerts: + if rule_matches(clean, alert): + hits.append({ + "kind": alert.kind, + "title": alert.title, + "instance_name": alert.instance_name, + "host": alert.host, + "org_name": alert.org_name, + "region": alert.region, + }) + return hits diff --git a/triagelib/ui.py b/triagelib/ui.py new file mode 100644 index 0000000..81cfc6f --- /dev/null +++ b/triagelib/ui.py @@ -0,0 +1,429 @@ +"""The single-page UI, served inline so the app needs no assets or CDN.""" +from __future__ import annotations + +PAGE = r""" + + + + +CX Triage + + + +
+

CX Triage alert diagnosis over CX-Tools

+ read-only · suggests, never acts +
+ +
+ +
+
+
+
CX runbooks
+
Infrastructure
+
+
+
+ + +
+
+ Paste an alert manually + + +
+
+
+
+ +
Pick an alert to diagnose.
+
+ + + + +""" diff --git a/triagelib/ui_linkage.py b/triagelib/ui_linkage.py new file mode 100644 index 0000000..86da5bb --- /dev/null +++ b/triagelib/ui_linkage.py @@ -0,0 +1,181 @@ +"""Linkage scan page: Infrahub records and OpenStack servers that lost each other.""" +from __future__ import annotations + +PAGE = r""" + + + + +CX Triage — Linkage + + + +
+
CX Triage — linkage scan
+ ← Alert queue +
+ + +
+ +
+

A VM in ERROR is not always a failed build. If the server was created but the link back to + Infrahub was never written, Infrahub shows ERROR or CREATING with no usable openstack_id while a + perfectly good server of the same name is running. This scans both sides in bulk and pairs them up by name — + and finds the reverse too: OpenStack servers that no Infrahub record claims, which is what + Suspected Orphan VM was meant to catch before its input metric went empty.

+ +
Run a scan to begin. It lists every server in all four regions, so it takes a few minutes.
+
+ + + + +""" diff --git a/triagelib/ui_settings.py b/triagelib/ui_settings.py new file mode 100644 index 0000000..61f9235 --- /dev/null +++ b/triagelib/ui_settings.py @@ -0,0 +1,221 @@ +"""Settings page: suppression rules and comms identity.""" +from __future__ import annotations + +PAGE = r""" + + + + +CX Triage — Settings + + + +
+
CX Triage — settings
+ ← Alert queue + Linkage scan +
+ +
+ +
+

Suppression rules

+

Hide alerts you already know about. A rule fires when every condition it sets matches, + so you can combine them — for example type error and organisation containing + modal. Suppressed alerts are not deleted: they stay reachable under the + hidden by a rule filter on the queue.

+ +
+ + +
+ +

Comms identity

+
+
+ +
+
Appended after “Kind Regards” in customer emails.
+ +
+
An alert whose condition has held longer than this is treated as chronic rather than new + work — except for types with a runbook time commitment, which become overdue instead.
+
+
+
+ +

Where this is stored

+
+
+ + + + +""" diff --git a/triagelib/ui_v2.py b/triagelib/ui_v2.py new file mode 100644 index 0000000..c8551e6 --- /dev/null +++ b/triagelib/ui_v2.py @@ -0,0 +1,572 @@ +"""Action-oriented UI. + +Design intent: the reader already knows the runbooks. Each case shows the state +of the world as a picture, one line of verdict, and the buttons that actually +move it forward. Everything else is collapsed. +""" +from __future__ import annotations + +PAGE = r""" + + + + +CX Triage + + + +
+
CX Triage
+ Linkage scan + Settings +
+ + +
+
+
Status
+
Type
+
+ +
+ +
Select a case.
+
+ +
+ + + + + +"""