Split into a FastAPI backend and a React frontend, add case state and SSO

The single-file stdlib server became the limit: no way to track what had been
done about an alert, no accounts, and a UI that had to be hand-rolled in
template strings. This restructures it into something deployable.

Backend (FastAPI)
- app/ holds config, database, auth, delivery and the routers; triagelib keeps
  the triage engine unchanged, so the validated screening and runbook logic is
  untouched.
- Cases persist per alert fingerprint with a status workflow (investigating,
  customer contacted, escalated to Infra, waiting, remediated, resolved, won't
  fix, false positive), an assignee, notes and an append-only history. An alert
  that stops and re-fires lands back on the same case and counts as a reopen.
- Suppression rules move from a JSON file into the database.

Auth
- Signed session cookies over PBKDF2 local accounts, plus an OIDC flow ready for
  Authentik: users are created on first login and admin follows a group claim.
  Local login can be switched off entirely once SSO is live.

Zendesk and Jira
- Delivery is now implemented, behind three gates: the integration must be
  configured, its feature flag on, and CX_FEATURE_SEND_ENABLED on. A demo
  instance leaves the last off and cannot mail anyone. Both search before
  creating, so re-diagnosing an alert updates one ticket rather than opening
  several, and a rolling daily cap stops a loop mailing everybody.

Deployment
- Multi-stage Dockerfile builds the bundle and serves it from the API origin.
- docker-compose for local and single-host use; Gitea Actions runs the tests,
  builds the image and renders deploy/k8s with envsubst.

Two fixes found while testing: assigning a case returned a null assignee, and
add_event could leave an already-loaded history collection stale.

Known gap: the engine reaches OpenStack via `docker exec <region>-osc`, which
does not work in a pod without the CX-Tools containers alongside it.
docs/DEPLOYMENT.md sets out the three ways to close that.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

docs/DEPLOYMENT.md

@@ -0,0 +1,87 @@
+# Deployment
+
+## Local
+
+```bash
+cp .env.example .env        # set CX_SECRET_KEY and CX_BOOTSTRAP_ADMIN_PASSWORD
+docker compose up --build
+```
+
+<http://localhost:8080>, sign in with the bootstrap admin.
+
+Without Docker:
+
+```bash
+cd backend && pip install -r requirements.txt
+uvicorn app.main:app --reload --port 8080     # terminal 1
+cd frontend && npm install && npm run dev      # terminal 2 -> :5173, proxies /api
+```
+
+## Kubernetes via Gitea Actions
+
+`.gitea/workflows/ci.yaml` runs tests → builds the image → renders
+`deploy/k8s/*.yaml` with `envsubst` → applies them.
+
+### Repository **variables** (not secret)
+
+| Variable | Example | Meaning |
+|---|---|---|
+| `CX_DOMAIN` | `cx-triage.ngbackend.cloud` | Ingress host; `CX_BASE_URL` derives from it |
+| `K8S_NAMESPACE` | `cx-triage` | Target namespace |
+| `REGISTRY` | `git.ngbackend.cloud` | Image registry |
+| `IMAGE_NAME` | `parham.monfared/cx-ui` | Image repository |
+| `CX_PROMETHEUS_BASE` | `http://10.11.254.250:9090` | Alert source |
+| `CX_OIDC_ENABLED` | `true` | Authentik on |
+| `CX_OIDC_ISSUER` | `https://sso…/application/o/cx-triage/` | Discovery base |
+| `CX_OIDC_ADMIN_GROUP` | `cx-triage-admins` | Group granting admin |
+| `CX_FEATURE_SEND_ENABLED` | `false` | **Master send switch** |
+| `CX_FEATURE_ZENDESK` / `CX_FEATURE_JIRA` | `false` | Per-integration flags |
+| `CX_FEATURE_LINKAGE_SCAN` | `true` | Expensive scan on/off |
+| `CX_SEND_DAILY_CAP` | `25` | Rolling 24h send limit |
+| `CX_JIRA_PROJECT` / `CX_JIRA_BASE` | `INFRA` | Jira target |
+| `INGRESS_CLASS` / `TLS_ISSUER` | `nginx` / `letsencrypt-prod` | Ingress wiring |
+| `REPLICAS` | `1` | See the caveat below |
+
+### Repository **secrets**
+
+`KUBECONFIG` (base64), `REGISTRY_USERNAME`, `REGISTRY_TOKEN`, `CX_SECRET_KEY`,
+`CX_DATABASE_URL`, `CX_OIDC_CLIENT_ID`, `CX_OIDC_CLIENT_SECRET`,
+`CX_ZENDESK_SUBDOMAIN`, `CX_ZENDESK_EMAIL`, `CX_ZENDESK_TOKEN`,
+`CX_JIRA_EMAIL`, `CX_JIRA_TOKEN`, `CX_BOOTSTRAP_ADMIN_PASSWORD`.
+
+## Authentik
+
+1. **Applications → Providers → Create → OAuth2/OpenID Provider**
+   - Client type: **Confidential**
+   - Redirect URI: `https://<CX_DOMAIN>/api/auth/oidc/callback`
+   - Scopes: `openid`, `email`, `profile`
+2. Create the Application and bind the provider.
+3. Copy the client ID/secret into the Gitea secrets above.
+4. Create a group `cx-triage-admins`; its members get admin rights.
+5. Set `CX_OIDC_ISSUER` to the provider's OpenID configuration base URL — the
+   app appends `/.well-known/openid-configuration`.
+
+Users are created on first login. `CX_AUTH_LOCAL_ENABLED=false` in the cluster
+config turns off password login entirely once SSO works.
+
+## Two things to plan for
+
+**Reaching OpenStack.** The diagnosis engine shells out to
+`docker exec <region>-osc openstack …`, and the internal Prometheus is only
+reachable from inside those containers. That works on a laptop with CX-Tools
+running; it does **not** work in a pod by default. Options, cheapest first:
+
+1. Run the app on a host that already has the CX-Tools containers, mounting the
+   Docker socket (what `docker-compose.yml` does).
+2. Run the `*-osc` containers as sidecars in the pod.
+3. Replace `cxbridge.os_json` with direct authenticated OpenStack API calls and
+   give the pod a network route. Cleanest, most work.
+
+Until one of those is in place, a cluster deployment can read Prometheus (if
+routable) but per-alert diagnosis will fail. That is a real gap, not an
+oversight.
+
+**`REPLICAS` should stay at 1** for now. The Prometheus caches, the background
+triage jobs and the linkage scan are per-process, so a second replica would
+duplicate the work and serve inconsistent job IDs. Moving jobs into the database
+or a queue is what unlocks scaling out.

docs/INTEGRATIONS.md

@@ -0,0 +1,133 @@
+# Enabling Zendesk and Jira
+
+Nothing leaves this app until **three separate gates** are open. Until then the
+UI builds the full payload, shows it to you, and the Send button stays disabled
+with the reason written on it.
+
+```
+  1. the integration is configured   CX_ZENDESK_* / CX_JIRA_*
+  2. its feature flag is on          CX_FEATURE_ZENDESK / CX_FEATURE_JIRA
+  3. sending is enabled globally     CX_FEATURE_SEND_ENABLED
+```
+
+Gate 3 is the important one. A demo or staging instance simply leaves it off,
+and then no combination of clicks can email a customer.
+
+Check where you stand at any time on **Settings → Integrations**, or:
+
+```bash
+curl -s localhost:8080/api/health | python3 -m json.tool
+```
+
+---
+
+## Zendesk
+
+### 1. Create an API token
+
+Zendesk **Admin Center → Apps and integrations → APIs → Zendesk API**, turn on
+*Token access*, then **Add API token**. Copy it — Zendesk shows it once.
+
+### 2. Decide which agent owns the tickets
+
+Use a dedicated agent (e.g. `cx-triage@…`) rather than a person's account, so
+the audit trail stays clear when someone leaves.
+
+### 3. Set the variables
+
+```bash
+CX_FEATURE_ZENDESK=true
+CX_ZENDESK_SUBDOMAIN=nexgencloud        # from https://<this>.zendesk.com
+CX_ZENDESK_EMAIL=cx-triage@nexgencloud.com
+CX_ZENDESK_TOKEN=<the token>
+CX_ZENDESK_PUBLIC_REPLY=true            # false posts an internal note instead
+
+CX_FEATURE_SEND_ENABLED=true            # the master switch
+```
+
+Restart. The Send button becomes live.
+
+### What happens on send
+
+1. Searches for `external_id:cx-triage-<fingerprint>`.
+2. If a ticket exists → adds a comment. If not → creates one, with the requester
+   set from the Infrahub owner, priority from the screening verdict, and tags
+   `cx-triage`, `alert-<kind>`.
+3. Records a `zendesk_sent` event on the case and moves it to
+   **Customer contacted**.
+
+So re-diagnosing the same alert updates one ticket instead of opening five.
+
+---
+
+## Jira
+
+### 1. Create an API token
+
+<https://id.atlassian.com/manage-profile/security/api-tokens> → *Create API
+token*.
+
+### 2. Confirm the project and issue type
+
+The defaults are `INFRA` / `Task`. If your Infrastructure project uses something
+else, set it — a wrong `issuetype` is the usual cause of a 400 from Jira.
+
+### 3. Set the variables
+
+```bash
+CX_FEATURE_JIRA=true
+CX_JIRA_BASE=https://nexgencloud.atlassian.net
+CX_JIRA_EMAIL=cx-triage@nexgencloud.com
+CX_JIRA_TOKEN=<the token>
+CX_JIRA_PROJECT=INFRA
+CX_JIRA_ISSUE_TYPE=Task
+
+CX_FEATURE_SEND_ENABLED=true
+```
+
+Issues are labelled `cx-triage-<fingerprint>` and searched for before creating,
+so the same alert never opens two tickets.
+
+---
+
+## Where the credentials go
+
+**Never commit them.**
+
+| Where | How |
+|---|---|
+| Local | `.env` (git-ignored) — copy from `.env.example` |
+| Kubernetes | The `cx-triage` Secret, written by the pipeline from Gitea secrets |
+| Gitea | Repository → Settings → Actions → Secrets |
+
+The deploy job creates the Secret imperatively from the secret store, so no
+credential is ever in a manifest in git.
+
+---
+
+## Safety rails that stay on
+
+- Every send needs a click **and** a confirm naming the recipient.
+- No auto-send: a verdict never triggers an email by itself.
+- `CX_SEND_DAILY_CAP` (default 25) refuses further sends in a rolling 24 hours,
+  so a loop cannot mail every customer.
+- The body is editable before sending.
+- Every attempt is written to the case history — including failures.
+- Deleting, shelving and InfraInsight edits stay copy-a-command. The read-only
+  guarantee is what makes this safe against production.
+
+## First run
+
+Point at a **Zendesk sandbox**, or send the first ticket to your own address by
+editing the To field. Once one round trip looks right, switch it on for real.
+
+## When it fails
+
+| Symptom | Cause |
+|---|---|
+| Button disabled, "sending is switched off" | `CX_FEATURE_SEND_ENABLED` is false |
+| Button disabled, "not configured" | A `CX_ZENDESK_*` / `CX_JIRA_*` value is missing |
+| `401` from Zendesk | The email must be the **agent** address, and token access must be enabled |
+| `400` from Jira | Usually `issuetype` or `project` does not exist |
+| "Daily send cap reached" | Raise `CX_SEND_DAILY_CAP` if deliberate |
+| "Open the case first" | The alert has no case yet — open it in the queue once |

docs/LINKAGE.md

@@ -0,0 +1,120 @@
+# How the linkage scan works
+
+## The problem it solves
+
+Infrahub and OpenStack are two databases that are supposed to agree about which
+VMs exist. The link between them is a single field: `openstack_id` on the Infrahub
+record.
+
+When a VM is created, roughly: Infrahub writes a record → asks OpenStack to build
+a server → OpenStack returns a UUID → Infrahub stores that UUID. If the last step
+fails, you get a record stuck in `CREATING` or `ERROR` with no `openstack_id`,
+while the server itself is running perfectly.
+
+On an alert dashboard that is indistinguishable from a genuine build failure. They
+need opposite responses:
+
+| | Genuine failure | Linkage failure |
+|---|---|---|
+| Server exists? | no | **yes, and running** |
+| Fix | customer recreates | repair the Infrahub record |
+| Billing | nothing to bill | **running, billing nobody** |
+| Customer sees | "my VM failed" | "my VM failed" |
+
+Tell a customer to recreate a VM that is actually running and you have doubled
+their spend on a machine they think is broken.
+
+## The scan, step by step
+
+### 1. Pull everything OpenStack has
+
+For each region, one call:
+
+```
+openstack server list --all-projects --long -f json
+```
+
+~2,500 rows for ca1, ~500 for no1, ~80 for us1. Each row gives ID, name, status,
+task state, host, project. About 45 seconds per region, so ~2.5 minutes total —
+which is why it is a button, not something that runs on page load.
+
+Two indexes are built: **by UUID** and **by lowercased name**.
+
+### 2. Pull everything Infrahub has
+
+Free — it is already in the `Resources` metric snapshot the queue uses (~4,300
+records, one Prometheus query). Each carries `openstack_id`, `instance_name`,
+`status`, `region`, `organization`, and GPU count.
+
+### 3. Only judge regions that actually answered
+
+If a region's listing failed, every VM in it would look "missing from OpenStack".
+So records in unscanned regions are **excluded entirely** and counted separately.
+
+This is not hypothetical. On the first real run `ca2` failed
+(`Could not find versioned identity endpoints`) and produced **317 false
+positives**. With the filter, the same scan returns **6**.
+
+### 4. Find Infrahub records with a broken link
+
+For each Infrahub record whose status is one where a live server is expected —
+`ERROR`, `CREATING`, `BUILD`, `ACTIVE`, `REBOOTING`, `RESTORING` — flag it if:
+
+- it has **no** `openstack_id`, or
+- it has one that **is not in the OpenStack index**
+
+`HIBERNATED` is deliberately excluded: a shelved instance legitimately has no
+running server, and including it would drown the result (2,400 records).
+
+### 5. Pair them up by name
+
+For each flagged record, look for OpenStack servers with the **same name**. In
+Hyperstack the OpenStack server name matches the Infrahub instance name, which
+makes this a strong signal.
+
+Confidence is then graded:
+
+| Confidence | Meaning |
+|---|---|
+| **high** | A same-named server exists, **no other Infrahub record claims it**, and it is not itself in ERROR. This is a linkage failure — the server is real and orphaned. |
+| **medium** | A same-named server exists but another Infrahub record already owns that UUID, or it is SHELVED_OFFLOADED. Could be a name collision; needs a human. |
+| **none** | No same-named server. The record's server genuinely does not exist — a real failure, or it was deleted. |
+
+**Details** on a row runs `server show` for the candidate to fetch its creation
+time, launch time and fault, so you can confirm the timing lines up with when the
+Infrahub record was created.
+
+### 6. The reverse: servers nobody claims
+
+Every OpenStack server whose UUID appears in **no** Infrahub record. These are
+running, consuming GPUs, and billing nobody. The scan also notes whether the
+*name* is known to Infrahub, which separates "record exists but the link is
+broken" from "completely unknown".
+
+This is what `:pirate_flag:Suspected Orphan VM` is supposed to catch — but that
+rule is built on `openstack_nova_server_status`, which currently returns zero
+series, so it cannot fire at all. Until that exporter is fixed, this scan is the
+only thing looking.
+
+## What it found on the first live run
+
+- **`daring-rutherford`** — Infrahub `CREATING`, no `openstack_id` recorded;
+  OpenStack had an **ACTIVE** server of exactly that name in ca1 that no other
+  record claimed. Textbook linkage failure. (It had linked itself by the next
+  scan — caught mid-flight.)
+- **177 OpenStack servers** with no Infrahub record at all.
+- **ca2 unreachable**, so that region is excluded and reported rather than
+  silently producing garbage.
+
+## Known limits
+
+- **Name matching is a heuristic.** Two VMs can share a name across orgs. That is
+  what the confidence grade and the "claimed by another record" flag are for —
+  nothing here should be actioned without opening the candidate.
+- **No creation-time matching yet.** The bulk list does not include creation
+  timestamps, so time correlation is per-candidate, on demand. If you want
+  time-window matching as a primary signal rather than a confirmation, that needs
+  a `server show` per candidate — fine for tens, not for thousands.
+- **Point in time.** A VM mid-build will look unlinked. Two scans a few minutes
+  apart separate "still settling" from "actually stuck".
+- **ca2 is currently unscannable** — an environment problem, not a scan problem.

docs/PLAN.md

@@ -0,0 +1,130 @@
+# From "here's what to do" to "do it"
+
+## 1. Validation against live data
+
+Every recommendation was checked against OpenStack directly, not against the
+app's own view. Results:
+
+| Case | App said | Ground truth | Verdict |
+|---|---|---|---|
+| **Rogue VM `CA1-ESC8-057`** | "No mismatch — go check InfraInsight for HIBERNATED VMs carrying a host" | OpenStack 7 GPUs across 5 VMs; Infrahub 7 GPUs across the same 5 VMs. Perfect agreement. No hibernated VMs on the host at all. | **Wrong** |
+| **Rogue VM, 10 hosts sampled** | 50 alerts "needs action" | OpenStack and Infrahub agreed **exactly** on all 10 | **Wrong** |
+| `SHUTOFF` sn56-week6-transfer-e1 | Billing notice to `shettyatulya@gmail.com` | OpenStack `SHUTOFF`, no fault | Correct |
+| `DELETING` hs-gaussian-splatting | Intent verified from events; delete in OpenStack | OpenStack `ACTIVE`, delete requested 17:00 | Correct |
+| `DELETING` c3-pool-1785947188798 | Intent verified; delete in OpenStack | OpenStack `ERROR`, host `None`, "No valid host was found" | Correct action, **incomplete** — it never built, so the customer is owed the creation-failure notice too |
+| `ERROR` hyperstack-minion-lydhjev | `no_valid_host` fault → never ACTIVE → insufficient-stock template → Lars Vagnes | Fault is exactly "No valid host was found. There are not enough hosts available."; host `None` | Correct — but was **hidden as "chronic"** |
+
+### The Rogue VM rule is broken at source
+
+```promql
+sum by (instance) (In_Use_Gpus)
+  - sum by (instance) (Resources{status=~"ACTIVE|SHUTOFF|PRE_ACTIVE"}) >= 1
+```
+
+`In_Use_Gpus` equals `Total_Gpus` — the **physical** GPU count — on **71 of the 75**
+firing hosts. So the expression reduces to *physical GPUs minus allocated GPUs*,
+i.e. **"this host has at least one free GPU."** That is spare capacity, not a
+rogue VM.
+
+`CA1-ESC8-057` is the clean example: 8 physical, 7 allocated across 5 VMs that
+both systems agree on, so it fires with a gap of 1.
+
+Two consequences:
+- 65 of the 72 rogue-VM alerts are now classified **invalid — alert rule defect**
+  and screened out, with the reason stated.
+- The 2 that survive are genuine: `CA1-ESC812-252` (In_Use 2, one 1-GPU VM) and
+  `CA1-ESC812-289` (In_Use 7, 5 GPUs allocated). Both now read *"N GPU(s) in use
+  belong to no instance on either side"* and route to Infrastructure, because no
+  CX action can fix a leaked hypervisor allocation.
+
+**This needs fixing in the Prometheus rule, not just filtered here.** Whoever owns
+`infrahub-rules.yml` should either repair `In_Use_Gpus` or rewrite the expression.
+
+### "Chronic" was hiding overdue customer contact
+
+13 ERROR alerts have held for 6–7 days. The ERROR runbook says to contact the
+customer if a stock-failure instance is not deleted **within a day**. Filing those
+as "chronic, probably already ticketed" was wrong — they are overdue.
+
+New `overdue` verdict: for kinds with a runbook time commitment, age makes a case
+*more* urgent and it can never be demoted to chronic.
+
+### Net effect
+
+| | before validation | after |
+|---|---|---|
+| To action | 54 | **19** |
+| of which overdue customer contact | 0 (hidden) | **13** |
+| Rogue VM noise | 50 "needs action" | 65 flagged as a rule defect |
+
+---
+
+## 2. Integration plan
+
+The principle: the app already knows *who* to contact and *what* to say. Wiring
+delivery turns a 5-minute copy-paste into one reviewed click — without ever
+sending on its own.
+
+### Phase 1 — Zendesk (the main win)
+
+`integrations.build_zendesk()` already produces a complete `POST /api/v2/tickets`
+body: requester resolved from Infrahub owners, subject, the approved runbook
+wording with placeholders filled, priority derived from the verdict, tags, and
+`external_id = cx-triage-<fingerprint>` for idempotency.
+
+To finish it:
+
+1. **Credentials** — `CX_ZENDESK_SUBDOMAIN`, `CX_ZENDESK_EMAIL`, `CX_ZENDESK_TOKEN`
+   (API token, Basic auth as `email/token:token`). Read from 1Password via the
+   existing CX-Tools loader rather than env vars, so nothing lands on disk.
+2. **Search before create** — `GET /api/v2/search?query=external_id:<fp>` so a
+   re-diagnosed alert updates the existing ticket instead of opening a duplicate.
+3. **Send** — implement `App.send_zendesk`, which currently refuses by design.
+4. **Guard rails** (all already scaffolded in the UI):
+   - two-step confirm naming the recipient
+   - never auto-send; no bulk send in v1
+   - an `--allow-send` startup flag, so a demo instance physically cannot email
+   - append the ticket URL back onto the case and log it to `vmc-audit.log`
+5. **Requester matching** — Infrahub gives owner name + email; Zendesk may already
+   have that user. Search by email, fall back to creating the requester inline.
+
+### Phase 2 — Jira
+
+`integrations.build_jira()` produces the `POST /rest/api/3/issue` body with the
+evidence block already assembled. Needs: project key confirmation (`INFRA`?),
+issue type, and the same search-before-create against a `cx-triage-<fp>` label.
+The GPU-leak escalations above are the immediate use case.
+
+### Phase 3 — closing the loop
+
+- **Slack** — react to the alert in `#infrahub-errors` and thread the findings,
+  which the runbook asks for manually today.
+- **Case state** — persist handled/snoozed/ticketed per fingerprint (SQLite) so
+  the queue reflects work already done and survives a restart.
+- **Bulk actions** — the 13 overdue ERROR alerts are one org-grouped mail-merge;
+  worth doing only once single-send is trusted.
+
+### What stays manual, deliberately
+
+Deleting, shelving and InfraInsight edits stay copy-a-command. The read-only
+guarantee is what makes this safe to run against production, and the destructive
+steps are exactly where a wrong verdict would be expensive.
+
+---
+
+## 3. UI
+
+Live at `http://127.0.0.1:8765/` (previous version kept at `/classic`).
+
+- **Left rail** — one line per case: subject, org, region, how long the condition
+  has actually held. Grouped and collapsible. Verdict filter chips across the top.
+- **Case card** — verdict as a sentence, then a *picture* of the problem:
+  - state alerts: `Infrahub says X` ≠ `OpenStack says Y`
+  - GPU cases: a segmented allocation bar (green allocated / red unaccounted)
+  - duplicate IPs: one row per claimant
+- **"Do this"** — the customer's name and address, then the action buttons.
+  Nothing else competes with them.
+- **Everything else collapsed** — Caveats, Why, Runbook steps, Raw evidence.
+- **Zendesk drawer** — recipient, subject, editable body, priority, tags, and a
+  Send button that is disabled and labelled *"Preview only. Nothing will be sent."*
+  until credentials exist and sending is explicitly enabled.