Split into a FastAPI backend and a React frontend, add case state and SSO

The single-file stdlib server became the limit: no way to track what had been
done about an alert, no accounts, and a UI that had to be hand-rolled in
template strings. This restructures it into something deployable.

Backend (FastAPI)
- app/ holds config, database, auth, delivery and the routers; triagelib keeps
  the triage engine unchanged, so the validated screening and runbook logic is
  untouched.
- Cases persist per alert fingerprint with a status workflow (investigating,
  customer contacted, escalated to Infra, waiting, remediated, resolved, won't
  fix, false positive), an assignee, notes and an append-only history. An alert
  that stops and re-fires lands back on the same case and counts as a reopen.
- Suppression rules move from a JSON file into the database.

Auth
- Signed session cookies over PBKDF2 local accounts, plus an OIDC flow ready for
  Authentik: users are created on first login and admin follows a group claim.
  Local login can be switched off entirely once SSO is live.

Zendesk and Jira
- Delivery is now implemented, behind three gates: the integration must be
  configured, its feature flag on, and CX_FEATURE_SEND_ENABLED on. A demo
  instance leaves the last off and cannot mail anyone. Both search before
  creating, so re-diagnosing an alert updates one ticket rather than opening
  several, and a rolling daily cap stops a loop mailing everybody.

Deployment
- Multi-stage Dockerfile builds the bundle and serves it from the API origin.
- docker-compose for local and single-host use; Gitea Actions runs the tests,
  builds the image and renders deploy/k8s with envsubst.

Two fixes found while testing: assigning a case returned a null assignee, and
add_event could leave an already-loaded history collection stale.

Known gap: the engine reaches OpenStack via `docker exec <region>-osc`, which
does not work in a pod without the CX-Tools containers alongside it.
docs/DEPLOYMENT.md sets out the three ways to close that.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

backend/Dockerfile

@@ -0,0 +1,35 @@
+# ---- frontend ---------------------------------------------------------------
+FROM node:22-alpine AS ui
+WORKDIR /ui
+COPY frontend/package.json frontend/package-lock.json* ./
+RUN npm ci --no-audit --no-fund 2>/dev/null || npm install --no-audit --no-fund
+COPY frontend/ .
+RUN npm run build
+
+# ---- backend ----------------------------------------------------------------
+FROM python:3.12-slim
+ENV PYTHONUNBUFFERED=1 PYTHONDONTWRITEBYTECODE=1
+
+# curl is what the triage engine shells out to for the Infrahub API; the docker
+# CLI is only needed when Prometheus/OpenStack are reachable through the
+# CX-Tools containers rather than directly (see docs/DEPLOYMENT.md).
+RUN apt-get update \
+ && apt-get install -y --no-install-recommends curl ca-certificates docker.io \
+ && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /app
+COPY backend/requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+
+COPY backend/app ./app
+COPY backend/triagelib ./triagelib
+COPY --from=ui /ui/dist ./static
+
+RUN useradd --uid 10001 --create-home cx && mkdir -p /data && chown -R cx /data /app
+USER cx
+
+ENV CX_STATIC_DIR=/app/static CX_DATABASE_URL=sqlite:////data/cx-triage.db
+EXPOSE 8080
+HEALTHCHECK --interval=30s --timeout=5s --start-period=40s \
+  CMD python -c "import urllib.request;urllib.request.urlopen('http://127.0.0.1:8080/api/health').read()"
+CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8080"]

backend/app/__init__.py

@@ -0,0 +1,2 @@
+"""CX Triage backend."""
+VERSION = "0.2.0"

backend/app/auth.py

@@ -0,0 +1,253 @@
+"""Authentication: signed session cookies, local accounts, and OIDC.
+
+Local accounts exist so the tool runs on a laptop and in a cluster that has not
+been wired to SSO yet. When CX_OIDC_ENABLED is set, Authentik becomes the source
+of truth: users are created on first login, and admin rights follow a group
+claim rather than being set here.
+
+The session itself is the same either way - a signed, expiring cookie - so the
+rest of the app never has to care which provider a user came from.
+"""
+from __future__ import annotations
+
+import base64
+import datetime as dt
+import hashlib
+import hmac
+import json
+import secrets
+import time
+from typing import Any, Optional
+
+import httpx
+from fastapi import Depends, HTTPException, Request, status
+from sqlalchemy.orm import Session
+
+from .config import get_settings
+from .db import get_db
+from .models import AuthProvider, User
+
+SESSION_COOKIE = "cx_session"
+settings = get_settings()
+
+
+# --- password hashing -------------------------------------------------------
+# PBKDF2 from the standard library: no native build step in the image, and
+# strong enough for a small internal user table.
+
+def hash_password(password: str) -> str:
+    salt = secrets.token_bytes(16)
+    digest = hashlib.pbkdf2_hmac("sha256", password.encode(), salt, 240_000)
+    return f"pbkdf2_sha256$240000${base64.b64encode(salt).decode()}${base64.b64encode(digest).decode()}"
+
+
+def verify_password(password: str, stored: str) -> bool:
+    try:
+        algo, rounds, salt_b64, digest_b64 = stored.split("$")
+        if algo != "pbkdf2_sha256":
+            return False
+        expected = base64.b64decode(digest_b64)
+        actual = hashlib.pbkdf2_hmac("sha256", password.encode(),
+                                     base64.b64decode(salt_b64), int(rounds))
+        return hmac.compare_digest(expected, actual)
+    except (ValueError, TypeError):
+        return False
+
+
+# --- session cookie ---------------------------------------------------------
+
+def _sign(payload: bytes) -> str:
+    mac = hmac.new(settings.secret_key.encode(), payload, hashlib.sha256).digest()
+    return f"{base64.urlsafe_b64encode(payload).decode()}.{base64.urlsafe_b64encode(mac).decode()}"
+
+
+def issue_session(user_id: int) -> str:
+    payload = json.dumps({
+        "uid": user_id,
+        "exp": int(time.time()) + settings.session_hours * 3600,
+    }).encode()
+    return _sign(payload)
+
+
+def read_session(token: str) -> Optional[int]:
+    try:
+        body_b64, mac_b64 = token.split(".")
+        payload = base64.urlsafe_b64decode(body_b64)
+        expected = hmac.new(settings.secret_key.encode(), payload, hashlib.sha256).digest()
+        if not hmac.compare_digest(expected, base64.urlsafe_b64decode(mac_b64)):
+            return None
+        data = json.loads(payload)
+        if int(data.get("exp", 0)) < time.time():
+            return None
+        return int(data["uid"])
+    except (ValueError, TypeError, KeyError, json.JSONDecodeError):
+        return None
+
+
+# --- dependencies -----------------------------------------------------------
+
+def current_user(request: Request, db: Session = Depends(get_db)) -> User:
+    token = request.cookies.get(SESSION_COOKIE, "")
+    uid = read_session(token) if token else None
+    if not uid:
+        raise HTTPException(status.HTTP_401_UNAUTHORIZED, "Not signed in")
+    user = db.get(User, uid)
+    if not user or not user.is_active:
+        raise HTTPException(status.HTTP_401_UNAUTHORIZED, "Account is not active")
+    return user
+
+
+def require_admin(user: User = Depends(current_user)) -> User:
+    if not user.is_admin:
+        raise HTTPException(status.HTTP_403_FORBIDDEN, "Administrator access required")
+    return user
+
+
+def optional_user(request: Request, db: Session = Depends(get_db)) -> Optional[User]:
+    token = request.cookies.get(SESSION_COOKIE, "")
+    uid = read_session(token) if token else None
+    return db.get(User, uid) if uid else None
+
+
+# --- local accounts ---------------------------------------------------------
+
+def authenticate_local(db: Session, email: str, password: str) -> Optional[User]:
+    if not settings.auth_local_enabled:
+        return None
+    user = db.query(User).filter(User.email == email.strip().lower()).one_or_none()
+    if not user or not user.is_active or user.provider != AuthProvider.LOCAL:
+        return None
+    if not user.password_hash or not verify_password(password, user.password_hash):
+        return None
+    return user
+
+
+def ensure_bootstrap_admin(db: Session) -> Optional[str]:
+    """Create the first admin so a fresh deployment is reachable.
+
+    Only ever runs when the user table is empty, and only with a password
+    supplied through the environment - it never invents one.
+    """
+    if db.query(User).count():
+        return None
+    if not settings.bootstrap_admin_password:
+        return ("No users exist and CX_BOOTSTRAP_ADMIN_PASSWORD is unset. "
+                "Set it and restart, or sign in through SSO.")
+    db.add(User(
+        email=settings.bootstrap_admin_email.strip().lower(),
+        name="Bootstrap admin",
+        is_admin=True,
+        provider=AuthProvider.LOCAL,
+        password_hash=hash_password(settings.bootstrap_admin_password),
+    ))
+    db.commit()
+    return f"Created bootstrap admin {settings.bootstrap_admin_email}"
+
+
+# --- OIDC / Authentik -------------------------------------------------------
+
+class OIDCError(RuntimeError):
+    pass
+
+
+_discovery_cache: dict[str, Any] = {}
+
+
+async def oidc_discovery() -> dict[str, Any]:
+    """Fetch and cache the provider metadata."""
+    if not settings.oidc_issuer:
+        raise OIDCError("CX_OIDC_ISSUER is not set.")
+    if _discovery_cache.get("_issuer") == settings.oidc_issuer:
+        return _discovery_cache
+    url = settings.oidc_issuer.rstrip("/") + "/.well-known/openid-configuration"
+    async with httpx.AsyncClient(timeout=10) as client:
+        resp = await client.get(url)
+    if resp.status_code != 200:
+        raise OIDCError(f"OIDC discovery failed ({resp.status_code}) at {url}")
+    data = resp.json()
+    data["_issuer"] = settings.oidc_issuer
+    _discovery_cache.clear()
+    _discovery_cache.update(data)
+    return data
+
+
+def oidc_state() -> str:
+    """A signed, short-lived value tying the callback to this browser."""
+    payload = json.dumps({"n": secrets.token_urlsafe(16), "exp": int(time.time()) + 600}).encode()
+    return _sign(payload)
+
+
+def oidc_state_valid(state: str) -> bool:
+    try:
+        body_b64, mac_b64 = state.split(".")
+        payload = base64.urlsafe_b64decode(body_b64)
+        expected = hmac.new(settings.secret_key.encode(), payload, hashlib.sha256).digest()
+        if not hmac.compare_digest(expected, base64.urlsafe_b64decode(mac_b64)):
+            return False
+        return int(json.loads(payload).get("exp", 0)) >= time.time()
+    except (ValueError, TypeError, KeyError, json.JSONDecodeError):
+        return False
+
+
+async def oidc_exchange(code: str, redirect_uri: str) -> dict[str, Any]:
+    meta = await oidc_discovery()
+    async with httpx.AsyncClient(timeout=15) as client:
+        token_resp = await client.post(meta["token_endpoint"], data={
+            "grant_type": "authorization_code",
+            "code": code,
+            "redirect_uri": redirect_uri,
+            "client_id": settings.oidc_client_id,
+            "client_secret": settings.oidc_client_secret,
+        }, headers={"Accept": "application/json"})
+        if token_resp.status_code != 200:
+            raise OIDCError(f"Token exchange failed ({token_resp.status_code}): {token_resp.text[:200]}")
+        access = token_resp.json().get("access_token")
+        if not access:
+            raise OIDCError("Token response contained no access_token.")
+
+        info_resp = await client.get(meta["userinfo_endpoint"],
+                                     headers={"Authorization": f"Bearer {access}"})
+        if info_resp.status_code != 200:
+            raise OIDCError(f"userinfo failed ({info_resp.status_code}): {info_resp.text[:200]}")
+        return info_resp.json()
+
+
+def upsert_oidc_user(db: Session, claims: dict[str, Any]) -> User:
+    """Find or create the local record for an SSO identity.
+
+    Matching is on `sub` first so a changed email still lands on the same user;
+    an existing local account with the same address is adopted rather than
+    duplicated.
+    """
+    sub = str(claims.get("sub") or "").strip()
+    email = str(claims.get("email") or "").strip().lower()
+    if not sub and not email:
+        raise OIDCError("SSO returned neither sub nor email.")
+
+    groups = claims.get(settings.oidc_groups_claim) or []
+    if isinstance(groups, str):
+        groups = [groups]
+    is_admin = settings.oidc_admin_group in {str(g) for g in groups}
+
+    user = None
+    if sub:
+        user = db.query(User).filter(User.oidc_sub == sub).one_or_none()
+    if user is None and email:
+        user = db.query(User).filter(User.email == email).one_or_none()
+
+    if user is None:
+        user = User(email=email or f"{sub}@sso.local", provider=AuthProvider.OIDC)
+        db.add(user)
+
+    user.oidc_sub = sub or user.oidc_sub
+    user.provider = AuthProvider.OIDC
+    user.name = str(claims.get("name") or claims.get("preferred_username") or user.name or email)
+    if email:
+        user.email = email
+    user.is_admin = is_admin
+    user.is_active = True
+    user.last_login = dt.datetime.now(dt.timezone.utc)
+    if not user.signoff_name:
+        user.signoff_name = user.name
+    db.commit()
+    return user

backend/app/config.py

@@ -0,0 +1,101 @@
+"""Environment-driven configuration.
+
+Everything deployment-specific comes from the environment so the same image runs
+locally under compose and in Kubernetes with only a ConfigMap/Secret difference.
+"""
+from __future__ import annotations
+
+import os
+from functools import lru_cache
+
+
+def _bool(name: str, default: bool = False) -> bool:
+    return str(os.environ.get(name, str(default))).strip().lower() in {"1", "true", "yes", "on"}
+
+
+def _int(name: str, default: int) -> int:
+    try:
+        return int(os.environ.get(name, default))
+    except (TypeError, ValueError):
+        return default
+
+
+class Settings:
+    # --- app ---------------------------------------------------------------
+    app_name = os.environ.get("CX_APP_NAME", "CX Triage")
+    base_url = os.environ.get("CX_BASE_URL", "http://localhost:8080")
+    secret_key = os.environ.get("CX_SECRET_KEY", "dev-only-change-me")
+    session_hours = _int("CX_SESSION_HOURS", 12)
+    static_dir = os.environ.get("CX_STATIC_DIR", "/app/static")
+
+    # --- database ----------------------------------------------------------
+    # sqlite for local/compose, postgres in the cluster.
+    database_url = os.environ.get("CX_DATABASE_URL", "sqlite:////data/cx-triage.db")
+
+    # --- data sources ------------------------------------------------------
+    prometheus_base = os.environ.get("CX_PROMETHEUS_BASE", "http://10.11.254.250:9090")
+    prometheus_relay = os.environ.get("CX_PROMETHEUS_RELAY", "")
+    cx_tools_path = os.environ.get("CX_TOOLS_PATH", "")
+
+    # --- auth --------------------------------------------------------------
+    # Local accounts are for development and for a cluster without SSO yet.
+    # When CX_OIDC_ENABLED is on, Authentik becomes the source of truth.
+    auth_local_enabled = _bool("CX_AUTH_LOCAL_ENABLED", True)
+    bootstrap_admin_email = os.environ.get("CX_BOOTSTRAP_ADMIN_EMAIL", "admin@localhost")
+    bootstrap_admin_password = os.environ.get("CX_BOOTSTRAP_ADMIN_PASSWORD", "")
+
+    oidc_enabled = _bool("CX_OIDC_ENABLED", False)
+    oidc_issuer = os.environ.get("CX_OIDC_ISSUER", "")            # e.g. https://sso/application/o/cx-triage/
+    oidc_client_id = os.environ.get("CX_OIDC_CLIENT_ID", "")
+    oidc_client_secret = os.environ.get("CX_OIDC_CLIENT_SECRET", "")
+    oidc_scopes = os.environ.get("CX_OIDC_SCOPES", "openid email profile")
+    oidc_admin_group = os.environ.get("CX_OIDC_ADMIN_GROUP", "cx-triage-admins")
+    oidc_groups_claim = os.environ.get("CX_OIDC_GROUPS_CLAIM", "groups")
+
+    # --- feature flags -----------------------------------------------------
+    # Sending must be switched on deliberately; a demo instance cannot email.
+    feature_send_enabled = _bool("CX_FEATURE_SEND_ENABLED", False)
+    feature_zendesk = _bool("CX_FEATURE_ZENDESK", False)
+    feature_jira = _bool("CX_FEATURE_JIRA", False)
+    feature_linkage_scan = _bool("CX_FEATURE_LINKAGE_SCAN", True)
+    send_daily_cap = _int("CX_SEND_DAILY_CAP", 25)
+
+    # --- integrations ------------------------------------------------------
+    zendesk_subdomain = os.environ.get("CX_ZENDESK_SUBDOMAIN", "")
+    zendesk_email = os.environ.get("CX_ZENDESK_EMAIL", "")
+    zendesk_token = os.environ.get("CX_ZENDESK_TOKEN", "")
+    zendesk_default_public = _bool("CX_ZENDESK_PUBLIC_REPLY", True)
+
+    jira_base = os.environ.get("CX_JIRA_BASE", "")
+    jira_email = os.environ.get("CX_JIRA_EMAIL", "")
+    jira_token = os.environ.get("CX_JIRA_TOKEN", "")
+    jira_project = os.environ.get("CX_JIRA_PROJECT", "INFRA")
+    jira_issue_type = os.environ.get("CX_JIRA_ISSUE_TYPE", "Task")
+
+    @property
+    def zendesk_ready(self) -> bool:
+        return bool(self.feature_zendesk and self.zendesk_subdomain
+                    and self.zendesk_email and self.zendesk_token)
+
+    @property
+    def jira_ready(self) -> bool:
+        return bool(self.feature_jira and self.jira_base and self.jira_email
+                    and self.jira_token and self.jira_project)
+
+    def public_flags(self) -> dict:
+        """What the frontend is allowed to know - never secrets."""
+        return {
+            "app_name": self.app_name,
+            "oidc_enabled": self.oidc_enabled,
+            "local_login": self.auth_local_enabled,
+            "zendesk_ready": self.zendesk_ready,
+            "jira_ready": self.jira_ready,
+            "send_enabled": self.feature_send_enabled,
+            "linkage_scan": self.feature_linkage_scan,
+            "jira_project": self.jira_project if self.jira_ready else "",
+        }
+
+
+@lru_cache
+def get_settings() -> Settings:
+    return Settings()

backend/app/db.py

@@ -0,0 +1,29 @@
+"""Database engine and session handling."""
+from __future__ import annotations
+
+from collections.abc import Iterator
+
+from sqlalchemy import create_engine
+from sqlalchemy.orm import Session, sessionmaker
+
+from .config import get_settings
+
+settings = get_settings()
+
+_connect_args = {"check_same_thread": False} if settings.database_url.startswith("sqlite") else {}
+engine = create_engine(settings.database_url, pool_pre_ping=True, connect_args=_connect_args)
+SessionLocal = sessionmaker(bind=engine, autoflush=False, expire_on_commit=False)
+
+
+def get_db() -> Iterator[Session]:
+    db = SessionLocal()
+    try:
+        yield db
+    finally:
+        db.close()
+
+
+def init_db() -> None:
+    from . import models  # noqa: F401  (registers the tables)
+
+    models.Base.metadata.create_all(engine)

backend/app/delivery.py

@@ -0,0 +1,174 @@
+"""Outbound delivery to Zendesk and Jira.
+
+Three independent gates have to be open before anything leaves this process:
+
+  1. the integration is configured (subdomain/email/token present)
+  2. its feature flag is on            - CX_FEATURE_ZENDESK / CX_FEATURE_JIRA
+  3. sending is globally enabled       - CX_FEATURE_SEND_ENABLED
+
+A demo or staging instance simply leaves the third off, and then no combination
+of clicks can email a customer. Every send is recorded as a case event before it
+is attempted, so an audit trail exists even when the call fails.
+"""
+from __future__ import annotations
+
+import datetime as dt
+from typing import Any, Optional
+
+import httpx
+from sqlalchemy.orm import Session
+
+from .config import get_settings
+from .models import Case, CaseEvent, CaseStatus, User
+from .services import add_event, set_status
+
+settings = get_settings()
+
+
+class DeliveryError(RuntimeError):
+    pass
+
+
+def _guard(kind: str) -> None:
+    if not settings.feature_send_enabled:
+        raise DeliveryError(
+            "Sending is disabled on this instance (CX_FEATURE_SEND_ENABLED is off). "
+            "The payload is ready but nothing will leave the server."
+        )
+    if kind == "zendesk" and not settings.zendesk_ready:
+        raise DeliveryError("Zendesk is not configured. Set CX_FEATURE_ZENDESK plus "
+                            "CX_ZENDESK_SUBDOMAIN, CX_ZENDESK_EMAIL and CX_ZENDESK_TOKEN.")
+    if kind == "jira" and not settings.jira_ready:
+        raise DeliveryError("Jira is not configured. Set CX_FEATURE_JIRA plus "
+                            "CX_JIRA_BASE, CX_JIRA_EMAIL, CX_JIRA_TOKEN and CX_JIRA_PROJECT.")
+
+
+def sends_today(db: Session) -> int:
+    since = dt.datetime.now(dt.timezone.utc) - dt.timedelta(days=1)
+    return (db.query(CaseEvent)
+            .filter(CaseEvent.action.in_(["zendesk_sent", "jira_created"]))
+            .filter(CaseEvent.created_at >= since).count())
+
+
+def _check_cap(db: Session) -> None:
+    used = sends_today(db)
+    if used >= settings.send_daily_cap:
+        raise DeliveryError(
+            f"Daily send cap reached ({used}/{settings.send_daily_cap}). Raise CX_SEND_DAILY_CAP "
+            "if this is deliberate - the cap exists so a loop cannot mail every customer."
+        )
+
+
+# --- Zendesk ----------------------------------------------------------------
+
+async def send_zendesk(db: Session, case: Case, actor: User, *, to: str, subject: str,
+                       body: str, priority: str = "normal",
+                       tags: Optional[list[str]] = None, public: Optional[bool] = None) -> dict[str, Any]:
+    _guard("zendesk")
+    _check_cap(db)
+    if not to.strip():
+        raise DeliveryError("No recipient address.")
+
+    base = f"https://{settings.zendesk_subdomain}.zendesk.com/api/v2"
+    auth = (f"{settings.zendesk_email}/token", settings.zendesk_token)
+    external_id = f"cx-triage-{case.fingerprint}"
+
+    async with httpx.AsyncClient(timeout=30) as client:
+        # Search first so a re-diagnosed alert comments on the existing ticket
+        # instead of opening a second one for the same customer.
+        found = await client.get(f"{base}/search.json",
+                                 params={"query": f'type:ticket external_id:"{external_id}"'}, auth=auth)
+        existing = None
+        if found.status_code == 200:
+            results = found.json().get("results") or []
+            existing = results[0] if results else None
+
+        comment = {"body": body, "public": settings.zendesk_default_public if public is None else public}
+        if existing:
+            resp = await client.put(f"{base}/tickets/{existing['id']}.json",
+                                    json={"ticket": {"comment": comment}}, auth=auth)
+            action = "updated"
+        else:
+            payload = {"ticket": {
+                "subject": subject,
+                "comment": comment,
+                "requester": {"name": to.split("@")[0], "email": to},
+                "priority": priority,
+                "type": "incident",
+                "tags": tags or ["cx-triage", f"alert-{case.kind}"],
+                "external_id": external_id,
+            }}
+            resp = await client.post(f"{base}/tickets.json", json=payload, auth=auth)
+            action = "created"
+
+    if resp.status_code not in (200, 201):
+        add_event(db, case, actor, "zendesk_failed", f"HTTP {resp.status_code}: {resp.text[:300]}")
+        db.commit()
+        raise DeliveryError(f"Zendesk returned {resp.status_code}: {resp.text[:300]}")
+
+    ticket = resp.json().get("ticket") or {}
+    ticket_id = str(ticket.get("id") or (existing or {}).get("id") or "")
+    url = f"https://{settings.zendesk_subdomain}.zendesk.com/agent/tickets/{ticket_id}"
+
+    case.zendesk_ticket_id = ticket_id
+    case.zendesk_ticket_url = url
+    add_event(db, case, actor, "zendesk_sent",
+              f"Ticket {ticket_id} {action} for {to}",
+              {"ticket_id": ticket_id, "to": to, "subject": subject, "action": action})
+    if case.status in (CaseStatus.NEW, CaseStatus.INVESTIGATING):
+        set_status(db, case, CaseStatus.CUSTOMER_CONTACTED, actor, f"Zendesk ticket {ticket_id}")
+    db.commit()
+    return {"ok": True, "ticket_id": ticket_id, "url": url, "action": action}
+
+
+# --- Jira -------------------------------------------------------------------
+
+async def create_jira(db: Session, case: Case, actor: User, *, summary: str, description: str,
+                      project: str = "", issue_type: str = "",
+                      labels: Optional[list[str]] = None) -> dict[str, Any]:
+    _guard("jira")
+    _check_cap(db)
+
+    base = settings.jira_base.rstrip("/")
+    auth = (settings.jira_email, settings.jira_token)
+    label = f"cx-triage-{case.fingerprint}"
+    all_labels = sorted(set((labels or []) + ["cx-triage", label]))
+
+    async with httpx.AsyncClient(timeout=30) as client:
+        found = await client.get(f"{base}/rest/api/3/search",
+                                 params={"jql": f'labels = "{label}"', "maxResults": 1}, auth=auth)
+        if found.status_code == 200 and (found.json().get("issues") or []):
+            issue = found.json()["issues"][0]
+            key = issue["key"]
+            url = f"{base}/browse/{key}"
+            case.jira_issue_key, case.jira_issue_url = key, url
+            add_event(db, case, actor, "jira_exists", f"Issue {key} already exists for this case")
+            db.commit()
+            return {"ok": True, "key": key, "url": url, "action": "existing"}
+
+        payload = {"fields": {
+            "project": {"key": project or settings.jira_project},
+            "summary": summary[:250],
+            "issuetype": {"name": issue_type or settings.jira_issue_type},
+            "labels": all_labels,
+            "description": {
+                "type": "doc", "version": 1,
+                "content": [{"type": "paragraph",
+                             "content": [{"type": "text", "text": description[:30000]}]}],
+            },
+        }}
+        resp = await client.post(f"{base}/rest/api/3/issue", json=payload, auth=auth)
+
+    if resp.status_code not in (200, 201):
+        add_event(db, case, actor, "jira_failed", f"HTTP {resp.status_code}: {resp.text[:300]}")
+        db.commit()
+        raise DeliveryError(f"Jira returned {resp.status_code}: {resp.text[:300]}")
+
+    key = resp.json().get("key", "")
+    url = f"{base}/browse/{key}"
+    case.jira_issue_key, case.jira_issue_url = key, url
+    add_event(db, case, actor, "jira_created", f"Issue {key} created", {"key": key, "summary": summary})
+    if case.status in (CaseStatus.NEW, CaseStatus.INVESTIGATING):
+        set_status(db, case, CaseStatus.ESCALATED_INFRA, actor, f"Jira {key}")
+    db.commit()
+    return {"ok": True, "key": key, "url": url, "action": "created"}

backend/app/main.py

@@ -0,0 +1,66 @@
+"""FastAPI application entry point."""
+from __future__ import annotations
+
+import os
+from contextlib import asynccontextmanager
+
+from fastapi import FastAPI
+from fastapi.responses import FileResponse, JSONResponse
+from fastapi.staticfiles import StaticFiles
+
+from . import VERSION
+from .auth import ensure_bootstrap_admin
+from .config import get_settings
+from .db import SessionLocal, init_db
+from .routers import (actions_router, alerts_router, auth_router, cases_router,
+                      linkage_router, settings_router)
+
+settings = get_settings()
+
+
+@asynccontextmanager
+async def lifespan(app: FastAPI):
+    init_db()
+    db = SessionLocal()
+    try:
+        message = ensure_bootstrap_admin(db)
+        if message:
+            print(f"[auth] {message}")
+    finally:
+        db.close()
+    print(f"[startup] {settings.app_name} {VERSION}")
+    print(f"[startup] prometheus: {settings.prometheus_base}")
+    print(f"[startup] sso: {'on' if settings.oidc_enabled else 'off'} | "
+          f"zendesk: {'ready' if settings.zendesk_ready else 'off'} | "
+          f"jira: {'ready' if settings.jira_ready else 'off'} | "
+          f"sending: {'ENABLED' if settings.feature_send_enabled else 'disabled'}")
+    alerts_router.engine.warm()
+    yield
+
+
+app = FastAPI(title=settings.app_name, version=VERSION, lifespan=lifespan)
+
+for module in (auth_router, alerts_router, cases_router, settings_router,
+               linkage_router, actions_router):
+    app.include_router(module.router)
+
+
+@app.get("/api/health")
+def health():
+    return {"status": "ok", "version": VERSION, "config": settings.public_flags()}
+
+
+# The built React bundle is copied into the image; in development Vite serves it
+# instead and proxies /api here, so a missing directory is not an error.
+if os.path.isdir(settings.static_dir):
+    app.mount("/assets", StaticFiles(directory=os.path.join(settings.static_dir, "assets")),
+              name="assets")
+
+    @app.get("/{full_path:path}")
+    def spa(full_path: str):
+        if full_path.startswith("api/"):
+            return JSONResponse({"detail": "Not found"}, status_code=404)
+        index = os.path.join(settings.static_dir, "index.html")
+        if os.path.isfile(index):
+            return FileResponse(index)
+        return JSONResponse({"detail": "Frontend not built"}, status_code=404)

backend/app/models.py

@@ -0,0 +1,208 @@
+"""Persistent state: who is working what, and what has been done about it.
+
+The alert queue itself stays stateless - it is recomputed from Prometheus every
+minute. What is worth persisting is the human layer on top: which alerts someone
+has picked up, what was done, and the audit trail behind it. Cases are keyed by
+the alert fingerprint so an alert that stops and re-fires lands back on the same
+case rather than losing its history.
+"""
+from __future__ import annotations
+
+import datetime as dt
+import enum
+from typing import Any
+
+from sqlalchemy import (JSON, Boolean, DateTime, Enum, ForeignKey, Index, Integer,
+                        String, Text, UniqueConstraint)
+from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, relationship
+
+
+class Base(DeclarativeBase):
+    pass
+
+
+def _now() -> dt.datetime:
+    return dt.datetime.now(dt.timezone.utc)
+
+
+class CaseStatus(str, enum.Enum):
+    """Where a piece of work has got to."""
+
+    NEW = "new"                      # seen, nobody has touched it
+    INVESTIGATING = "investigating"  # someone has picked it up
+    CUSTOMER_CONTACTED = "customer_contacted"
+    ESCALATED_INFRA = "escalated_infra"
+    WAITING_CUSTOMER = "waiting_customer"
+    WAITING_INFRA = "waiting_infra"
+    REMEDIATED = "remediated"        # fixed, waiting for the alert to clear
+    RESOLVED = "resolved"
+    WONT_FIX = "wont_fix"            # deliberate no-action
+    FALSE_POSITIVE = "false_positive"  # the alert itself was wrong
+
+
+OPEN_STATUSES = {
+    CaseStatus.NEW, CaseStatus.INVESTIGATING, CaseStatus.CUSTOMER_CONTACTED,
+    CaseStatus.ESCALATED_INFRA, CaseStatus.WAITING_CUSTOMER, CaseStatus.WAITING_INFRA,
+    CaseStatus.REMEDIATED,
+}
+
+
+class AuthProvider(str, enum.Enum):
+    LOCAL = "local"
+    OIDC = "oidc"
+
+
+class User(Base):
+    __tablename__ = "users"
+
+    id: Mapped[int] = mapped_column(primary_key=True)
+    email: Mapped[str] = mapped_column(String(320), unique=True, index=True)
+    name: Mapped[str] = mapped_column(String(200), default="")
+    is_admin: Mapped[bool] = mapped_column(Boolean, default=False)
+    is_active: Mapped[bool] = mapped_column(Boolean, default=True)
+    provider: Mapped[AuthProvider] = mapped_column(Enum(AuthProvider), default=AuthProvider.LOCAL)
+    # Local accounts only; OIDC users never have one.
+    password_hash: Mapped[str] = mapped_column(String(255), default="")
+    # Stable Authentik subject, so a rename or email change keeps the same user.
+    oidc_sub: Mapped[str] = mapped_column(String(255), default="", index=True)
+    signoff_name: Mapped[str] = mapped_column(String(200), default="")
+    created_at: Mapped[dt.datetime] = mapped_column(DateTime(timezone=True), default=_now)
+    last_login: Mapped[dt.datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
+
+    def to_json(self) -> dict[str, Any]:
+        return {
+            "id": self.id, "email": self.email, "name": self.name or self.email,
+            "is_admin": self.is_admin, "provider": self.provider.value,
+            "signoff_name": self.signoff_name or self.name,
+        }
+
+
+class Case(Base):
+    """One tracked alert, keyed by its fingerprint."""
+
+    __tablename__ = "cases"
+    __table_args__ = (Index("ix_cases_status_seen", "status", "last_seen_at"),)
+
+    id: Mapped[int] = mapped_column(primary_key=True)
+    fingerprint: Mapped[str] = mapped_column(String(64), unique=True, index=True)
+
+    kind: Mapped[str] = mapped_column(String(40), index=True)
+    title: Mapped[str] = mapped_column(String(300), default="")
+    subject: Mapped[str] = mapped_column(String(300), default="")  # VM name, host or IP
+    openstack_id: Mapped[str] = mapped_column(String(64), default="", index=True)
+    instance_name: Mapped[str] = mapped_column(String(200), default="", index=True)
+    host: Mapped[str] = mapped_column(String(120), default="", index=True)
+    region: Mapped[str] = mapped_column(String(16), default="")
+    org_id: Mapped[str] = mapped_column(String(32), default="", index=True)
+    org_name: Mapped[str] = mapped_column(String(300), default="")
+
+    status: Mapped[CaseStatus] = mapped_column(Enum(CaseStatus), default=CaseStatus.NEW, index=True)
+    assignee_id: Mapped[int | None] = mapped_column(ForeignKey("users.id"), nullable=True)
+    assignee: Mapped[User | None] = relationship(lazy="joined")
+
+    # Outbound references, so a case shows what already exists elsewhere.
+    zendesk_ticket_id: Mapped[str] = mapped_column(String(40), default="")
+    zendesk_ticket_url: Mapped[str] = mapped_column(String(500), default="")
+    jira_issue_key: Mapped[str] = mapped_column(String(40), default="")
+    jira_issue_url: Mapped[str] = mapped_column(String(500), default="")
+
+    notes: Mapped[str] = mapped_column(Text, default="")
+    snooze_until: Mapped[dt.datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
+
+    first_seen_at: Mapped[dt.datetime] = mapped_column(DateTime(timezone=True), default=_now)
+    last_seen_at: Mapped[dt.datetime] = mapped_column(DateTime(timezone=True), default=_now)
+    closed_at: Mapped[dt.datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
+    # How many separate times this alert has come back after being closed.
+    reopen_count: Mapped[int] = mapped_column(Integer, default=0)
+
+    events: Mapped[list["CaseEvent"]] = relationship(
+        back_populates="case", cascade="all, delete-orphan", order_by="CaseEvent.created_at.desc()")
+
+    @property
+    def is_open(self) -> bool:
+        return self.status in OPEN_STATUSES
+
+    def to_json(self, with_events: bool = False) -> dict[str, Any]:
+        data = {
+            "id": self.id, "fingerprint": self.fingerprint, "kind": self.kind,
+            "title": self.title, "subject": self.subject,
+            "openstack_id": self.openstack_id, "instance_name": self.instance_name,
+            "host": self.host, "region": self.region,
+            "org_id": self.org_id, "org_name": self.org_name,
+            "status": self.status.value, "is_open": self.is_open,
+            "assignee": self.assignee.to_json() if self.assignee else None,
+            "zendesk_ticket_id": self.zendesk_ticket_id,
+            "zendesk_ticket_url": self.zendesk_ticket_url,
+            "jira_issue_key": self.jira_issue_key,
+            "jira_issue_url": self.jira_issue_url,
+            "notes": self.notes,
+            "snooze_until": self.snooze_until.isoformat() if self.snooze_until else None,
+            "first_seen_at": self.first_seen_at.isoformat() if self.first_seen_at else None,
+            "last_seen_at": self.last_seen_at.isoformat() if self.last_seen_at else None,
+            "closed_at": self.closed_at.isoformat() if self.closed_at else None,
+            "reopen_count": self.reopen_count,
+        }
+        if with_events:
+            data["events"] = [e.to_json() for e in self.events]
+        return data
+
+
+class CaseEvent(Base):
+    """Append-only history. Nothing here is ever edited or deleted."""
+
+    __tablename__ = "case_events"
+
+    id: Mapped[int] = mapped_column(primary_key=True)
+    case_id: Mapped[int] = mapped_column(ForeignKey("cases.id"), index=True)
+    case: Mapped[Case] = relationship(back_populates="events")
+
+    actor_id: Mapped[int | None] = mapped_column(ForeignKey("users.id"), nullable=True)
+    actor: Mapped[User | None] = relationship(lazy="joined")
+    actor_label: Mapped[str] = mapped_column(String(200), default="")  # survives user deletion
+
+    action: Mapped[str] = mapped_column(String(60))   # status_changed | zendesk_sent | note | ...
+    detail: Mapped[str] = mapped_column(Text, default="")
+    payload: Mapped[dict[str, Any] | None] = mapped_column(JSON, nullable=True)
+    created_at: Mapped[dt.datetime] = mapped_column(DateTime(timezone=True), default=_now, index=True)
+
+    def to_json(self) -> dict[str, Any]:
+        return {
+            "id": self.id, "action": self.action, "detail": self.detail,
+            "actor": self.actor_label or (self.actor.email if self.actor else "system"),
+            "created_at": self.created_at.isoformat() if self.created_at else None,
+            "payload": self.payload,
+        }
+
+
+class SuppressionRule(Base):
+    """Alerts the team has decided not to see, with the reason recorded."""
+
+    __tablename__ = "suppression_rules"
+    __table_args__ = (UniqueConstraint("name", name="uq_rule_name"),)
+
+    id: Mapped[int] = mapped_column(primary_key=True)
+    name: Mapped[str] = mapped_column(String(200))
+    reason: Mapped[str] = mapped_column(Text, default="")
+    enabled: Mapped[bool] = mapped_column(Boolean, default=True)
+    # {"kind": ["error"], "organization": ["modal"]} - all keys must match.
+    conditions: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict)
+    created_by: Mapped[str] = mapped_column(String(200), default="")
+    created_at: Mapped[dt.datetime] = mapped_column(DateTime(timezone=True), default=_now)
+
+    def to_json(self) -> dict[str, Any]:
+        return {
+            "id": str(self.id), "name": self.name, "reason": self.reason,
+            "enabled": self.enabled, "conditions": self.conditions or {},
+            "created_by": self.created_by,
+            "created": self.created_at.strftime("%Y-%m-%d") if self.created_at else "",
+        }
+
+
+class AppSetting(Base):
+    """Small key/value bag for things an admin can change at runtime."""
+
+    __tablename__ = "app_settings"
+
+    key: Mapped[str] = mapped_column(String(80), primary_key=True)
+    value: Mapped[dict[str, Any] | None] = mapped_column(JSON, nullable=True)
+    updated_at: Mapped[dt.datetime] = mapped_column(DateTime(timezone=True), default=_now, onupdate=_now)

backend/app/routers/actions_router.py

@@ -0,0 +1,77 @@
+"""Outbound actions: contact the customer, escalate to Infrastructure."""
+from __future__ import annotations
+
+from fastapi import APIRouter, Depends, HTTPException, status
+from pydantic import BaseModel
+from sqlalchemy.orm import Session
+
+from ..auth import current_user
+from ..config import get_settings
+from ..db import get_db
+from ..delivery import DeliveryError, create_jira, send_zendesk, sends_today
+from ..models import Case, User
+
+router = APIRouter(prefix="/api/actions", tags=["actions"])
+settings = get_settings()
+
+
+class ZendeskBody(BaseModel):
+    fingerprint: str
+    to: str
+    subject: str
+    body: str
+    priority: str = "normal"
+    tags: list[str] = []
+    public: bool | None = None
+
+
+class JiraBody(BaseModel):
+    fingerprint: str
+    summary: str
+    description: str
+    project: str = ""
+    issue_type: str = ""
+    labels: list[str] = []
+
+
+def _case(db: Session, fingerprint: str) -> Case:
+    found = db.query(Case).filter(Case.fingerprint == fingerprint).one_or_none()
+    if not found:
+        raise HTTPException(status.HTTP_404_NOT_FOUND,
+                            "Open the case first - nothing is tracked for that alert yet.")
+    return found
+
+
+@router.get("/status")
+def action_status(db: Session = Depends(get_db), user: User = Depends(current_user)):
+    return {
+        "zendesk_ready": settings.zendesk_ready,
+        "jira_ready": settings.jira_ready,
+        "send_enabled": settings.feature_send_enabled,
+        "sends_today": sends_today(db),
+        "daily_cap": settings.send_daily_cap,
+    }
+
+
+@router.post("/zendesk")
+async def zendesk(body: ZendeskBody, db: Session = Depends(get_db),
+                  user: User = Depends(current_user)):
+    case = _case(db, body.fingerprint)
+    try:
+        return await send_zendesk(db, case, user, to=body.to, subject=body.subject,
+                                  body=body.body, priority=body.priority,
+                                  tags=body.tags or None, public=body.public)
+    except DeliveryError as exc:
+        raise HTTPException(status.HTTP_400_BAD_REQUEST, str(exc)) from exc
+
+
+@router.post("/jira")
+async def jira(body: JiraBody, db: Session = Depends(get_db),
+               user: User = Depends(current_user)):
+    case = _case(db, body.fingerprint)
+    try:
+        return await create_jira(db, case, user, summary=body.summary,
+                                 description=body.description, project=body.project,
+                                 issue_type=body.issue_type, labels=body.labels or None)
+    except DeliveryError as exc:
+        raise HTTPException(status.HTTP_400_BAD_REQUEST, str(exc)) from exc

backend/app/routers/alerts_router.py

@@ -0,0 +1,88 @@
+"""The alert queue and per-alert diagnosis."""
+from __future__ import annotations
+
+import threading
+import time
+import traceback
+import uuid
+from concurrent.futures import ThreadPoolExecutor
+from typing import Any
+
+from fastapi import APIRouter, Depends, HTTPException, status
+from pydantic import BaseModel
+from sqlalchemy.orm import Session
+
+from triagelib import runbooks
+
+from ..auth import current_user
+from ..db import SessionLocal, get_db
+from ..models import User
+from ..services import Engine, RuleAdapter, get_or_create_case
+
+router = APIRouter(prefix="/api", tags=["alerts"])
+engine = Engine()
+
+_jobs: dict[str, dict[str, Any]] = {}
+_jobs_lock = threading.Lock()
+_pool = ThreadPoolExecutor(max_workers=3, thread_name_prefix="triage")
+JOB_TTL = 30 * 60
+
+
+class TriageBody(BaseModel):
+    fingerprint: str
+    force: bool = False
+
+
+@router.get("/alerts")
+def alert_queue(force: bool = False, db: Session = Depends(get_db),
+                user: User = Depends(current_user)):
+    return engine.queue(db, force=force)
+
+
+@router.post("/triage")
+def start_triage(body: TriageBody, db: Session = Depends(get_db),
+                 user: User = Depends(current_user)):
+    alert = engine.find_alert(db, body.fingerprint)
+    if alert is None:
+        raise HTTPException(status.HTTP_404_NOT_FOUND,
+                            "That alert is no longer firing. Refresh the queue.")
+    case = get_or_create_case(db, alert, user)
+    job_id = uuid.uuid4().hex[:12]
+    with _jobs_lock:
+        _reap()
+        _jobs[job_id] = {"id": job_id, "state": "running", "created": time.time(),
+                         "result": None, "error": ""}
+    _pool.submit(_run, job_id, alert, body.force)
+    return {"job_id": job_id, "alert": alert.to_json(), "case": case.to_json(with_events=True)}
+
+
+@router.get("/jobs/{job_id}")
+def job(job_id: str, user: User = Depends(current_user)):
+    with _jobs_lock:
+        found = _jobs.get(job_id)
+    if not found:
+        raise HTTPException(status.HTTP_404_NOT_FOUND, "Unknown job")
+    return found
+
+
+def _run(job_id: str, alert: Any, force: bool) -> None:
+    started = time.monotonic()
+    db = SessionLocal()
+    try:
+        diagnosis = runbooks.diagnose(alert, engine.prom, engine.snapshot.get(), force,
+                                      RuleAdapter(db))
+        payload = diagnosis.to_json()
+        payload["elapsed_seconds"] = round(time.monotonic() - started, 1)
+        with _jobs_lock:
+            _jobs[job_id].update({"state": "done", "result": payload})
+    except Exception:
+        with _jobs_lock:
+            _jobs[job_id].update({"state": "error", "error": traceback.format_exc(limit=4)})
+    finally:
+        db.close()
+
+
+def _reap() -> None:
+    cutoff = time.time() - JOB_TTL
+    for key in [k for k, v in _jobs.items() if v["created"] < cutoff]:
+        _jobs.pop(key, None)

backend/app/routers/auth_router.py

@@ -0,0 +1,99 @@
+"""Sign-in: local accounts and the Authentik OIDC round trip."""
+from __future__ import annotations
+
+import datetime as dt
+
+from fastapi import APIRouter, Depends, HTTPException, Request, Response, status
+from fastapi.responses import RedirectResponse
+from pydantic import BaseModel
+from sqlalchemy.orm import Session
+
+from .. import auth as auth_mod
+from ..config import get_settings
+from ..db import get_db
+from ..models import User
+
+router = APIRouter(prefix="/api/auth", tags=["auth"])
+settings = get_settings()
+
+
+class LoginBody(BaseModel):
+    email: str
+    password: str
+
+
+def _set_cookie(response: Response, user: User) -> None:
+    response.set_cookie(
+        auth_mod.SESSION_COOKIE, auth_mod.issue_session(user.id),
+        max_age=settings.session_hours * 3600, httponly=True, samesite="lax",
+        secure=settings.base_url.startswith("https://"), path="/",
+    )
+
+
+@router.get("/me")
+def me(user: User | None = Depends(auth_mod.optional_user)):
+    return {"user": user.to_json() if user else None, "config": settings.public_flags()}
+
+
+@router.post("/login")
+def login(body: LoginBody, response: Response, db: Session = Depends(get_db)):
+    if not settings.auth_local_enabled:
+        raise HTTPException(status.HTTP_403_FORBIDDEN, "Local login is disabled; use SSO.")
+    user = auth_mod.authenticate_local(db, body.email, body.password)
+    if not user:
+        raise HTTPException(status.HTTP_401_UNAUTHORIZED, "Incorrect email or password")
+    user.last_login = dt.datetime.now(dt.timezone.utc)
+    db.commit()
+    _set_cookie(response, user)
+    return {"user": user.to_json()}
+
+
+@router.post("/logout")
+def logout(response: Response):
+    response.delete_cookie(auth_mod.SESSION_COOKIE, path="/")
+    return {"ok": True}
+
+
+@router.get("/oidc/start")
+async def oidc_start(request: Request):
+    if not settings.oidc_enabled:
+        raise HTTPException(status.HTTP_404_NOT_FOUND, "SSO is not enabled")
+    try:
+        meta = await auth_mod.oidc_discovery()
+    except auth_mod.OIDCError as exc:
+        raise HTTPException(status.HTTP_502_BAD_GATEWAY, str(exc)) from exc
+
+    redirect_uri = f"{settings.base_url.rstrip('/')}/api/auth/oidc/callback"
+    state = auth_mod.oidc_state()
+    url = (f"{meta['authorization_endpoint']}?response_type=code"
+           f"&client_id={settings.oidc_client_id}"
+           f"&redirect_uri={redirect_uri}"
+           f"&scope={settings.oidc_scopes.replace(' ', '%20')}"
+           f"&state={state}")
+    response = RedirectResponse(url, status_code=302)
+    response.set_cookie("cx_oidc_state", state, max_age=600, httponly=True, samesite="lax", path="/")
+    return response
+
+
+@router.get("/oidc/callback")
+async def oidc_callback(request: Request, code: str = "", state: str = "",
+                        db: Session = Depends(get_db)):
+    if not settings.oidc_enabled:
+        raise HTTPException(status.HTTP_404_NOT_FOUND, "SSO is not enabled")
+    # The state must match the cookie we set *and* still verify - one guards
+    # against a swapped browser, the other against a forged value.
+    if not code or not state or state != request.cookies.get("cx_oidc_state") \
+            or not auth_mod.oidc_state_valid(state):
+        raise HTTPException(status.HTTP_400_BAD_REQUEST, "Invalid or expired SSO state")
+
+    redirect_uri = f"{settings.base_url.rstrip('/')}/api/auth/oidc/callback"
+    try:
+        claims = await auth_mod.oidc_exchange(code, redirect_uri)
+        user = auth_mod.upsert_oidc_user(db, claims)
+    except auth_mod.OIDCError as exc:
+        raise HTTPException(status.HTTP_502_BAD_GATEWAY, str(exc)) from exc
+
+    response = RedirectResponse("/", status_code=302)
+    _set_cookie(response, user)
+    response.delete_cookie("cx_oidc_state", path="/")
+    return response

backend/app/routers/cases_router.py

@@ -0,0 +1,99 @@
+"""Case state: the human layer over the alert queue."""
+from __future__ import annotations
+
+import datetime as dt
+
+from fastapi import APIRouter, Depends, HTTPException, status
+from pydantic import BaseModel
+from sqlalchemy.orm import Session
+
+from ..auth import current_user
+from ..db import get_db
+from ..models import Case, CaseStatus, User
+from ..services import add_event, set_status
+
+router = APIRouter(prefix="/api/cases", tags=["cases"])
+
+
+class StatusBody(BaseModel):
+    status: str
+    note: str = ""
+
+
+class NoteBody(BaseModel):
+    note: str
+
+
+class SnoozeBody(BaseModel):
+    hours: int = 24
+    note: str = ""
+
+
+def _case(db: Session, fingerprint: str) -> Case:
+    found = db.query(Case).filter(Case.fingerprint == fingerprint).one_or_none()
+    if not found:
+        raise HTTPException(status.HTTP_404_NOT_FOUND, "No case for that alert yet")
+    return found
+
+
+@router.get("")
+def list_cases(open_only: bool = True, limit: int = 200, db: Session = Depends(get_db),
+               user: User = Depends(current_user)):
+    query = db.query(Case).order_by(Case.last_seen_at.desc())
+    rows = [c for c in query.limit(max(1, min(limit, 1000))).all()
+            if (c.is_open or not open_only)]
+    return {"cases": [c.to_json() for c in rows], "statuses": [s.value for s in CaseStatus]}
+
+
+@router.get("/{fingerprint}")
+def get_case(fingerprint: str, db: Session = Depends(get_db), user: User = Depends(current_user)):
+    return _case(db, fingerprint).to_json(with_events=True)
+
+
+@router.post("/{fingerprint}/status")
+def change_status(fingerprint: str, body: StatusBody, db: Session = Depends(get_db),
+                  user: User = Depends(current_user)):
+    try:
+        new_status = CaseStatus(body.status)
+    except ValueError as exc:
+        raise HTTPException(status.HTTP_400_BAD_REQUEST,
+                            f"Unknown status '{body.status}'") from exc
+    case = set_status(db, _case(db, fingerprint), new_status, user, body.note)
+    return case.to_json(with_events=True)
+
+
+@router.post("/{fingerprint}/assign")
+def assign(fingerprint: str, db: Session = Depends(get_db), user: User = Depends(current_user)):
+    case = _case(db, fingerprint)
+    # Set the relationship, not just the id: the response is serialised from
+    # this same object and a bare id leaves `assignee` null in the payload.
+    case.assignee = user
+    if case.status == CaseStatus.NEW:
+        case.status = CaseStatus.INVESTIGATING
+    add_event(db, case, user, "assigned", f"Picked up by {user.email}")
+    db.commit()
+    db.refresh(case)
+    return case.to_json(with_events=True)
+
+
+@router.post("/{fingerprint}/note")
+def add_note(fingerprint: str, body: NoteBody, db: Session = Depends(get_db),
+             user: User = Depends(current_user)):
+    if not body.note.strip():
+        raise HTTPException(status.HTTP_400_BAD_REQUEST, "Empty note")
+    case = _case(db, fingerprint)
+    case.notes = (case.notes + "\n" if case.notes else "") + body.note.strip()
+    add_event(db, case, user, "note", body.note.strip())
+    db.commit()
+    return case.to_json(with_events=True)
+
+
+@router.post("/{fingerprint}/snooze")
+def snooze(fingerprint: str, body: SnoozeBody, db: Session = Depends(get_db),
+           user: User = Depends(current_user)):
+    case = _case(db, fingerprint)
+    until = dt.datetime.now(dt.timezone.utc) + dt.timedelta(hours=max(1, body.hours))
+    case.snooze_until = until
+    add_event(db, case, user, "snoozed", f"Snoozed for {body.hours}h. {body.note}".strip())
+    db.commit()
+    return case.to_json(with_events=True)

backend/app/routers/linkage_router.py

@@ -0,0 +1,42 @@
+"""Linkage scan endpoints."""
+from __future__ import annotations
+
+import threading
+
+from fastapi import APIRouter, Depends, HTTPException, status
+from pydantic import BaseModel
+
+from triagelib import linkage as linkage_mod
+
+from ..auth import current_user
+from ..config import get_settings
+from ..models import User
+from ..routers.alerts_router import engine
+
+router = APIRouter(prefix="/api/linkage", tags=["linkage"])
+settings = get_settings()
+
+
+class EnrichBody(BaseModel):
+    region: str
+    openstack_id: str
+
+
+@router.get("")
+def scan_state(user: User = Depends(current_user)):
+    return engine.scan.to_json()
+
+
+@router.post("/scan")
+def start_scan(user: User = Depends(current_user)):
+    if not settings.feature_linkage_scan:
+        raise HTTPException(status.HTTP_403_FORBIDDEN, "The linkage scan is disabled on this instance.")
+    if engine.scan.state == "running":
+        return {"started": False, "reason": "already running"}
+    threading.Thread(target=engine.scan.run, args=(engine.snapshot.get(),), daemon=True).start()
+    return {"started": True}
+
+
+@router.post("/enrich")
+def enrich(body: EnrichBody, user: User = Depends(current_user)):
+    return linkage_mod.enrich(body.region, body.openstack_id)

backend/app/routers/settings_router.py

@@ -0,0 +1,103 @@
+"""Suppression rules and per-instance preferences."""
+from __future__ import annotations
+
+from typing import Any
+
+from fastapi import APIRouter, Depends, HTTPException, status
+from pydantic import BaseModel
+from sqlalchemy.orm import Session
+
+from triagelib import settings as legacy
+
+from ..auth import current_user, require_admin
+from ..config import get_settings
+from ..db import get_db
+from ..models import AppSetting, SuppressionRule, User
+from ..routers.alerts_router import engine
+
+router = APIRouter(prefix="/api/settings", tags=["settings"])
+app_settings = get_settings()
+
+
+class RuleBody(BaseModel):
+    id: str | None = None
+    name: str
+    reason: str = ""
+    enabled: bool = True
+    conditions: dict[str, Any] = {}
+
+
+class GeneralBody(BaseModel):
+    agent_name: str | None = None
+    chronic_days: int | None = None
+
+
+@router.get("")
+def read_settings(db: Session = Depends(get_db), user: User = Depends(current_user)):
+    row = db.get(AppSetting, "general")
+    general = (row.value if row else {}) or {}
+    return {
+        "rules": [r.to_json() for r in db.query(SuppressionRule).order_by(SuppressionRule.id).all()],
+        "conditions": legacy.CONDITIONS,
+        "agent_name": general.get("agent_name") or "",
+        "chronic_days": general.get("chronic_days") or 3,
+        "config": app_settings.public_flags(),
+    }
+
+
+@router.post("/general")
+def save_general(body: GeneralBody, db: Session = Depends(get_db),
+                 user: User = Depends(current_user)):
+    row = db.get(AppSetting, "general") or AppSetting(key="general", value={})
+    value = dict(row.value or {})
+    if body.agent_name is not None:
+        value["agent_name"] = body.agent_name.strip()
+    if body.chronic_days is not None:
+        value["chronic_days"] = max(1, int(body.chronic_days))
+    row.value = value
+    db.merge(row)
+    db.commit()
+    return read_settings(db, user)
+
+
+@router.post("/rules")
+def save_rule(body: RuleBody, db: Session = Depends(get_db), user: User = Depends(require_admin)):
+    conditions = {k: v for k, v in (body.conditions or {}).items() if k in legacy.CONDITIONS and v}
+    if not conditions:
+        raise HTTPException(status.HTTP_400_BAD_REQUEST,
+                            "A rule needs at least one condition, otherwise it would hide everything.")
+    rule = db.get(SuppressionRule, int(body.id)) if (body.id or "").isdigit() else None
+    if rule is None:
+        rule = SuppressionRule(created_by=user.email)
+        db.add(rule)
+    rule.name = body.name.strip() or "Untitled rule"
+    rule.reason = body.reason.strip()
+    rule.enabled = body.enabled
+    rule.conditions = conditions
+    db.commit()
+    return read_settings(db, user)
+
+
+@router.delete("/rules/{rule_id}")
+def delete_rule(rule_id: int, db: Session = Depends(get_db), user: User = Depends(require_admin)):
+    rule = db.get(SuppressionRule, rule_id)
+    if rule:
+        db.delete(rule)
+        db.commit()
+    return read_settings(db, user)
+
+
+@router.post("/rules/preview")
+def preview_rule(body: RuleBody, db: Session = Depends(get_db), user: User = Depends(current_user)):
+    """Show which firing alerts a rule would hide, before it is saved."""
+    from triagelib import alerts as alertlib
+
+    raw, _error, _age = engine.cache.get()
+    candidates = [a for a in (alertlib.from_prometheus(x, engine.rules) for x in raw)
+                  if not alertlib.is_excluded(a) and alertlib.cx_relevant(a)]
+    rule = legacy._normalize_rule({"name": body.name, "conditions": body.conditions})
+    hits = [{
+        "kind": a.kind, "title": a.title, "instance_name": a.instance_name,
+        "host": a.host, "org_name": a.org_name, "region": a.region,
+    } for a in candidates if legacy.rule_matches(rule, a)]
+    return {"count": len(hits), "matches": hits[:60]}

backend/app/services.py

@@ -0,0 +1,200 @@
+"""Glue between the triage engine, the database and the outside world."""
+from __future__ import annotations
+
+import datetime as dt
+import threading
+from typing import Any, Optional
+
+from sqlalchemy.orm import Session
+
+from triagelib import alerts as alertlib, screening, settings as legacy_settings
+from triagelib.prometheus import (AlertCache, PrometheusClient, PrometheusError,
+                                  RuleIndex, StateSnapshot, TrueAgeIndex)
+from triagelib import linkage as linkage_mod
+
+from .config import get_settings
+from .models import AppSetting, Case, CaseEvent, CaseStatus, SuppressionRule, User
+
+settings = get_settings()
+
+
+class RuleAdapter:
+    """Presents DB-backed suppression rules the way the engine expects."""
+
+    def __init__(self, db: Session):
+        self._rules = [r.to_json() for r in db.query(SuppressionRule).all()]
+        row = db.get(AppSetting, "general")
+        general = (row.value if row else {}) or {}
+        self._agent = str(general.get("agent_name") or "")
+        self._chronic = int(general.get("chronic_days") or 3)
+
+    @property
+    def rules(self) -> list[dict[str, Any]]:
+        return self._rules
+
+    @property
+    def agent_name(self) -> str:
+        return self._agent
+
+    @property
+    def chronic_days(self) -> int:
+        return self._chronic
+
+
+class Engine:
+    """Process-wide caches over Prometheus. Cheap to share, expensive to rebuild."""
+
+    def __init__(self):
+        self.prom = PrometheusClient(settings.prometheus_base)
+        self.cache = AlertCache(self.prom)
+        self.rules = RuleIndex(self.prom)
+        self.snapshot = StateSnapshot(self.prom)
+        self.true_age = TrueAgeIndex(self.prom)
+        self.scan = linkage_mod.Scan()
+        self._lock = threading.Lock()
+
+    def warm(self, log=print) -> None:
+        try:
+            self.rules.ensure()
+            snap = self.snapshot.get()
+            log(f"  state snapshot: {len(snap.by_openstack_id)} VMs, {len(snap.total_gpus)} hosts")
+            ages = self.true_age.get()
+            log(f"  alert history: {ages.count} alerts indexed over {ages.WINDOW_DAYS} days")
+        except PrometheusError as exc:
+            log(f"  WARN: Prometheus caches not warmed: {exc}")
+
+    def queue(self, db: Session, force: bool = False) -> dict[str, Any]:
+        raw, error, age = self.cache.get(force=force)
+        snap = self.snapshot.get()
+        ages = self.true_age.get()
+        parsed = [alertlib.from_prometheus(a, self.rules, ages) for a in raw]
+
+        excluded = [a for a in parsed if alertlib.is_excluded(a)]
+        candidates = [a for a in parsed if not alertlib.is_excluded(a)]
+        cx = [a for a in candidates if a.category == "cx" and alertlib.cx_relevant(a)]
+        screening.screen_all(cx, snap, RuleAdapter(db))
+
+        cases = {c.fingerprint: c for c in
+                 db.query(Case).filter(Case.fingerprint.in_([a.fingerprint() for a in cx])).all()}
+        groups = alertlib.group_alerts(cx)
+        for group in groups:
+            for item in group["alerts"]:
+                case = cases.get(item["id"])
+                item["case"] = case.to_json() if case else None
+
+        in_cx = {id(a) for a in cx}
+        infra = sorted([a for a in candidates if id(a) not in in_cx], key=alertlib.sort_key)
+
+        return {
+            "error": error or snap.error or ages.error,
+            "warnings": screening.health_warnings(snap),
+            "totals": {"prometheus": len(parsed), "cx": len(cx),
+                       "infrastructure": len(infra), "excluded": len(excluded)},
+            "excluded_note": (f"{len(excluded)} '{', '.join(sorted({a.alertname for a in excluded}))}' alerts hidden"
+                              if excluded else ""),
+            "summary": screening.summarize(cx),
+            "groups": groups,
+            "infrastructure": _infra_sections(infra),
+            "cache_age_seconds": round(age, 1),
+        }
+
+    def find_alert(self, db: Session, fingerprint: str) -> Optional[Any]:
+        raw, _error, _age = self.cache.get()
+        ages = self.true_age.get()
+        for item in raw:
+            alert = alertlib.from_prometheus(item, self.rules, ages)
+            if alert.fingerprint() == fingerprint:
+                alert.screen = screening.screen(alert, self.snapshot.get(), RuleAdapter(db))
+                return alert
+        return None
+
+
+SOURCE_LABELS = {
+    "node-exporter-rules.yml": "Node exporter (hosts)",
+    "ceph-rules.yml": "Ceph", "mysql-rules.yml": "MySQL",
+    "mysql-performance-rules.yml": "MySQL performance", "galera-rules.yml": "Galera",
+    "openstack-rules.yml": "OpenStack services", "blackbox.yml": "Blackbox / OOB",
+    "infrahub-rules.yml": "Infrahub (no CX runbook)",
+}
+
+
+def _infra_sections(items: list[Any]) -> list[dict[str, Any]]:
+    buckets: dict[str, list[Any]] = {}
+    for alert in items:
+        buckets.setdefault(alert.rule_file or "unknown", []).append(alert)
+    sections = []
+    for source, members in buckets.items():
+        by_name: dict[str, int] = {}
+        for alert in members:
+            name = alertlib.clean_alertname(alert.alertname)
+            by_name[name] = by_name.get(name, 0) + 1
+        sections.append({
+            "source": source, "label": SOURCE_LABELS.get(source, source), "total": len(members),
+            "by_alertname": sorted(({"name": k, "count": v} for k, v in by_name.items()),
+                                   key=lambda x: (-x["count"], x["name"])),
+        })
+    sections.sort(key=lambda s: (s["source"] != alertlib.NODE_RULE_FILE, -s["total"]))
+    return sections
+
+
+# --- case bookkeeping -------------------------------------------------------
+
+def get_or_create_case(db: Session, alert: Any, actor: Optional[User] = None) -> Case:
+    case = db.query(Case).filter(Case.fingerprint == alert.fingerprint()).one_or_none()
+    now = dt.datetime.now(dt.timezone.utc)
+    subject = (alert.floating_ip if alert.kind == "duplicate_ip"
+               else alert.host if alert.kind in ("rogue_vm", "total_gpus", "orphan_vm")
+               else alert.instance_name or alert.openstack_id)
+    if case is None:
+        case = Case(
+            fingerprint=alert.fingerprint(), kind=alert.kind, title=alert.title,
+            subject=subject or "", openstack_id=alert.openstack_id,
+            instance_name=alert.instance_name, host=alert.host, region=alert.region,
+            org_id=alert.org_id, org_name=alert.org_name,
+        )
+        db.add(case)
+        db.flush()
+        add_event(db, case, actor, "opened", f"Case opened for {alert.title}")
+    else:
+        # A closed case whose alert has come back is new work again.
+        if not case.is_open and case.closed_at:
+            case.reopen_count += 1
+            case.status = CaseStatus.NEW
+            case.closed_at = None
+            add_event(db, case, None, "reopened",
+                      f"Alert fired again after being {case.status.value}")
+        case.last_seen_at = now
+        case.title = alert.title
+        case.subject = subject or case.subject
+    db.commit()
+    return case
+
+
+def add_event(db: Session, case: Case, actor: Optional[User], action: str,
+              detail: str = "", payload: Optional[dict[str, Any]] = None) -> CaseEvent:
+    event = CaseEvent(
+        actor_id=actor.id if actor else None,
+        actor_label=(actor.email if actor else "system"),
+        action=action, detail=detail, payload=payload,
+    )
+    # Appended through the relationship rather than inserted by id: sessions use
+    # expire_on_commit=False, so a collection already loaded would otherwise stay
+    # stale and the new event would be missing from the response.
+    case.events.append(event)
+    db.add(event)
+    return event
+
+
+def set_status(db: Session, case: Case, status: CaseStatus, actor: Optional[User],
+               note: str = "") -> Case:
+    previous = case.status
+    case.status = status
+    if status in (CaseStatus.RESOLVED, CaseStatus.WONT_FIX, CaseStatus.FALSE_POSITIVE):
+        case.closed_at = dt.datetime.now(dt.timezone.utc)
+    else:
+        case.closed_at = None
+    add_event(db, case, actor, "status_changed",
+              f"{previous.value} -> {status.value}" + (f": {note}" if note else ""),
+              {"from": previous.value, "to": status.value})
+    db.commit()
+    return case

backend/requirements.txt

@@ -0,0 +1,6 @@
+fastapi==0.141.1
+uvicorn[standard]==0.34.0
+sqlalchemy==2.0.51
+httpx==0.28.1
+pydantic==2.10.6
+psycopg[binary]==3.2.4

backend/tests/test_api.py

@@ -0,0 +1,124 @@
+"""API-level tests: auth gates, case lifecycle, suppression rules, send gating.
+
+Runs against an in-memory database with CX-Tools stubbed out, so it needs no
+credentials. Prometheus is only touched for cache warming, which is tolerant of
+being unreachable.
+"""
+import os
+import sys
+import tempfile
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
+
+os.environ.update({
+    "CX_DATABASE_URL": f"sqlite:///{tempfile.mkdtemp()}/test.db",
+    "CX_BOOTSTRAP_ADMIN_EMAIL": "admin@localhost",
+    "CX_BOOTSTRAP_ADMIN_PASSWORD": "test-password",
+    "CX_SECRET_KEY": "test-secret",
+    "CX_FEATURE_SEND_ENABLED": "false",
+})
+
+from triagelib import cxbridge  # noqa: E402
+
+cxbridge.bootstrap = lambda: (_ for _ in ()).throw(cxbridge.BridgeError("stubbed"))
+
+from fastapi.testclient import TestClient  # noqa: E402
+
+from app import auth as auth_mod  # noqa: E402
+from app.main import app  # noqa: E402
+
+FAILS = []
+
+
+def expect(label, cond, got=""):
+    print(("  PASS  " if cond else "  FAIL  ") + label + ("" if cond else f"   <- {got}"))
+    if not cond:
+        FAILS.append(label)
+
+
+print("\nPASSWORDS AND SESSIONS")
+h = auth_mod.hash_password("hunter2")
+expect("correct password verifies", auth_mod.verify_password("hunter2", h))
+expect("wrong password rejected", not auth_mod.verify_password("hunter3", h))
+expect("hash is salted (two hashes differ)", auth_mod.hash_password("x") != auth_mod.hash_password("x"))
+token = auth_mod.issue_session(7)
+expect("session round-trips", auth_mod.read_session(token) == 7)
+expect("tampered session rejected", auth_mod.read_session(token[:-4] + "aaaa") is None)
+expect("garbage session rejected", auth_mod.read_session("not-a-token") is None)
+expect("oidc state verifies", auth_mod.oidc_state_valid(auth_mod.oidc_state()))
+expect("forged oidc state rejected", not auth_mod.oidc_state_valid("aaa.bbb"))
+
+with TestClient(app) as client:
+    print("\nAUTH GATES")
+    expect("health is public", client.get("/api/health").status_code == 200)
+    expect("alerts need a session", client.get("/api/alerts").status_code == 401)
+    expect("cases need a session", client.get("/api/cases").status_code == 401)
+    expect("wrong password is 401", client.post(
+        "/api/auth/login", json={"email": "admin@localhost", "password": "no"}).status_code == 401)
+
+    login = client.post("/api/auth/login", json={"email": "admin@localhost", "password": "test-password"})
+    expect("login succeeds", login.status_code == 200, login.text[:120])
+    expect("bootstrap user is admin", login.json()["user"]["is_admin"])
+    expect("session works after login", client.get("/api/cases").status_code == 200)
+
+    print("\nCONFIG EXPOSURE")
+    cfg = client.get("/api/auth/me").json()["config"]
+    expect("send disabled by default", cfg["send_enabled"] is False, cfg)
+    expect("no secret leaks into public config",
+           not any("token" in k.lower() or "secret" in k.lower() for k in cfg), list(cfg))
+
+    print("\nSUPPRESSION RULES")
+    rule = {"name": "Modal ERROR churn", "reason": "known batch churn",
+            "conditions": {"kind": ["error"], "organization": ["modal"]}}
+    expect("admin can save a rule", client.post("/api/settings/rules", json=rule).status_code == 200)
+    expect("rule is persisted", len(client.get("/api/settings").json()["rules"]) == 1)
+    expect("a rule with no conditions is refused", client.post(
+        "/api/settings/rules", json={"name": "catch all", "conditions": {}}).status_code == 400)
+    expect("unknown condition fields are dropped", client.post(
+        "/api/settings/rules", json={"name": "bogus", "conditions": {"nope": ["x"]}}).status_code == 400)
+
+    print("\nSEND GATING")
+    status = client.get("/api/actions/status").json()
+    expect("send reported as disabled", status["send_enabled"] is False)
+    expect("zendesk reported as not ready", status["zendesk_ready"] is False)
+    blocked = client.post("/api/actions/zendesk", json={
+        "fingerprint": "does-not-exist", "to": "a@b.c", "subject": "s", "body": "b"})
+    expect("sending on an untracked alert is refused", blocked.status_code == 404, blocked.text[:120])
+
+    print("\nCASE LIFECYCLE")
+    from app.db import SessionLocal
+    from app.models import Case, CaseStatus
+    from app.services import add_event, set_status
+
+    db = SessionLocal()
+    case = Case(fingerprint="test-fp", kind="error", title="Instance in ERROR state", subject="vm-1")
+    db.add(case)
+    db.commit()
+    expect("new case starts open", case.is_open and case.status == CaseStatus.NEW)
+
+    set_status(db, case, CaseStatus.ESCALATED_INFRA, None, "INFRA-1")
+    expect("escalated is still open", case.is_open)
+    expect("status change is recorded", any(e.action == "status_changed" for e in case.events))
+
+    set_status(db, case, CaseStatus.RESOLVED, None)
+    expect("resolved closes the case", not case.is_open and case.closed_at is not None)
+
+    add_event(db, case, None, "note", "manual note")
+    db.commit()
+    expect("history is append-only and ordered newest first",
+           case.events[0].action in ("note", "status_changed"), [e.action for e in case.events])
+
+    payload = case.to_json(with_events=True)
+    expect("serialises for the API", payload["status"] == "resolved" and len(payload["events"]) >= 3)
+    db.close()
+
+    expect("bad status is rejected", client.post(
+        "/api/cases/test-fp/status", json={"status": "banana"}).status_code == 400)
+    expect("unknown case is 404", client.get("/api/cases/nope").status_code == 404)
+
+    print("\nLOGOUT")
+    client.post("/api/auth/logout")
+    expect("session is cleared", client.get("/api/cases").status_code == 401)
+
+print("\n" + ("ALL CHECKS PASSED" if not FAILS else f"{len(FAILS)} CHECK(S) FAILED: {FAILS}"))
+sys.exit(1 if FAILS else 0)

backend/tests/test_runbooks.py

@@ -0,0 +1,268 @@
+"""Runbook decision tests: fixtures shaped like real CX-Tools collector output."""
+import sys, os
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
+
+from triagelib import alerts as A, cxbridge, runbooks
+
+# --- stub the CX-Tools boundary so the decision logic can be tested alone ---
+class FakeCx:
+    def status_pair_ok(self, ih, os_, task):
+        ih, os_ = (ih or "").upper(), (os_ or "").upper()
+        if ih == os_: return True
+        return (ih, os_) in {("HIBERNATED","SHELVED_OFFLOADED"), ("REBOOTING","HARD_REBOOT")}
+    def mismatch_parts(self, item):
+        return str(item.get("check","Mismatch")), str(item.get("detail",""))
+    def normalize_empty(self, v): return "" if v in (None,"","None") else str(v)
+    def first_present(self, m, *keys, default=""):
+        for k in keys:
+            if isinstance(m, dict) and k in m: return m[k]
+        return default
+    def public_ip_from_server(self, s): return (s or {}).get("_public_ip","")
+    def json_safe(self, o): return o
+    def map_region(self, r): return {"CANADA-1":"ca1","US-1":"us1","NORWAY-1":"no1"}.get(r,"")
+
+STUB = {"host_health": {}, "gpu_census": {}, "failed_event": {}, "vm": {}, "host": {}}
+cxbridge.cx = lambda: FakeCx()
+cxbridge.host_health = lambda region, host: STUB["host_health"]
+cxbridge.host_gpu_census = lambda region, host: STUB["gpu_census"]
+cxbridge.failed_openstack_event = lambda region, osid, scan=5: STUB["failed_event"]
+cxbridge.collect_vm = lambda t, **k: STUB["vm"]
+cxbridge.collect_host = lambda h, **k: STUB["host"]
+cxbridge.json_safe = lambda o: o
+
+HEALTHY = {"ok": True, "nova_state": "up", "nova_status": "enabled", "ovs_alive": True,
+           "ovs_state": "UP", "uptime": "17:54", "aggregates": "agg", "bad_signals": []}
+SICK = {**HEALTHY, "nova_state": "down", "ovs_state": "DOWN", "ovs_alive": False,
+        "bad_signals": ["Nova state is down", "OVS state is DOWN"]}
+
+def vm(**over):
+    base = {"ok": True, "exit_code": 0, "mode": "vm", "infrahub_id": "123456",
+            "openstack_id": "9ec7a021-d741-484f-9387-4eaf8879fd77", "name": "test-vm",
+            "region": "ca1", "region_display": "CANADA-1", "ih_status": "ACTIVE", "os_status": "ACTIVE",
+            "task_state": "None", "host": "CA1-ESC8-040", "flavor": "n3-H100x8", "gpu_count": "8",
+            "floating_ip": "69.19.140.110", "created": "2026-07-01 10:00:00 UTC", "ssh_text": "reachable",
+            "ssh_raw": "reachable", "volumes_summary": "None", "openstack_fault": "None",
+            "faults": [], "ih_events": [], "ih_events_all": [], "mismatches": [], "warn_reasons": [],
+            "info_notes": [], "org_value": "8463 - Acme Corp", "owners": ["Lars <lars@simli.com>"],
+            "server": {"status": "ACTIVE"}, "infrahub": {"floating_ip": "69.19.140.110"}}
+    base.update(over); return base
+
+def alert(name, **labels):
+    return A.from_labels({"alertname": name, **labels})
+
+def run(name, labels, vmdata=None, hostdata=None, health=None, census=None, fevent=None, prom=None):
+    STUB.update({"vm": vmdata or {}, "host": hostdata or {}, "host_health": health or HEALTHY,
+                 "gpu_census": census or {}, "failed_event": fevent or {}})
+    return runbooks.diagnose(alert(name, **labels), prom)
+
+def check(label, cond, extra=""):
+    print(f"  {'PASS' if cond else 'FAIL'}  {label}" + (f"  <- {extra}" if not cond and extra else ""))
+    return cond
+
+fails = 0
+def expect(label, cond, extra=""):
+    global fails
+    if not check(label, cond, extra): fails += 1
+
+L_ERR = dict(openstack_id="9ec7a021-d741-484f-9387-4eaf8879fd77", region="CANADA-1",
+             instance_name="test-vm", organization="8463 - Acme Corp", status="ERROR")
+
+print("\n[1] ERROR - creation failed, never reached ACTIVE, insufficient stock")
+d = run("Instance in ERROR state in :flag-ca:CA-1", L_ERR,
+        vm(host="N/A", ih_status="ERROR", os_status="ERROR", server={"status": "ERROR"},
+           openstack_fault="No valid host was found. There are not enough hosts available"))
+expect("matched no_valid_host fault", "scheduler could not place" in d.verdict.lower(), d.verdict)
+expect("identified as never-ACTIVE", any("never placed on a host" in f.value for f in d.findings))
+expect("chose the insufficient-stock template", any(x.template_id == "error_never_active" for x in d.drafts),
+       [x.template_id for x in d.drafts])
+expect("mentions 7-day outreach window", any("7 days" in x.when for x in d.drafts))
+expect("escalates to Infrastructure", any(a.owner == runbooks.INFRA for a in d.actions))
+expect("contacts resolved", d.contacts.get("resolved"))
+
+print("\n[2] ERROR - was ACTIVE, stale LVM on host")
+d = run("Instance in ERROR state in :flag-ca:CA-1", L_ERR,
+        vm(ih_status="ERROR", os_status="ERROR",
+           faults=[["500", "Build of instance aborted: Failed to remove volume(s): lvremove -f /dev/nova-vg/x_disk", "2026-07-29"]]))
+expect("matched lvremove fault", "stale LVM" in d.verdict, d.verdict)
+expect("identified as previously ACTIVE", any("hypervisor is recorded" in f.value for f in d.findings))
+expect("chose the was-ACTIVE template", any(x.template_id == "error_was_active" for x in d.drafts),
+       [x.template_id for x in d.drafts])
+expect("assessment flags possible customer data", "customer data" in d.assessment)
+
+print("\n[3] ERROR - NUMA/PCI fault, host is FULL (proves host fine)")
+d = run("Instance in ERROR state in :flag-ca:CA-1", L_ERR,
+        vm(ih_status="ERROR", os_status="ERROR",
+           openstack_fault="Insufficient compute resources: Requested instance NUMA topology together with requested PCI devices cannot fit the given host NUMA topology; Claim pci failed."),
+        census={"ok": True, "total_gpus": 8, "instances": [{"name":"a"}]*4})
+expect("ran the GPU census", any("GPUs allocated on host" in f.label for f in d.findings))
+expect("concluded host is FULL", any("FULL" in f.detail for f in d.findings))
+expect("marked the capacity check as already done", any(a.status == "done" for a in d.actions))
+
+print("\n[3b] same fault, host NOT full -> must escalate")
+d = run("Instance in ERROR state in :flag-ca:CA-1", L_ERR,
+        vm(ih_status="ERROR", os_status="ERROR",
+           openstack_fault="Claim pci failed."),
+        census={"ok": True, "total_gpus": 4, "instances": [{"name":"a"}]*2})
+expect("raises a Jira for Infra", any("Jira" in a.text and a.owner == runbooks.INFRA for a in d.actions))
+
+print("\n[4] SHUTOFF - billing notice only")
+d = run("Instance in SHUTOFF state in :flag-ca:CA-1 for greater than 30 min",
+        dict(openstack_id="dc156762-3fa8-481d-89e0-12c2fb55e046", region="CANADA-1",
+             instance_name="energetic-galileo", organization="27358 - Zyra", status="SHUTOFF"),
+        vm(name="energetic-galileo", ih_status="SHUTOFF", os_status="SHUTOFF", server={"status":"SHUTOFF"}))
+expect("verdict is customer-initiated", "customer-initiated" in d.verdict.lower(), d.verdict)
+expect("used the shutoff snippet", any(x.template_id == "shutoff" for x in d.drafts))
+expect("substituted the VM name", any("energetic-galileo" in x.body for x in d.drafts))
+expect("no Infra escalation", not any(a.owner == runbooks.INFRA for a in d.actions))
+
+print("\n[5] DELETING - server still in OpenStack, delete request found")
+d = run("Instance in DELETING state in :flag-ca:CA-1 for greater than 30 min",
+        dict(openstack_id="13a855a4-c563-4350-9a8e-ffa9efa27d9e", region="CANADA-1",
+             instance_name="external-hs-h100", organization="10420 - Inceptions AI", status="DELETING"),
+        vm(ih_status="DELETING", os_status="ACTIVE", server={"status": "ACTIVE"},
+           ih_events_all=[["2026-07-29 21:40:00", "InstanceDeleteRequest", "Delete Instance Request Sent."]]))
+expect("confirmed customer intent from events", any(a.status == "done" and "intent" in a.text for a in d.actions))
+expect("tells CX to delete in OpenStack", any("Delete the server in OpenStack" in a.text for a in d.actions))
+expect("offers both the notice and the ticket-closing reply",
+       sorted(x.template_id for x in d.drafts) == ["deleting", "deleting_resolved"],
+       [x.template_id for x in d.drafts])
+expect("tells CX to close the Infrahub record too, not just the server",
+       any("InfraInsight" in a.text for a in d.actions), [a.text for a in d.actions])
+
+print("\n[5c] DELETING - server exists but never reached a host")
+d = run("Instance in DELETING state in :flag-ca:CA-1 for greater than 30 min",
+        dict(openstack_id="x", region="CANADA-1", status="DELETING"),
+        vm(ih_status="DELETING", os_status="ERROR", host="N/A",
+           server={"status": "ERROR"}, openstack_fault="No valid host was found."))
+expect("flags that the build never completed", any("build never completed" in f.value for f in d.findings))
+expect("still requires the InfraInsight close-out", any("InfraInsight" in a.text for a in d.actions))
+
+print("\n[5b] DELETING - already gone from OpenStack, no delete event")
+d = run("Instance in DELETING state in :flag-ca:CA-1 for greater than 30 min",
+        dict(openstack_id="x", region="CANADA-1", status="DELETING"),
+        vm(ih_status="DELETING", os_status="N/A", server={}, ih_events_all=[]))
+expect("notes the server is already gone", any("already gone" in f.value for f in d.findings))
+expect("routes attribution to DevOps", any(a.owner == runbooks.DEVOPS for a in d.actions))
+
+print("\n[6] CREATING - never got an OpenStack ID")
+d = run("Instance in CREATING state in :flag-ca:CA-1 for greater than 30min",
+        dict(openstack_id="None", region="CANADA-1", instance_name="vm-28070520-5d1f2",
+             organization="5574 - Nexgen", status="CREATING"),
+        vm(openstack_id="N/A", ih_status="CREATING", os_status="N/A", server={}, host="N/A"))
+expect("verdict says never got an OpenStack ID", "never got an OpenStack ID" in d.verdict, d.verdict)
+expect("used the creating template", any(x.template_id == "creating" for x in d.drafts))
+expect("instructs deletion", any("Delete the stuck instance" in a.text for a in d.actions))
+
+print("\n[7] HIBERNATING - sick host")
+d = run("Instance in HIBERNATING state in :flag-ca:CA-1 for greater than 120min",
+        dict(openstack_id="28171e6f", region="CANADA-1", instance="CA1-ESC812-211",
+             instance_name="apt25-prod", status="HIBERNATING"),
+        vm(ih_status="HIBERNATING", os_status="ACTIVE"), health=SICK)
+expect("verdict blames the host", "host problem" in d.verdict, d.verdict)
+expect("escalates to Infra with the bad signals", any(a.owner == runbooks.INFRA and "OVS" in a.text for a in d.actions))
+expect("still drives the shelve", any("shelve" in a.text.lower() for a in d.actions))
+
+print("\n[8] Suspected Rogue VM - host with two different mismatches")
+host_result = {"ok": True, "mode": "host", "host": "CA1-ESC8-068", "region": "ca1", "server_count": 3,
+    "hypervisor": {"state": "up", "status": "enabled"}, "ovs": {"alive": True, "state": "UP"},
+    "instances": [
+        {"idx": 1, "name": "vm-hib-shutoff", "infrahub_id": "1", "openstack_id": "a", "ih_status": "HIBERNATED",
+         "os_status": "SHUTOFF", "host": "CA1-ESC8-068", "mismatches": [{"check": "State", "detail": "IH HIBERNATED vs OS SHUTOFF"}],
+         "warn_reasons": ["mismatch detected"], "org_value": "1 - A", "owners": ["a@x.com"]},
+        {"idx": 2, "name": "vm-hib-active", "infrahub_id": "2", "openstack_id": "b", "ih_status": "HIBERNATED",
+         "os_status": "ACTIVE", "host": "CA1-ESC8-068", "mismatches": [{"check": "State", "detail": "IH HIBERNATED vs OS ACTIVE"}],
+         "warn_reasons": ["mismatch detected"], "org_value": "2 - B", "owners": ["b@x.com"]},
+        {"idx": 3, "name": "tempest-thing", "tempest": True, "ih_status": "N/A", "os_status": "ACTIVE",
+         "mismatches": [{"check": "Infrahub Missing", "detail": "not in Infrahub"}], "warn_reasons": []},
+    ]}
+d = run(":ninja:Suspected Rogue VM", dict(instance="CA1-ESC8-068"), hostdata=host_result)
+expect("counted 2 of 3 as mismatched", "2 of 3" in d.verdict, d.verdict)
+expect("ignored the tempest instance", d.evidence.get("ignored_tempest") == 1)
+expect("HIBERNATED/SHUTOFF -> Windmill stale-image cleanup", any("Windmill" in a.text for a in d.actions))
+expect("HIBERNATED/ACTIVE -> sync-error comms", any(x.template_id == "sync_state" for x in d.drafts))
+expect("actions are scoped per instance", any(a.text.startswith("[vm-hib-active]") for a in d.actions))
+
+print("\n[9] Suspected Rogue VM - clean host")
+d = run(":ninja:Suspected Rogue VM", dict(instance="CA1-ESC8-068"),
+        hostdata={**host_result, "instances": [{"idx":1,"name":"ok","ih_status":"ACTIVE","os_status":"ACTIVE",
+                  "mismatches": [], "warn_reasons": []}]})
+expect("verdict says no mismatch", "No Infrahub/OpenStack mismatch" in d.verdict, d.verdict)
+expect("redirects to the InfraInsight host query", any("InfraInsight" in a.text for a in d.actions))
+
+print("\n[10] Duplicated IPs - one DELETING claimant, one with a wrong Infrahub IP")
+multi = {"ok": True, "mode": "multi_vm", "instances": [
+    {"name": "old-vm", "infrahub_id": "1", "openstack_id": "a", "ih_status": "DELETING", "os_status": "ACTIVE",
+     "server": {"status": "ACTIVE", "_public_ip": "69.19.137.135"}, "infrahub": {"floating_ip": "69.19.137.135"},
+     "org_value": "1 - A", "owners": ["a@x.com"]},
+    {"name": "new-vm", "infrahub_id": "2", "openstack_id": "b", "ih_status": "ACTIVE", "os_status": "ACTIVE",
+     "server": {"status": "ACTIVE", "_public_ip": "69.19.140.9"}, "infrahub": {"floating_ip": "69.19.137.135"},
+     "org_value": "2 - B", "owners": ["b@x.com"]},
+]}
+class FakeProm:
+    def resources_by_floating_ip(self, fip):
+        return [{"instance_name": "preprod-vm", "status": "ACTIVE", "region": "CANADA-1", "environment": "preprod"}]
+d = run(":awkward:Duplicated IPs", dict(floating_ip="69.19.137.135"), vmdata=multi, prom=FakeProm())
+expect("found 2 claimants needing correction", "2 of 2" in d.verdict, d.verdict)
+expect("DELETING claimant -> delete it", any("[old-vm]" in a.text and "stuck DELETING" in a.text for a in d.actions))
+expect("wrong-IP claimant -> Scenario #2", any("Scenario #2" in f.value for f in d.findings))
+expect("Scenario #2 comms filled with the real IP", any(
+    x.template_id == "dupip_corrected" and "69.19.140.9" in x.body for x in d.drafts))
+expect("an unset sign-off name is reported rather than left as a placeholder", any(
+    "AGENT_NAME" in x.unfilled for x in d.drafts if x.template_id == "dupip_corrected"))
+expect("surfaced the PreProd claimant from Prometheus", any("preprod-vm" in f.label for f in d.findings))
+expect("asks for a re-check after 5-10 min", any("5-10 minutes" in a.text for a in d.actions))
+
+print("\n[11] Problem with Total GPUs - customers on host")
+d = run("Problem with Total GPUs in a System", dict(instance="CA1-ESC8-111", region="CANADA-1", gpu_name="B200-SXM"),
+        census={"ok": True, "total_gpus": 6, "instances": [
+            {"name": "cust-vm", "status": "ACTIVE", "flavor": "n3-B200x6", "gpus": "6", "openstack_id": "z"}]})
+expect("verdict names the host", "CA1-ESC8-111" in d.verdict, d.verdict)
+expect("lists the affected instance", any("cust-vm" in f.label for f in d.findings))
+expect("flags host-maintenance comms", any(a.kind == "comms" for a in d.actions))
+expect("escalates a Jira to Infra", any("Jira" in a.text and a.owner == runbooks.INFRA for a in d.actions))
+expect("notes there is no approved template", any("no approved customer template" in n for n in d.notes))
+expect("drafts nothing", not d.drafts)
+
+print("\n[12] K8s instance name is flagged")
+d = run("Instance in ERROR state in :flag-ca:CA-1", {**L_ERR, "instance_name": "hyperstack-minion-lydhjev"},
+        vm(host="N/A", ih_status="ERROR", os_status="ERROR"))
+expect("noted the likely K8s node", any("Kubernetes" in n for n in d.notes))
+
+
+print("\n[13] GPU sockets - every physical GPU accounted for")
+from triagelib.runbooks import _gpu_slots
+import collections as _c
+
+R289 = [{"name": "luminous-hubble", "gpus": "1", "linked": True, "match": True, "ih_status": "ACTIVE", "os_status": "ACTIVE"},
+        {"name": "vm832adbe203242", "gpus": "1", "linked": True, "match": True, "ih_status": "ACTIVE", "os_status": "ACTIVE"},
+        {"name": "noble-maxwell", "gpus": "1", "linked": True, "match": True, "ih_status": "ACTIVE", "os_status": "ACTIVE"},
+        {"name": "clever-schrodinger", "gpus": "2", "linked": True, "match": True, "ih_status": "ACTIVE", "os_status": "ACTIVE"}]
+
+# Real numbers from CA1-ESC812-289: 8 physical, 7 reported in use, 4 VMs on 5 GPUs.
+s = _gpu_slots({"physical": 8, "in_use_metric": 7, "spare_capacity_artifact": False}, R289)
+c = _c.Counter(x["kind"] for x in s)
+expect("8 sockets drawn for an 8-GPU host", len(s) == 8, len(s))
+expect("5 named + 2 unaccounted + 1 free", (c["vm"], c["unaccounted"], c["free"]) == (5, 2, 1), dict(c))
+
+s = _gpu_slots({"physical": 8, "in_use_metric": 8, "spare_capacity_artifact": True}, R289)
+c = _c.Counter(x["kind"] for x in s)
+expect("artifact host shows spare sockets as free, not unaccounted",
+       (c["vm"], c["unaccounted"], c["free"]) == (5, 0, 3), dict(c))
+
+s = _gpu_slots({"physical": None, "in_use_metric": 2, "spare_capacity_artifact": False},
+               [{"name": "basilica", "gpus": "1", "linked": True, "match": True, "ih_status": "ACTIVE", "os_status": "ACTIVE"}])
+c = _c.Counter(x["kind"] for x in s)
+expect("unknown socket count degrades gracefully", (c["vm"], c["unaccounted"]) == (1, 1), dict(c))
+
+s = _gpu_slots({"physical": 8, "in_use_metric": 8, "spare_capacity_artifact": False},
+               [{"name": "ghost", "gpus": "8", "linked": False, "match": False,
+                 "ih_status": "not in Infrahub", "os_status": "ACTIVE"}])
+expect("a VM with no Infrahub record still fills its sockets and is flagged",
+       len(s) == 8 and all(x["kind"] == "vm" and not x["linked"] for x in s))
+
+s = _gpu_slots({"physical": 8, "in_use_metric": 5, "spare_capacity_artifact": False}, R289)
+expect("no negative slots when in_use is below what VMs claim", len(s) >= 5 and all(
+    x["kind"] in ("vm", "free", "unaccounted") for x in s), len(s))
+
+print(f"\n{'ALL CHECKS PASSED' if not fails else str(fails) + ' CHECK(S) FAILED'}")
+sys.exit(1 if fails else 0)

backend/tests/test_screening.py

@@ -0,0 +1,290 @@
+"""Screening, exclusion, categorisation and ordering tests.
+
+These cover the noise-vs-real decisions, which are what keeps the queue small.
+No network and no CX-Tools: snapshots are synthetic.
+"""
+import datetime as dt
+import os
+import sys
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
+
+from triagelib import alerts as A, screening
+
+
+class Snap:
+    """Stands in for prometheus.StateSnapshot."""
+
+    loaded = True
+
+    def __init__(self, **kw):
+        self.by_openstack_id = {}
+        self.by_instance_name = {}
+        self.fip_counts = {}
+        self.rogue_delta = {}
+        self.total_gpus = {}
+        self.in_use_gpus = {}
+        self.resources_by_host = {}
+        self.broken_inputs = []
+        self.unattributed_active = 0
+        self.unattributed_active_gpus = 0
+        self.__dict__.update(kw)
+
+
+def al(name, **labels):
+    return A.from_labels({"alertname": name, **labels})
+
+
+def aged(alert, minutes):
+    alert.active_at = dt.datetime.now(dt.timezone.utc) - dt.timedelta(minutes=minutes)
+    return alert
+
+
+FAILS = []
+
+
+def expect(label, cond, got=""):
+    print(("  PASS  " if cond else "  FAIL  ") + label + ("" if cond else f"   <- {got}"))
+    if not cond:
+        FAILS.append(label)
+
+
+OSID = "abc-123"
+ERROR_ALERT = "Instance in ERROR state in :flag-ca:CA-1"
+SHUTOFF_ALERT = "Instance in SHUTOFF state in :flag-ca:CA-1 for greater than 30 min"
+
+print("\nSTATE ALERTS - does the condition still hold?")
+a = al(ERROR_ALERT, openstack_id=OSID, status="ERROR", region="CANADA-1")
+a.screen = screening.screen(a, Snap(by_openstack_id={OSID: {"status": "ERROR"}}))
+expect("Infrahub still ERROR -> real", a.screen["verdict"] == screening.REAL, a.screen)
+
+a = al(ERROR_ALERT, openstack_id=OSID, status="ERROR", region="CANADA-1")
+a.screen = screening.screen(a, Snap(by_openstack_id={OSID: {"status": "ACTIVE"}}))
+expect("recovered to ACTIVE -> resolved",
+       a.screen["verdict"] == screening.RESOLVED and "ACTIVE" in a.screen["reason"], a.screen)
+
+a = al(SHUTOFF_ALERT, openstack_id=OSID, status="SHUTOFF")
+a.screen = screening.screen(a, Snap())
+expect("record gone from Infrahub -> resolved", a.screen["verdict"] == screening.RESOLVED, a.screen)
+
+a = al("Instance in CREATING state in :flag-ca:CA-1 for greater than 30min",
+       openstack_id="None", instance_name="vm-x", status="CREATING")
+a.screen = screening.screen(a, Snap())
+expect("CREATING with no OpenStack ID -> real, not 'resolved'",
+       a.screen["verdict"] == screening.REAL, a.screen)
+
+a = al(ERROR_ALERT, openstack_id=OSID)
+a.screen = screening.screen(a, Snap(by_openstack_id={OSID: {"status": "ERROR"}}))
+expect("no status label -> unverified but kept", a.screen["verdict"] == screening.UNVERIFIED, a.screen)
+
+print("\nDUPLICATED IPs")
+a = al(":awkward:Duplicated IPs", floating_ip="1.2.3.4")
+a.screen = screening.screen(a, Snap(fip_counts={"1.2.3.4": 1}))
+expect("one claimant left -> resolved", a.screen["verdict"] == screening.RESOLVED, a.screen)
+a.screen = screening.screen(a, Snap(fip_counts={"1.2.3.4": 3}))
+expect("three claimants -> real", a.screen["verdict"] == screening.REAL and "3 VMs" in a.screen["reason"], a.screen)
+a.screen = screening.screen(a, Snap(fip_counts={"9.9.9.9": 2}))
+expect("IP held by nobody -> resolved", a.screen["verdict"] == screening.RESOLVED, a.screen)
+
+print("\nSUSPECTED ROGUE VM - per-host GPU accounting gap")
+a = al(":ninja:Suspected Rogue VM", instance="CA1-ESC8-068")
+a.screen = screening.screen(a, Snap(rogue_delta={"CA1-ESC8-068": 0.0}))
+expect("gap closed -> resolved", a.screen["verdict"] == screening.RESOLVED, a.screen)
+a.screen = screening.screen(a, Snap(rogue_delta={"CA1-ESC8-068": 4.0},
+                                    resources_by_host={"CA1-ESC8-068": [{}] * 4}))
+expect("gap of 4 GPUs -> real", a.screen["verdict"] == screening.REAL, a.screen)
+expect("reason quantifies the gap", "4 GPU(s)" in a.screen["reason"], a.screen["reason"])
+a.screen = screening.screen(a, Snap(rogue_delta={"other-host": 4.0}, total_gpus={"x": 8}))
+expect("no data for the host -> unverified, still actionable",
+       a.screen["verdict"] == screening.UNVERIFIED and a.screen["actionable"], a.screen)
+
+print("\nTOTAL GPUs")
+a = al("Problem with Total GPUs in a System", instance="h1", gpu_name="B200-SXM")
+a.screen = screening.screen(a, Snap(total_gpus={"h1": 8}))
+expect("full complement of 8 -> resolved", a.screen["verdict"] == screening.RESOLVED, a.screen)
+a.screen = screening.screen(a, Snap(total_gpus={"h1": 6}, in_use_gpus={"h1": 6}))
+expect("6 GPUs -> real", a.screen["verdict"] == screening.REAL, a.screen)
+
+print("\nSUPPRESSION RULES - what used to be hardcoded is now user-editable")
+from triagelib import settings as settings_mod
+import tempfile, os as _os
+
+_tmp = _os.path.join(tempfile.mkdtemp(), "settings.json")
+CFG = settings_mod.Settings(_tmp)
+
+a = al(SHUTOFF_ALERT, openstack_id=OSID, status="SHUTOFF",
+       organization="3491 - luis.sarabando+runpod@nexgencloud.coms-Organization")
+a.screen = screening.screen(a, Snap(by_openstack_id={OSID: {"status": "SHUTOFF"}}), CFG)
+expect("default rule hides internal nexgencloud orgs",
+       a.screen["verdict"] == screening.SUPPRESSED, a.screen)
+expect("suppression names the rule that did it", "Internal NexGen" in a.screen["reason"], a.screen["reason"])
+
+a = al("Instance in SHUTOFF state in :flag-no:NO-1 for greater than 30 min", openstack_id=OSID,
+       status="SHUTOFF", instance="no1-stor-runpod03", instance_name="no1-stor-runpod03",
+       organization="99 - Real Customer")
+a.screen = screening.screen(a, Snap(by_openstack_id={OSID: {"status": "SHUTOFF"}}), CFG)
+expect("default rule hides runpod storage nodes", a.screen["verdict"] == screening.SUPPRESSED, a.screen)
+
+# The combinational case the team asked for: type AND organisation.
+CFG.upsert_rule({"name": "Modal ERROR churn", "reason": "known batch churn",
+                 "conditions": {"kind": ["error"], "organization": ["modal"]}})
+hit = al(ERROR_ALERT, openstack_id=OSID, status="ERROR", organization="19417 - colin@modal.coms-Organization")
+hit.screen = screening.screen(hit, Snap(by_openstack_id={OSID: {"status": "ERROR"}}), CFG)
+expect("error + modal is suppressed", hit.screen["verdict"] == screening.SUPPRESSED, hit.screen)
+
+miss = al(ERROR_ALERT, openstack_id=OSID, status="ERROR", organization="123 - Someone Else")
+miss.screen = screening.screen(miss, Snap(by_openstack_id={OSID: {"status": "ERROR"}}), CFG)
+expect("error from another org is NOT suppressed", miss.screen["verdict"] != screening.SUPPRESSED, miss.screen)
+
+other = al(SHUTOFF_ALERT, openstack_id=OSID, status="SHUTOFF",
+           organization="19417 - colin@modal.coms-Organization")
+other.screen = screening.screen(other, Snap(by_openstack_id={OSID: {"status": "SHUTOFF"}}), CFG)
+expect("modal SHUTOFF is NOT suppressed - both conditions must match",
+       other.screen["verdict"] != screening.SUPPRESSED, other.screen)
+
+empty = {"name": "catch all", "conditions": {}}
+expect("a rule with no conditions never matches", not settings_mod.rule_matches(
+    settings_mod._normalize_rule(empty), hit))
+
+expect("rules survive a reload", settings_mod.Settings(_tmp).rules and any(
+    r["name"] == "Modal ERROR churn" for r in settings_mod.Settings(_tmp).rules))
+
+print("\nAGE DEMOTIONS")
+a = aged(al("Problem with Total GPUs in a System", instance="h1"), 7 * 24 * 60)
+a.screen = screening.screen(a, Snap(total_gpus={"h1": 6}))
+expect("firing 7 days -> chronic", a.screen["verdict"] == screening.CHRONIC, a.screen)
+
+a = aged(al("Problem with Total GPUs in a System", instance="h1"), 60)
+a.screen = screening.screen(a, Snap(total_gpus={"h1": 6}))
+expect("firing 1 hour -> stays real", a.screen["verdict"] == screening.REAL, a.screen)
+
+print("\nFAIL-SAFE BEHAVIOUR")
+a = al(ERROR_ALERT, openstack_id=OSID, status="ERROR")
+a.state = "pending"
+a.for_seconds = 1800
+a.screen = screening.screen(a, Snap(by_openstack_id={OSID: {"status": "ERROR"}}))
+expect("pending -> screened out", a.screen["verdict"] == screening.PENDING and not a.screen["actionable"], a.screen)
+
+a = al(ERROR_ALERT, openstack_id=OSID, status="ERROR")
+a.screen = screening.screen(a, None)
+expect("no snapshot -> unverified but NOT hidden",
+       a.screen["verdict"] == screening.UNVERIFIED and a.screen["actionable"], a.screen)
+
+warnings = screening.health_warnings(Snap(broken_inputs=["openstack_nova_server_status"]))
+expect("empty nova metric raises a monitoring warning",
+       len(warnings) == 1 and "openstack_nova_server_status" in warnings[0], warnings)
+
+print("\nEXCLUSION AND TAB ROUTING")
+ex = al("Exists in Infrahub but does not exist in OpenStack", openstack_id=OSID)
+expect("orphan spam excluded outright", A.is_excluded(ex) and not A.cx_relevant(ex))
+expect("node-exporter -> Infrastructure tab",
+       A.category("HostSwapIsFillingUp", "node-exporter-rules.yml") == "node")
+expect("ceph -> Infrastructure tab", A.category("CephOsdDown", "ceph-rules.yml") == "infra")
+expect("regional Infrahub rule -> CX tab",
+       A.category(ERROR_ALERT, "infrahub-rules-CA1.yml") == "cx")
+expect("main Infrahub rule -> CX tab",
+       A.category(":ninja:Suspected Rogue VM", "infrahub-rules.yml") == "cx")
+expect("status-mismatch rule classified",
+       A.classify("Openstack status=ACTIVE and Infrahub status!=ACTIVE in :flag-ca:CANADA-1 for greater "
+                  "than 30min") == "status_mismatch")
+expect("hibernation-failure rule maps to HIBERNATING",
+       A.classify("Failure of Hibernation on Infrahub in :flag-ca:CANADA-1 for greater than 30min") == "hibernating")
+expect("orphan VM rule classified", A.classify(":pirate_flag:Suspected Orphan VM") == "orphan_vm")
+
+print("\nORDERING AND GROUPING")
+
+
+def real(minutes, name=ERROR_ALERT, **labels):
+    x = aged(al(name, openstack_id="o%d" % minutes, status="ERROR", **labels), minutes)
+    x.screen = {"actionable": True, "verdict": "real", "label": "needs action", "reason": ""}
+    return x
+
+
+ages = [x["age_minutes"] for x in A.group_alerts([real(500), real(10), real(100), real(9331)])[0]["alerts"]]
+expect("newest first, oldest at the bottom", ages == [10, 100, 500, 9331], ages)
+
+unknown = real(50)
+unknown.active_at = None
+ages = [x["age_minutes"] for x in A.group_alerts([unknown, real(200), real(5)])[0]["alerts"]]
+expect("unknown start time sorts last", ages == [5, 200, None], ages)
+
+groups = A.group_alerts([real(5), real(6, ":ninja:Suspected Rogue VM", instance="h1")])
+expect("focus order puts rogue VM before ERROR", [g["kind"] for g in groups][0] == "rogue_vm",
+       [g["kind"] for g in groups])
+
+noisy = real(7)
+noisy.screen = {"actionable": False, "verdict": "resolved", "label": "already resolved", "reason": ""}
+group = A.group_alerts([real(5), noisy])[0]
+expect("group counts action vs noise separately",
+       group["actionable"] == 1 and group["noise"] == 1, group)
+
+expect("age_text renders days", real(9331).age_text == "6d 11h", real(9331).age_text)
+expect("age_text renders hours", real(431).age_text == "7h 11m", real(431).age_text)
+expect("age_text renders minutes", real(7).age_text == "7m", real(7).age_text)
+
+print("\nTRUE AGE - activeAt reset by pipeline dips")
+# activeAt says 7h; ALERTS history says 7 days. The true value must win.
+a = real(431)
+a.true_age_minutes, a.true_age_capped = 7 * 24 * 60, False
+expect("effective age prefers the recovered duration", a.effective_age_minutes == 10080, a.effective_age_minutes)
+expect("reset is detected", a.age_is_reset)
+expect("raw activeAt still reported", a.age_text == "7h 11m", a.age_text)
+expect("effective text renders days", a.effective_age_text == "7d", a.effective_age_text)
+a.screen = screening.screen(a, Snap(by_openstack_id={"o431": {"status": "ERROR"}}))
+expect("7-day ERROR -> overdue (runbook says contact within 24h), not chronic",
+       a.screen["verdict"] == screening.OVERDUE, a.screen)
+expect("overdue stays in the actionable queue", a.screen["actionable"])
+
+# A kind with no runbook SLA still demotes to chronic, and explains the reset.
+g = al("Problem with Total GPUs in a System", instance="h9")
+g.active_at = dt.datetime.now(dt.timezone.utc) - dt.timedelta(minutes=431)
+g.true_age_minutes, g.true_age_capped = 7 * 24 * 60, False
+g.screen = screening.screen(g, Snap(total_gpus={"h9": 6}))
+expect("no-SLA kind, 7 days -> chronic", g.screen["verdict"] == screening.CHRONIC, g.screen)
+expect("chronic reason explains the activeAt reset", "pipeline dip" in g.screen["detail"], g.screen["detail"])
+
+print("\nVALIDATION FINDINGS - rogue VM rule defect")
+r = al(":ninja:Suspected Rogue VM", instance="CA1-ESC8-057")
+r.screen = screening.screen(r, Snap(rogue_delta={"CA1-ESC8-057": 1.0},
+                                    in_use_gpus={"CA1-ESC8-057": 8.0},
+                                    total_gpus={"CA1-ESC8-057": 8.0},
+                                    resources_by_host={"CA1-ESC8-057": [{}] * 5}))
+expect("In_Use == Total -> rule defect, not a rogue VM",
+       r.screen["verdict"] == screening.RULE_DEFECT, r.screen)
+expect("rule defect is screened out of the queue", not r.screen["actionable"])
+expect("reason names it as spare capacity", "free GPU" in r.screen["reason"], r.screen["reason"])
+
+r2 = al(":ninja:Suspected Rogue VM", instance="CA1-ESC812-289")
+r2.screen = screening.screen(r2, Snap(rogue_delta={"CA1-ESC812-289": 2.0},
+                                      in_use_gpus={"CA1-ESC812-289": 7.0},
+                                      resources_by_host={"CA1-ESC812-289": [{}] * 4}))
+expect("In_Use with no Total reading -> still a real gap",
+       r2.screen["verdict"] == screening.REAL, r2.screen)
+
+r3 = al(":ninja:Suspected Rogue VM", instance="h3")
+r3.screen = screening.screen(r3, Snap(rogue_delta={"h3": 3.0}, in_use_gpus={"h3": 9.0},
+                                      total_gpus={"h3": 8.0}, resources_by_host={"h3": [{}]}))
+expect("In_Use != Total -> real gap", r3.screen["verdict"] == screening.REAL, r3.screen)
+
+b = real(431)
+b.true_age_minutes, b.true_age_capped = 7 * 24 * 60, True
+expect("window-capped age marked with +", b.effective_age_text == "7d+", b.effective_age_text)
+
+c = real(120)
+c.true_age_minutes, c.true_age_capped = 130, False
+expect("small drift is not flagged as a reset", not c.age_is_reset)
+expect("no true age -> falls back to activeAt", real(90).effective_age_minutes == 90)
+
+# Ordering must use the recovered duration, not activeAt.
+old, new = real(431), real(430)
+old.true_age_minutes = 7 * 24 * 60
+new.true_age_minutes = 30
+order = [x["true_age_minutes"] for x in A.group_alerts([old, new])[0]["alerts"]]
+expect("true age drives ordering, not activeAt", order == [30, 10080], order)
+
+dips = screening.health_warnings(Snap(pipeline_dips=[
+    {"start": 0, "end": __import__("time").time() - 600, "minutes": 6, "low": 1359, "normal": 4374}]))
+expect("pipeline dip raises a warning", len(dips) == 1 and "1359 of ~4374" in dips[0], dips)
+
+print("\n" + ("ALL CHECKS PASSED" if not FAILS else f"{len(FAILS)} CHECK(S) FAILED: {FAILS}"))
+sys.exit(1 if FAILS else 0)

backend/triagelib/__init__.py

@@ -0,0 +1,4 @@
+"""CX Triage: read-only alert diagnosis on top of the CX-Tools (vmc) collectors."""
+from __future__ import annotations
+
+VERSION = "cx-triage 0.1"

backend/triagelib/alerts.py

@@ -0,0 +1,421 @@
+"""Normalizes Prometheus alerts into the alert kinds the CX runbooks cover."""
+from __future__ import annotations
+
+import datetime as dt
+import hashlib
+import re
+from dataclasses import dataclass, field
+from typing import Any, Optional
+
+EMOJI_RE = re.compile(r":[a-z0-9_+\-]+:")
+THRESHOLD_RE = re.compile(r"greater than\s*(\d+)\s*min", re.I)
+ORG_RE = re.compile(r"^\s*(\d+)\s*-\s*(.*)$")
+NONE_VALUES = {"", "none", "null", "unknown", "n/a"}
+
+# Alerts excluded outright. "Exists in Infrahub but does not exist in OpenStack"
+# is built as `Resources unless on(openstack_id) openstack_nova_server_status`,
+# and that right-hand metric is currently empty - so every Infrahub VM matches
+# and the alert fires thousands of times. It is a monitoring fault, not a queue
+# of work, so it never reaches the UI.
+EXCLUDED_ALERTNAMES = frozenset({
+    "Exists in Infrahub but does not exist in OpenStack",
+})
+
+# Rule files whose alerts belong to CX. Everything else is infrastructure.
+CX_RULE_FILES = ("infrahub-rules",)
+NODE_RULE_FILE = "node-exporter-rules.yml"
+
+# The order CX wants to work the queue in.
+FOCUS_ORDER = (
+    "rogue_vm", "duplicate_ip", "total_gpus", "hibernating",
+    "creating", "shutoff", "deleting", "error",
+    "restoring", "rebooting", "build", "orphan_vm", "status_mismatch",
+)
+
+# Priority and estimated time to resolve, from the "Infrahub Errors
+# Remediation" alert-conditions and runbook tables.
+KIND_META: dict[str, dict[str, str]] = {
+    "error":       {"title": "Instance in ERROR state",           "priority": "LOW-HIGH", "ettr": "5-30 min", "delay": "none"},
+    "deleting":    {"title": "Instance in DELETING state",        "priority": "LOW",      "ettr": "5-15 min", "delay": "30 min"},
+    "shutoff":     {"title": "Instance in SHUTOFF state",         "priority": "LOW",      "ettr": "5-10 min", "delay": "30 min"},
+    "hibernating": {"title": "Instance in HIBERNATING state",     "priority": "HIGH",     "ettr": "5-30 min", "delay": "30 min"},
+    "creating":    {"title": "Instance in CREATING state",        "priority": "MEDIUM",   "ettr": "5-15 min", "delay": "30 min"},
+    "restoring":   {"title": "Instance in RESTORING state",       "priority": "HIGH",     "ettr": "5-15 min", "delay": "30 min"},
+    "rebooting":   {"title": "Instance in REBOOTING state",       "priority": "HIGH",     "ettr": "5-15 min", "delay": "30 min"},
+    "build":       {"title": "Instance in BUILD state",           "priority": "MEDIUM",   "ettr": "5-15 min", "delay": "30 min"},
+    "rogue_vm":    {"title": "Suspected Rogue VM",                "priority": "HIGH",     "ettr": "5-30 min", "delay": "4 hours"},
+    "duplicate_ip": {"title": "Duplicated IPs",                   "priority": "HIGH",     "ettr": "5-15 min", "delay": "10 min"},
+    "total_gpus":  {"title": "Problem with Total GPUs in a System", "priority": "HIGH",   "ettr": "5-15 min", "delay": "5 min"},
+    "orphan_vm":   {"title": "Suspected Orphan VM",                 "priority": "HIGH",   "ettr": "5-30 min", "delay": "4 hours"},
+    "status_mismatch": {"title": "Infrahub/OpenStack status mismatch", "priority": "HIGH", "ettr": "5-30 min", "delay": "30 min"},
+}
+
+STATE_KINDS = ("error", "deleting", "shutoff", "hibernating", "creating", "restoring", "rebooting", "build")
+
+
+def clean_alertname(name: str) -> str:
+    """Strip the Slack emoji shortcodes Prometheus embeds in alert names."""
+    return EMOJI_RE.sub("", str(name or "")).strip()
+
+
+def classify(alertname: str) -> str:
+    name = clean_alertname(alertname).lower()
+    if alertname in EXCLUDED_ALERTNAMES or clean_alertname(alertname) in EXCLUDED_ALERTNAMES:
+        return "excluded"
+    if "rogue vm" in name:
+        return "rogue_vm"
+    if "orphan vm" in name:
+        return "orphan_vm"
+    if "duplicated ip" in name or "duplicate ip" in name:
+        return "duplicate_ip"
+    if "total gpus" in name:
+        return "total_gpus"
+    match = re.search(r"instance in (\w+) state", name)
+    if match:
+        state = match.group(1).lower()
+        if state in STATE_KINDS:
+            return state
+    # The per-region cross-check rules, e.g.
+    # "Openstack status=SHUTOFF and Infrahub status!=SHUTOFF in CANADA-1".
+    if "failure of hibernation" in name:
+        return "hibernating"
+    if "openstack status=" in name and "infrahub status" in name:
+        return "status_mismatch"
+    return "other"
+
+
+def category(alertname: str, rule_file: str = "") -> str:
+    """Which tab an alert belongs in: 'cx', 'node', or 'infra'."""
+    if any(token in rule_file for token in CX_RULE_FILES):
+        return "cx"
+    if rule_file == NODE_RULE_FILE:
+        return "node"
+    if rule_file:
+        return "infra"
+    # No rule metadata (e.g. a pasted alert): fall back to the classifier.
+    return "cx" if classify(alertname) not in ("other", "excluded") else "infra"
+
+
+def _clean(value: Any) -> str:
+    text = str(value or "").strip()
+    return "" if text.lower() in NONE_VALUES else text
+
+
+def split_organization(value: str) -> tuple[str, str]:
+    """Split the `organization` label ("8463 - Some Org") into id and name."""
+    match = ORG_RE.match(str(value or ""))
+    if match:
+        return match.group(1), match.group(2).strip()
+    return "", _clean(value)
+
+
+def _parse_active_at(value: Any) -> Optional[dt.datetime]:
+    text = str(value or "").strip()
+    if not text:
+        return None
+    text = re.sub(r"(\.\d{1,6})\d*Z?$", r"\1", text.replace("Z", "+00:00"))
+    if text.endswith("+00:00") is False and "+" not in text[10:]:
+        text += "+00:00"
+    try:
+        return dt.datetime.fromisoformat(text)
+    except ValueError:
+        return None
+
+
+@dataclass
+class Alert:
+    """One normalized Prometheus alert."""
+
+    kind: str
+    alertname: str
+    labels: dict[str, str] = field(default_factory=dict)
+    annotations: dict[str, str] = field(default_factory=dict)
+    state: str = "firing"
+    active_at: Optional[dt.datetime] = None
+
+    # Fields the runbooks key off.
+    openstack_id: str = ""
+    instance_name: str = ""
+    host: str = ""
+    region_label: str = ""
+    region: str = ""
+    status: str = ""
+    floating_ip: str = ""
+    flavor_name: str = ""
+    flavor_gpu: str = ""
+    org_id: str = ""
+    org_name: str = ""
+    contract_id: str = ""
+    gpu_name: str = ""
+    threshold_min: Optional[int] = None
+
+    # Where the rule came from (from RuleIndex), and the screening result.
+    rule_file: str = ""
+    rule_group: str = ""
+    for_seconds: int = 0
+    category: str = "cx"
+    screen: dict[str, Any] = field(default_factory=dict)
+
+    # How long the condition has actually held, recovered from ALERTS history.
+    # activeAt alone is unreliable: a metric-pipeline dip resets it on every
+    # live alert at once, which is why raw ages cluster on one timestamp.
+    true_age_minutes: Optional[int] = None
+    true_age_capped: bool = False
+
+    @property
+    def title(self) -> str:
+        return KIND_META.get(self.kind, {}).get("title", clean_alertname(self.alertname))
+
+    @property
+    def priority(self) -> str:
+        return KIND_META.get(self.kind, {}).get("priority", "UNKNOWN")
+
+    @property
+    def ettr(self) -> str:
+        return KIND_META.get(self.kind, {}).get("ettr", "unknown")
+
+    @property
+    def is_kubernetes(self) -> bool:
+        """Per the general process: kube* instance names are likely K8s nodes."""
+        return self.instance_name.lower().startswith("kube") or "-minion-" in self.instance_name.lower()
+
+    @property
+    def age_minutes(self) -> Optional[int]:
+        if not self.active_at:
+            return None
+        now = dt.datetime.now(dt.timezone.utc)
+        return max(0, int((now - self.active_at).total_seconds() // 60))
+
+    @staticmethod
+    def _duration_text(minutes: Optional[int]) -> str:
+        if minutes is None:
+            return "unknown"
+        if minutes < 60:
+            return f"{minutes}m"
+        hours, mins = divmod(minutes, 60)
+        if hours < 24:
+            return f"{hours}h {mins}m" if mins else f"{hours}h"
+        days, hours = divmod(hours, 24)
+        return f"{days}d {hours}h" if hours else f"{days}d"
+
+    @property
+    def age_text(self) -> str:
+        """Raw Prometheus activeAt duration."""
+        return self._duration_text(self.age_minutes)
+
+    @property
+    def effective_age_minutes(self) -> Optional[int]:
+        """True condition duration where known, else the raw activeAt age."""
+        return self.true_age_minutes if self.true_age_minutes is not None else self.age_minutes
+
+    @property
+    def effective_age_text(self) -> str:
+        text = self._duration_text(self.effective_age_minutes)
+        if self.true_age_minutes is not None and self.true_age_capped:
+            return f"{text}+"
+        return text
+
+    @property
+    def age_is_reset(self) -> bool:
+        """True when activeAt materially understates how long this has held."""
+        if self.true_age_minutes is None or self.age_minutes is None:
+            return False
+        return self.true_age_minutes - self.age_minutes > 60
+
+    @property
+    def is_internal_org(self) -> bool:
+        """Internal/test organizations are not customer-impacting."""
+        return "nexgencloud.com" in self.org_name.lower()
+
+    @property
+    def is_infra_owned(self) -> bool:
+        """Platform-owned nodes (storage etc.) name themselves after their host."""
+        return bool(self.instance_name) and self.instance_name.lower() == self.host.lower()
+
+    def fingerprint(self) -> str:
+        basis = "|".join([
+            self.kind,
+            self.openstack_id or self.instance_name or "",
+            self.host,
+            self.floating_ip,
+            self.region,
+        ])
+        return hashlib.sha1(basis.encode()).hexdigest()[:16]
+
+    def to_json(self) -> dict[str, Any]:
+        return {
+            "id": self.fingerprint(),
+            "kind": self.kind,
+            "title": self.title,
+            "alertname": clean_alertname(self.alertname),
+            "raw_alertname": self.alertname,
+            "state": self.state,
+            "priority": self.priority,
+            "ettr": self.ettr,
+            "active_at": self.active_at.isoformat() if self.active_at else "",
+            "age_minutes": self.age_minutes,
+            "age_text": self.age_text,
+            "true_age_minutes": self.true_age_minutes,
+            "true_age_capped": self.true_age_capped,
+            "effective_age_minutes": self.effective_age_minutes,
+            "effective_age_text": self.effective_age_text,
+            "age_is_reset": self.age_is_reset,
+            "threshold_min": self.threshold_min,
+            "rule_file": self.rule_file,
+            "rule_group": self.rule_group,
+            "for_seconds": self.for_seconds,
+            "category": self.category,
+            "screen": self.screen,
+            "is_internal_org": self.is_internal_org,
+            "is_infra_owned": self.is_infra_owned,
+            "openstack_id": self.openstack_id,
+            "instance_name": self.instance_name,
+            "host": self.host,
+            "region": self.region,
+            "region_label": self.region_label,
+            "status": self.status,
+            "floating_ip": self.floating_ip,
+            "flavor_name": self.flavor_name,
+            "flavor_gpu": self.flavor_gpu,
+            "gpu_name": self.gpu_name,
+            "org_id": self.org_id,
+            "org_name": self.org_name,
+            "contract_id": self.contract_id,
+            "is_kubernetes": self.is_kubernetes,
+            "labels": self.labels,
+            "annotations": self.annotations,
+        }
+
+
+def map_region(region_label: str) -> str:
+    """CANADA-1 -> ca1.
+
+    Deliberately a local table rather than a call into CX-Tools: building the
+    alert queue must not touch cxlib, because constructing a CX-Tools Config
+    loads credentials and would pop a 1Password prompt just to list alerts.
+    Kept in sync with SUPPORTED_REGIONS in cxlib/constants.py.
+    """
+    fallback = {
+        "canada-1": "ca1", "canada-2": "ca2", "us-1": "us1", "norway-1": "no1",
+        "ca-1": "ca1", "ca-2": "ca2", "no-1": "no1",
+        "ca1": "ca1", "ca2": "ca2", "us1": "us1", "no1": "no1",
+    }
+    return fallback.get(str(region_label or "").strip().lower(), "")
+
+
+def from_labels(labels: dict[str, str], annotations: Optional[dict[str, str]] = None,
+                state: str = "firing", active_at: Any = None) -> Alert:
+    labels = {str(k): str(v) for k, v in (labels or {}).items()}
+    alertname = labels.get("alertname", "")
+    kind = classify(alertname)
+
+    region_label = _clean(labels.get("region"))
+    org_id, org_name = split_organization(labels.get("organization", ""))
+    threshold = THRESHOLD_RE.search(alertname)
+
+    # For host-scoped alerts the `instance` label is the hypervisor; for
+    # VM-scoped alerts it is the hypervisor too, or "Unknown" when the VM never
+    # landed on a host.
+    host = _clean(labels.get("instance"))
+
+    alert = Alert(
+        kind=kind,
+        alertname=alertname,
+        labels=labels,
+        annotations={str(k): str(v) for k, v in (annotations or {}).items()},
+        state=str(state or "firing"),
+        active_at=_parse_active_at(active_at),
+        openstack_id=_clean(labels.get("openstack_id")),
+        instance_name=_clean(labels.get("instance_name")),
+        host=host,
+        region_label=region_label,
+        region=map_region(region_label) or _infer_region_from_host(host),
+        status=_clean(labels.get("status")),
+        floating_ip=_clean(labels.get("floating_ip")),
+        flavor_name=_clean(labels.get("flavor_name")),
+        flavor_gpu=_clean(labels.get("flavor_gpu")),
+        gpu_name=_clean(labels.get("gpu_name")),
+        org_id=org_id,
+        org_name=org_name,
+        contract_id=_clean(labels.get("contract_id")),
+        threshold_min=int(threshold.group(1)) if threshold else None,
+    )
+    return alert
+
+
+def _infer_region_from_host(host: str) -> str:
+    match = re.match(r"^(ca1|ca2|no1|us1)-", str(host or "").strip(), re.I)
+    return match.group(1).lower() if match else ""
+
+
+def from_prometheus(raw: dict[str, Any], rule_index: Any = None, true_age: Any = None) -> Alert:
+    alert = from_labels(
+        raw.get("labels") or {},
+        raw.get("annotations") or {},
+        state=str(raw.get("state") or "firing"),
+        active_at=raw.get("activeAt"),
+    )
+    meta = rule_index.get(alert.alertname) if rule_index is not None else {}
+    if meta:
+        alert.rule_file = str(meta.get("file") or "")
+        alert.rule_group = str(meta.get("group") or "")
+        alert.for_seconds = int(meta.get("for_seconds") or 0)
+    alert.category = category(alert.alertname, alert.rule_file)
+    if true_age is not None:
+        alert.true_age_minutes, alert.true_age_capped = true_age.lookup(alert.labels)
+    return alert
+
+
+def cx_relevant(alert: Alert) -> bool:
+    """True for alert kinds the CX runbooks cover."""
+    return alert.kind in KIND_META and alert.kind != "excluded"
+
+
+def is_excluded(alert: Alert) -> bool:
+    return alert.kind == "excluded" or alert.alertname in EXCLUDED_ALERTNAMES
+
+
+def sort_key(alert: Alert) -> tuple:
+    """Newest first: a fresh alert is the one that still needs a decision.
+
+    Long-running alerts sink to the bottom - they are either chronic and
+    already ticketed, or noise nobody has silenced. Alerts with no known start
+    time sort last rather than jumping to the top.
+
+    Sorted on the true condition duration, not activeAt: a pipeline dip resets
+    activeAt on every live alert at once, which would otherwise flatten the
+    ordering into a single tie.
+    """
+    age = alert.effective_age_minutes
+    return (age if age is not None else 10**9, alert.title)
+
+
+def focus_rank(kind: str) -> int:
+    try:
+        return FOCUS_ORDER.index(kind)
+    except ValueError:
+        return len(FOCUS_ORDER)
+
+
+def group_alerts(items: list[Alert]) -> list[dict[str, Any]]:
+    """Group alerts into collapsible sections, in CX's working order."""
+    buckets: dict[str, list[Alert]] = {}
+    for alert in items:
+        buckets.setdefault(alert.kind, []).append(alert)
+
+    groups: list[dict[str, Any]] = []
+    for kind, members in buckets.items():
+        members.sort(key=sort_key)
+        actionable = [a for a in members if a.screen.get("actionable", True)]
+        groups.append({
+            "kind": kind,
+            "title": KIND_META.get(kind, {}).get("title", kind),
+            "priority": KIND_META.get(kind, {}).get("priority", "UNKNOWN"),
+            "ettr": KIND_META.get(kind, {}).get("ettr", "unknown"),
+            "total": len(members),
+            "actionable": len(actionable),
+            "noise": len(members) - len(actionable),
+            "alerts": [a.to_json() for a in members],
+        })
+    groups.sort(key=lambda g: (focus_rank(g["kind"]), -g["actionable"]))
+    return groups

backend/triagelib/comms.py

@@ -0,0 +1,383 @@
+"""Customer comms templates, transcribed from the CX runbooks.
+
+Wording is kept verbatim from Confluence so what CX sends stays consistent with
+the approved snippets; only the named placeholders are substituted. Nothing here
+sends anything - the app renders the draft for a human to review and send from
+HubSpot.
+"""
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Any, Optional
+
+INSTANCE_PLACEHOLDER = "INFRAHUB_INSTANCE_NAME"
+FIP_PLACEHOLDER = "NEW_INFRAHUB_FLOATING_IP"
+ID_PLACEHOLDER = "INFRAHUB_ID"
+OSID_PLACEHOLDER = "OPENSTACK_ID"
+NAME_PLACEHOLDER = "GREETING_NAME"
+AGENT_PLACEHOLDER = "AGENT_NAME"
+
+
+def first_name(owner: str) -> str:
+    """'Bojan Jovanovic <bojan@polycam.ai>' -> 'Bojan'.
+
+    Falls back to an empty greeting rather than guessing: the examples show
+    'Hello,' is acceptable, but 'Hello shettyatulya@gmail.com,' is not.
+    """
+    text = str(owner or "").split("<", 1)[0].strip()
+    if not text or "@" in text:
+        return ""
+    first = text.split()[0]
+    return first if first[:1].isalpha() else ""
+
+
+@dataclass
+class Draft:
+    template_id: str
+    label: str
+    subject: str
+    body: str
+    channel: str = "HubSpot ticket"
+    when: str = ""
+    unfilled: list[str] = field(default_factory=list)
+    source: str = ""
+
+    def to_json(self) -> dict[str, Any]:
+        return {
+            "template_id": self.template_id,
+            "label": self.label,
+            "subject": self.subject,
+            "body": self.body,
+            "channel": self.channel,
+            "when": self.when,
+            "unfilled": self.unfilled,
+            "source": self.source,
+        }
+
+
+# Wording follows the house style CX actually sends: first-name greeting, the VM
+# named with its Infrahub ID, an explicit "you will not be charged" line, the
+# billing-states link, and a personal sign-off. Placeholders are substituted;
+# everything else is left alone so what goes out stays consistent.
+BILLING_DOC = ("Here is our documentation on VM states and their cost:\n"
+               "Which virtual machine states incur billing costs?")
+STOCK_DOC = ("You can use our Stock API to check availability at the time of deploying a VM here - "
+             "Stock Availability")
+
+_TEMPLATES: dict[str, dict[str, str]] = {
+    "error_never_active": {
+        "label": "ERROR - never deployed (transient stock issue)",
+        "subject": "VM in Error state",
+        "when": "The instance never reached a host, so nothing was built. Recommend delete and retry.",
+        "source": "Instance in ERROR state",
+        "body": f"""Hello GREETING_NAME,
+
+We hope you are well.
+
+We are emailing you in regards to VM: INFRAHUB_INSTANCE_NAME (INFRAHUB_ID)
+
+Due to a transient stock issue this VM never fully deployed and is currently showing in Error.
+
+Our recommendation would be to delete the VM and try recreating a new one. Please be aware that whilst the VM is in an Error state you will not be charged for its usage.
+
+{BILLING_DOC}
+
+{STOCK_DOC}
+
+Just so you are aware, if the VM is not deleted after 14 calendar days we will proceed with deleting the VM on your behalf.
+
+Kind Regards,
+
+AGENT_NAME""",
+    },
+    "error_was_active": {
+        "label": "ERROR - VM had been running, escalated",
+        "subject": "VM in Error state",
+        "when": "The instance had reached ACTIVE, so customer data may be involved. Escalate first, then send.",
+        "source": "Instance in ERROR state",
+        "body": f"""Hello GREETING_NAME,
+
+We hope you are well.
+
+We are emailing you in regards to VM: INFRAHUB_INSTANCE_NAME (INFRAHUB_ID)
+
+We can see this VM has gone into an Error state. We have escalated this to the appropriate team on your behalf and will come back to you as soon as we have more information.
+
+Please be assured that whilst a VM is in an Error state you will not be charged for its usage.
+
+{BILLING_DOC}
+
+Kind Regards,
+
+AGENT_NAME""",
+    },
+    "creating": {
+        "label": "CREATING - stuck on deploy, VM deleted for the customer",
+        "subject": "VM stuck in creating state",
+        "when": "Send after the stuck instance has been deleted.",
+        "source": "Instance in CREATING state",
+        "body": f"""Hello GREETING_NAME,
+
+We hope you are well.
+
+We are emailing you in regards to VM INFRAHUB_INSTANCE_NAME (INFRAHUB_ID).
+
+We can see you tried to deploy this VM but due to a transient error the VM got stuck in a creating state.
+
+Unfortunately as the VM was not able to fully deploy, the safest option was to delete the VM which we have actioned for you.
+
+Please be assured that whilst a VM is in a creating state you will not be charged for it's usage.
+
+{BILLING_DOC}
+
+If you have any queries please let us know.
+
+Kind Regards,
+
+AGENT_NAME""",
+    },
+    "deleting": {
+        "label": "DELETING - stuck delete finalised for the customer",
+        "subject": "VM stuck in deleting state",
+        "when": "Send once the delete has actually been finalised.",
+        "source": "Instance in DELETING state",
+        "body": f"""Hello GREETING_NAME,
+
+We hope you are well.
+
+We are emailing you in regards to VM: INFRAHUB_INSTANCE_NAME (INFRAHUB_ID)
+
+We can see one of your users requested the deletion and that due to a transient error the VM got stuck in a deleting state. I wanted to make sure you are aware that we have gone and finalised the deletion for you.
+
+Please be assured that whilst a VM is in a deleting state you will not be charged for it's usage.
+
+{BILLING_DOC}
+
+Kind Regards,
+
+AGENT_NAME""",
+    },
+    "deleting_resolved": {
+        "label": "DELETING - confirming resolution and closing the ticket",
+        "subject": "VM deleted - closing this ticket",
+        "when": "Use when replying on an existing ticket that can now be closed.",
+        "source": "Instance in DELETING state",
+        "body": """Hello GREETING_NAME,
+
+Upon reviewing this ticket, we found that the VM below, which was previously stuck in a DELETING state, has now been deleted:
+
+* OPENSTACK_ID (INFRAHUB_ID)
+
+As the VM has been deleted, we are marking the issue as resolved and closing this ticket.
+
+If you require further assistance, please feel free to contact us at support@hyperstack.cloud or open a new Live Chat via the Hyperstack Console.
+
+Have a great rest of your day and thank you for using Hyperstack.
+
+Kind regards,
+
+AGENT_NAME""",
+    },
+    "build": {
+        "label": "BUILD - failed to build, escalated",
+        "subject": "VM stuck in build state",
+        "when": "Send once escalated to Infrastructure. The instance must be recreated.",
+        "source": "Instance in BUILD state",
+        "body": f"""Hello GREETING_NAME,
+
+We hope you are well.
+
+We are emailing you in regards to VM INFRAHUB_INSTANCE_NAME (INFRAHUB_ID).
+
+We can see you tried to deploy this VM but due to a transient error it did not finish building. We have escalated this on your behalf and ask that you retry creating the instance at your convenience.
+
+Please be assured that whilst a VM is in this state you will not be charged for it's usage.
+
+{BILLING_DOC}
+
+If you have any queries please let us know.
+
+Kind Regards,
+
+AGENT_NAME""",
+    },
+    "rebooting": {
+        "label": "REBOOTING - reboot failed, now resolved",
+        "subject": "VM stuck rebooting",
+        "when": "Send only once the instance is confirmed ACTIVE in both Infrahub and OpenStack.",
+        "source": "Instance in REBOOTING state",
+        "body": f"""Hello GREETING_NAME,
+
+We hope you are well.
+
+We are emailing you in regards to VM INFRAHUB_INSTANCE_NAME (INFRAHUB_ID).
+
+We can see a reboot was requested but due to a transient error the VM got stuck. This has now been resolved and you may retry rebooting at your convenience.
+
+{BILLING_DOC}
+
+If you have any queries please let us know.
+
+Kind Regards,
+
+AGENT_NAME""",
+    },
+    "restoring": {
+        "label": "RESTORING - restore failed, now resolved",
+        "subject": "VM stuck restoring",
+        "when": "Send after the instance is back to SHELVED_OFFLOADED in OpenStack and HIBERNATED in Infrahub.",
+        "source": "Instance in RESTORING state",
+        "body": f"""Hello GREETING_NAME,
+
+We hope you are well.
+
+We are emailing you in regards to VM INFRAHUB_INSTANCE_NAME (INFRAHUB_ID).
+
+We can see you tried to restore this VM but due to a transient error it got stuck. This has now been resolved and you may retry restoring at your convenience.
+
+{BILLING_DOC}
+
+If you have any queries please let us know.
+
+Kind Regards,
+
+AGENT_NAME""",
+    },
+    "shutoff": {
+        "label": "SHUTOFF - billing awareness notice",
+        "subject": "VM in SHUT-OFF state is still accruing costs",
+        "when": "Send as-is. A HubSpot snippet also exists: type #shutoff.",
+        "source": "Instance in SHUTOFF state",
+        "body": f"""Hello GREETING_NAME,
+
+We hope you are well.
+
+We are emailing you in regards to VM: INFRAHUB_INSTANCE_NAME (INFRAHUB_ID)
+
+We can see that this is in a SHUT-OFF state and wanted to make sure you are aware that in this state the VM is still accruing full costs.
+
+{BILLING_DOC}
+
+Kind Regards,
+
+AGENT_NAME""",
+    },
+    "dupip_removed": {
+        "label": "Duplicated IP - incorrect IP removed, customer must attach a new one",
+        "subject": "Instance assigned an incorrect public IP",
+        "when": "The VM has no floating IP in OpenStack and the stale IP was removed in InfraInsight.",
+        "source": "Duplicated IPs",
+        "body": """Hello GREETING_NAME,
+
+We hope you are well.
+
+We are emailing you in regards to VM: INFRAHUB_INSTANCE_NAME (INFRAHUB_ID)
+
+Due to a transient synchronisation issue this instance was showing an incorrect public IP. This has now been corrected. To restore external connectivity the instance will need a new public IP attached, which you can do at your convenience using either the Hyperstack UI or API.
+
+We apologise for any inconvenience this may have caused.
+
+Kind Regards,
+
+AGENT_NAME""",
+    },
+    "dupip_corrected": {
+        "label": "Duplicated IP - Infrahub corrected to match OpenStack",
+        "subject": "Instance assigned an incorrect public IP",
+        "when": "The VM does have a floating IP in OpenStack and Infrahub was corrected to match.",
+        "source": "Duplicated IPs",
+        "body": """Hello GREETING_NAME,
+
+We hope you are well.
+
+We are emailing you in regards to VM: INFRAHUB_INSTANCE_NAME (INFRAHUB_ID)
+
+Due to a transient synchronisation issue this instance was showing an incorrect public IP. This has now been resolved and your instance is reachable at NEW_INFRAHUB_FLOATING_IP.
+
+We apologise for any inconvenience this may have caused.
+
+Kind Regards,
+
+AGENT_NAME""",
+    },
+    "sync_state": {
+        "label": "Rogue VM - instance was in the incorrect state",
+        "subject": "VM was showing an incorrect state",
+        "when": "Send after the state mismatch has been remediated.",
+        "source": "Suspected Rogue VM",
+        "body": """Hello GREETING_NAME,
+
+We hope you are well.
+
+We are emailing you in regards to VM: INFRAHUB_INSTANCE_NAME (INFRAHUB_ID)
+
+Due to a transient sync error this instance was showing an incorrect state. This has since been resolved, and we ask that you retry any operations that failed as a result.
+
+We apologise for any delay this may have caused.
+
+Kind Regards,
+
+AGENT_NAME""",
+    },
+}
+
+
+def draft(template_id: str, *, instance_name: str = "", floating_ip: str = "",
+          infrahub_id: str = "", openstack_id: str = "", greeting_name: str = "",
+          agent_name: str = "", note: Optional[str] = None) -> Optional[Draft]:
+    spec = _TEMPLATES.get(template_id)
+    if not spec:
+        return None
+    body = spec["body"]
+    unfilled: list[str] = []
+
+    # "Hello Bojan," when we know the name, "Hello," when we do not - never
+    # "Hello <placeholder>,".
+    body = body.replace(f"Hello {NAME_PLACEHOLDER},", f"Hello {greeting_name}," if greeting_name else "Hello,")
+
+    substitutions = {
+        INSTANCE_PLACEHOLDER: instance_name,
+        ID_PLACEHOLDER: infrahub_id,
+        OSID_PLACEHOLDER: openstack_id,
+        FIP_PLACEHOLDER: floating_ip,
+        AGENT_PLACEHOLDER: agent_name,
+    }
+    for placeholder, value in substitutions.items():
+        if placeholder not in body:
+            continue
+        if value and value not in ("N/A", "None"):
+            body = body.replace(placeholder, str(value))
+        else:
+            unfilled.append(placeholder)
+
+    # "(INFRAHUB_ID)" with nothing to put in it reads worse than no bracket.
+    if ID_PLACEHOLDER in unfilled:
+        body = body.replace(f" ({ID_PLACEHOLDER})", "").replace(f"({ID_PLACEHOLDER})", "")
+        unfilled.remove(ID_PLACEHOLDER)
+
+    when = spec["when"]
+    if note:
+        when = f"{when} {note}".strip()
+
+    return Draft(
+        template_id=template_id,
+        label=spec["label"],
+        subject=spec["subject"],
+        body=body,
+        when=when,
+        unfilled=unfilled,
+        source=spec["source"],
+    )
+
+
+def contacts_from_result(result: dict[str, Any]) -> dict[str, Any]:
+    """Pull the organization and owner contacts CX-Tools resolved for a VM."""
+    if not isinstance(result, dict):
+        return {"organization": "", "owners": [], "resolved": False}
+    org = str(result.get("org_value") or "").strip()
+    owners = [str(x) for x in (result.get("owners") or []) if str(x).strip()]
+    return {
+        "organization": org if org and org != "N/A" else "",
+        "owners": owners,
+        "resolved": bool(owners),
+    }

backend/triagelib/cxbridge.py

@@ -0,0 +1,286 @@
+"""Read-only adapter over the CX-Tools (vmc) collectors.
+
+CX-Tools is imported as an unmodified library: this module never writes to the
+CX-Tools tree and only calls collectors and query helpers that read. Every
+OpenStack subcommand this module can reach is checked against READ_ONLY_VERBS
+before it runs, so a bug here cannot mutate a live instance.
+"""
+from __future__ import annotations
+
+import os
+import sys
+import threading
+from typing import Any, Optional
+
+DEFAULT_CX_TOOLS_PATHS = (
+    os.environ.get("CX_TOOLS_PATH", ""),
+    os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "CX-Tools"),
+    os.path.expanduser("~/scripts/CX-Tools"),
+    os.path.expanduser("~/scripts/cx-tooling"),
+)
+
+# OpenStack verbs the diagnosis path is allowed to reach. Anything that could
+# change state (create/delete/set/unset/shelve/reboot/...) is absent on purpose.
+READ_ONLY_VERBS = frozenset({"show", "list"})
+
+
+class BridgeError(RuntimeError):
+    """Raised when CX-Tools cannot be located, imported, or authenticated."""
+
+
+def locate_cx_tools() -> str:
+    for candidate in DEFAULT_CX_TOOLS_PATHS:
+        if not candidate:
+            continue
+        path = os.path.abspath(os.path.expanduser(candidate))
+        if os.path.isfile(os.path.join(path, "cxlib", "__init__.py")):
+            return path
+    raise BridgeError(
+        "Could not find the CX-Tools checkout. Set CX_TOOLS_PATH to the directory "
+        "that contains cxlib/ and the vmc entry point."
+    )
+
+
+_lock = threading.Lock()
+_state: dict[str, Any] = {"path": "", "cx": None, "config": None}
+
+
+def _import_cxlib(path: str):
+    if path not in sys.path:
+        sys.path.insert(0, path)
+    try:
+        import cxlib  # noqa: PLC0415  (import is intentionally deferred)
+    except Exception as exc:  # pragma: no cover - depends on the local checkout
+        raise BridgeError(f"Failed to import cxlib from {path}: {exc}") from exc
+    return cxlib
+
+
+def bootstrap() -> tuple[Any, Any]:
+    """Import cxlib and build the shared Config, loading secrets exactly once.
+
+    Call this from the foreground at startup: constructing Config triggers the
+    CX-Tools 1Password loader, which may need an interactive sign-in.
+    """
+    with _lock:
+        if _state["config"] is not None:
+            return _state["cx"], _state["config"]
+        path = locate_cx_tools()
+        cx = _import_cxlib(path)
+        config = cx.Config(no_color=True, debug=bool(os.environ.get("CX_DEBUG")))
+        if not config.api_key or config.api_key in {"REDACT", "REPLACE_WITH_API_KEY"}:
+            raise BridgeError(
+                "CX-Tools could not load the Infrahub API key from 1Password. "
+                "Run `op signin` in this shell, then restart cx-triage."
+            )
+        _state.update({"path": path, "cx": cx, "config": config})
+        return cx, config
+
+
+def cx() -> Any:
+    return bootstrap()[0]
+
+
+def config() -> Any:
+    return bootstrap()[1]
+
+
+def cx_tools_path() -> str:
+    bootstrap()
+    return str(_state["path"])
+
+
+def quiet_progress() -> Any:
+    """A Progress object that renders nothing, for use off the terminal."""
+    c = cx()
+    return c.Progress(c.C(False), 1, enabled=False)
+
+
+def _guard_openstack(args: list[str]) -> None:
+    verbs = [a for a in args if not str(a).startswith("-")]
+    if not any(v in READ_ONLY_VERBS for v in verbs):
+        raise BridgeError(f"Refusing to run a non-read-only OpenStack command: {' '.join(args)}")
+
+
+def os_json(region: str, args: list[str], timeout: int = 90) -> tuple[bool, Any, str]:
+    """Run a read-only `openstack ... -f json` command through CX-Tools."""
+    _guard_openstack(args)
+    return cx().os_json(config(), region, args, timeout=timeout)
+
+
+# --- collectors -------------------------------------------------------------
+
+def collect_vm(target: str, *, region: str = "", org_id: Optional[str] = None, ssh_timeout: int = 3) -> dict[str, Any]:
+    """Full VM reconciliation: the same payload `vmc --json <target>` emits."""
+    return cx().collect_vm(
+        config(),
+        target,
+        region_arg=region or "",
+        org_id=org_id,
+        ssh_timeout=ssh_timeout,
+        include_ih_events=True,
+        include_volumes=True,
+        include_all_ih_events=True,
+        progress=quiet_progress(),
+    )
+
+
+def collect_host(host: str, *, ssh_timeout: int = 3) -> dict[str, Any]:
+    """Host reconciliation: the same payload `vmc --json --host <host>` emits."""
+    return cx().collect_host(
+        config(),
+        host,
+        ssh_timeout=ssh_timeout,
+        include_ih_events=True,
+        include_volumes=True,
+        progress=quiet_progress(),
+    )
+
+
+def collect_vm_contacts(target: str, *, region: str = "", org_id: Optional[str] = None) -> dict[str, Any]:
+    return cx().collect_vm_contacts(
+        config(),
+        target,
+        region_arg=region or "",
+        org_id=org_id,
+        progress=quiet_progress(),
+    )
+
+
+# --- targeted queries used by individual runbooks --------------------------
+
+def openstack_events(region: str, openstack_id: str, limit: Optional[int] = 5) -> list[dict[str, Any]]:
+    ok, events, _raw = cx().server_event_list(config(), region, openstack_id)
+    if not ok:
+        return []
+    return events[:limit] if limit else events
+
+
+def openstack_event_detail(region: str, openstack_id: str, request_id: str) -> dict[str, Any]:
+    ok, detail, _raw = cx().server_event_show(config(), region, openstack_id, request_id)
+    return detail if ok else {}
+
+
+def failed_openstack_event(region: str, openstack_id: str, scan: int = 5) -> dict[str, Any]:
+    """Return the most recent OpenStack event whose detail reports a failure.
+
+    The state runbooks all say "the most recent failed event is the thing to
+    escalate", so this walks recent events newest-first and returns the first
+    one whose result is not Success, together with its detail rows.
+    """
+    c = cx()
+    for event in openstack_events(region, openstack_id, limit=scan):
+        request_id = c.event_request_id(event)
+        if not request_id:
+            continue
+        detail = openstack_event_detail(region, openstack_id, request_id)
+        if not detail:
+            continue
+        rows = dict((str(k), str(v)) for k, v in c.event_detail_rows(detail))
+        result = rows.get("Result", "")
+        if result and result.lower() != "success":
+            return {"request_id": request_id, "action": rows.get("Action", ""), "rows": rows}
+    return {}
+
+
+def infrahub_events(infrahub_id: str, limit: Optional[int] = None) -> list[list[str]]:
+    c = cx()
+    ok, data, _raw = c.query_vm_events(config(), str(infrahub_id))
+    if not ok:
+        return []
+    return c.infrahub_event_rows(data, limit)
+
+
+def host_health(region: str, host: str) -> dict[str, Any]:
+    """Hypervisor, Nova service and OVS agent signals for one host.
+
+    This is the cheap subset of `vmc --host` - the runbooks' "Host Health
+    Checks" entry point - without collecting every instance on the host.
+    """
+    c = cx()
+    cfg = config()
+    ok_hv, hv, raw_hv, hv_name = c.hypervisor_show_host(cfg, region, host)
+    if not ok_hv:
+        return {"ok": False, "error": raw_hv, "host": host, "region": region}
+
+    state = c.normalize_empty(c.first_present(hv, "state", "State", default=""))
+    status = c.normalize_empty(c.first_present(hv, "status", "Status", default=""))
+
+    disabled_reason = ""
+    if status.lower() == "disabled":
+        for candidate in dict.fromkeys([x for x in (c.normalize_empty(hv_name), host) if x]):
+            ok_svc, services, _raw = c.compute_service_list_host(cfg, region, candidate)
+            if ok_svc:
+                disabled_reason = c.disabled_reason_from_services(services)
+                if disabled_reason:
+                    break
+
+    ovs: dict[str, Any] = {}
+    ok_agents, agents, _raw_agents = c.network_agent_list_host(cfg, region, host)
+    if ok_agents:
+        ovs = c.ovs_agent_summary(agents)
+        if ovs.get("agent_id"):
+            ok_show, detail, _raw_show = c.network_agent_show(cfg, region, str(ovs["agent_id"]))
+            if ok_show:
+                ovs["last_heartbeat_at"] = c.normalize_empty(
+                    detail.get("last_heartbeat_at") or detail.get("Last Heartbeat At") or ovs.get("last_heartbeat_at")
+                )
+
+    return {
+        "ok": True,
+        "error": "",
+        "host": host,
+        "hypervisor_name": hv_name,
+        "region": region,
+        "nova_state": state or "N/A",
+        "nova_status": status or "N/A",
+        "disabled_reason": disabled_reason,
+        "uptime": c.host_uptime_summary(hv),
+        "aggregates": c.host_aggregates_summary(hv),
+        "ovs_alive": ovs.get("alive"),
+        "ovs_state": ovs.get("state"),
+        "ovs_last_heartbeat": ovs.get("last_heartbeat_at") or "",
+        "running_vms": c.normalize_empty(c.first_present(hv, "running_vms", "Running VMs", default="")) or "N/A",
+        "free_disk_gb": c.normalize_empty(c.first_present(hv, "free_disk_gb", "Free Disk GB", default="")) or "N/A",
+        "local_disk_free": c.normalize_empty(c.first_present(hv, "disk_available_least", "Disk Available Least", default="")) or "N/A",
+    }
+
+
+def host_gpu_census(region: str, host: str) -> dict[str, Any]:
+    """Sum GPU counts of every instance on a host.
+
+    Implements the ERROR-runbook check for the NUMA/PCI fault: "check if the
+    host is full prior to escalation - add the values after the x, if it = 8
+    then it is FULL".
+    """
+    c = cx()
+    ok, rows, raw = c.server_list_on_host(config(), region, host)
+    if not ok:
+        return {"ok": False, "error": raw, "total_gpus": None, "instances": []}
+    total = 0
+    unknown = 0
+    instances: list[dict[str, str]] = []
+    for row in rows:
+        flavor = c.get_row_field(row, "Flavor", "flavor") or c.flavor_name_from_any(row)
+        count = c.gpu_count_from_flavor_name(flavor)
+        if count.isdigit():
+            total += int(count)
+        else:
+            unknown += 1
+        instances.append({
+            "name": c.get_row_field(row, "Name", "name") or "N/A",
+            "openstack_id": c.openstack_id_from_row(row) or "N/A",
+            "status": c.get_row_field(row, "Status", "status") or "N/A",
+            "flavor": flavor or "N/A",
+            "gpus": count,
+        })
+    return {
+        "ok": True,
+        "error": "",
+        "total_gpus": total,
+        "unknown_flavors": unknown,
+        "instances": instances,
+    }
+
+
+def json_safe(obj: Any) -> Any:
+    return cx().json_safe(obj)

backend/triagelib/integrations.py

@@ -0,0 +1,196 @@
+"""Outbound action payloads: Zendesk tickets and Jira issues.
+
+This module *builds* payloads and never sends them. Delivery is a separate,
+explicitly configured step - see `outbox.py` - so that a diagnosis can never
+contact a customer as a side effect of being viewed.
+
+Every payload carries the evidence that justified it, so the ticket a customer
+or the Infrastructure team receives is self-contained.
+"""
+from __future__ import annotations
+
+import os
+from dataclasses import dataclass, field
+from typing import Any, Optional
+
+# Set these to enable the Send buttons. Absent = preview only.
+ZENDESK_SUBDOMAIN = os.environ.get("CX_ZENDESK_SUBDOMAIN", "")
+ZENDESK_EMAIL = os.environ.get("CX_ZENDESK_EMAIL", "")
+ZENDESK_TOKEN = os.environ.get("CX_ZENDESK_TOKEN", "")
+JIRA_BASE = os.environ.get("CX_JIRA_BASE", "https://nexgencloud.atlassian.net")
+JIRA_EMAIL = os.environ.get("CX_JIRA_EMAIL", "")
+JIRA_TOKEN = os.environ.get("CX_JIRA_TOKEN", "")
+JIRA_PROJECT = os.environ.get("CX_JIRA_PROJECT", "INFRA")
+
+PRIORITY_BY_VERDICT = {"overdue": "high", "real": "normal", "unverified": "low"}
+
+
+def zendesk_configured() -> bool:
+    return bool(ZENDESK_SUBDOMAIN and ZENDESK_EMAIL and ZENDESK_TOKEN)
+
+
+def jira_configured() -> bool:
+    return bool(JIRA_BASE and JIRA_EMAIL and JIRA_TOKEN)
+
+
+@dataclass
+class Action:
+    """One proposed outbound action, ready to send once a human confirms."""
+
+    id: str
+    kind: str                     # zendesk | jira | manual
+    label: str
+    summary: str                  # one line: what this does
+    payload: dict[str, Any] = field(default_factory=dict)
+    recipients: list[str] = field(default_factory=list)
+    enabled: bool = False         # is the integration configured?
+    blocked_reason: str = ""
+    requires_confirmation: bool = True
+
+    def to_json(self) -> dict[str, Any]:
+        return {
+            "id": self.id, "kind": self.kind, "label": self.label, "summary": self.summary,
+            "payload": self.payload, "recipients": self.recipients, "enabled": self.enabled,
+            "blocked_reason": self.blocked_reason, "requires_confirmation": self.requires_confirmation,
+        }
+
+
+def _parse_owner(owner: str) -> tuple[str, str]:
+    """'Name <email@x>' -> ('Name', 'email@x')."""
+    text = str(owner or "").strip()
+    if "<" in text and ">" in text:
+        name = text.split("<", 1)[0].strip()
+        email = text.split("<", 1)[1].split(">", 1)[0].strip()
+        return name, email
+    return ("", text) if "@" in text else (text, "")
+
+
+def _evidence_block(diagnosis: Any) -> str:
+    lines = [f"Alert: {diagnosis.alert.title}", f"Verdict: {diagnosis.verdict}", ""]
+    for finding in diagnosis.findings[:16]:
+        lines.append(f"- {finding.label}: {finding.value}")
+    return "\n".join(lines)
+
+
+def build_zendesk(diagnosis: Any) -> Optional[Action]:
+    """A customer ticket, only when the runbook actually calls for contact."""
+    if not diagnosis.drafts:
+        return None
+    draft = diagnosis.drafts[0]
+    alert = diagnosis.alert
+    contacts = diagnosis.contacts or {}
+    owners = contacts.get("owners") or []
+    if not owners:
+        return Action(
+            id="zendesk", kind="zendesk", label="Contact customer (Zendesk)",
+            summary="No owner contact resolved from Infrahub - look the organization up first.",
+            enabled=False, blocked_reason="No customer contact could be resolved.",
+        )
+
+    name, email = _parse_owner(owners[0])
+    verdict = (alert.screen or {}).get("verdict", "real")
+    payload = {
+        "ticket": {
+            "subject": draft.subject,
+            "comment": {"body": draft.body, "public": True},
+            "requester": {"name": name or email, "email": email},
+            "priority": PRIORITY_BY_VERDICT.get(verdict, "normal"),
+            "type": "incident",
+            "tags": ["cx-triage", f"alert-{alert.kind}", f"region-{alert.region or 'unknown'}"],
+            "external_id": f"cx-triage-{alert.fingerprint()}",
+            "custom_fields_note": {
+                "instance_name": alert.instance_name,
+                "openstack_id": alert.openstack_id,
+                "organization": alert.org_name,
+                "infrahub_org_id": alert.org_id,
+            },
+        },
+        "_template": draft.template_id,
+        "_when": draft.when,
+        "_unfilled": draft.unfilled,
+    }
+    blocked = ""
+    if draft.unfilled:
+        blocked = f"Template still has placeholders: {', '.join(draft.unfilled)}"
+    return Action(
+        id="zendesk", kind="zendesk",
+        label="Contact customer (Zendesk)",
+        summary=f"Public reply to {name or email} - {draft.label}",
+        payload=payload,
+        recipients=[o for o in owners],
+        enabled=zendesk_configured() and not blocked,
+        blocked_reason=blocked or ("" if zendesk_configured() else "Zendesk is not configured."),
+    )
+
+
+def build_jira(diagnosis: Any) -> Optional[Action]:
+    """An Infrastructure escalation, only when a step is owned by Infra."""
+    infra_steps = [a for a in diagnosis.actions if a.owner != "CX" and a.kind == "escalate"]
+    if not infra_steps:
+        return None
+    alert = diagnosis.alert
+    subject = alert.host or alert.instance_name or alert.floating_ip or "unknown"
+
+    description = "\n".join([
+        _evidence_block(diagnosis),
+        "",
+        "Requested of Infrastructure:",
+        *[f"- {s.text}" for s in infra_steps],
+        "",
+        f"Raised from CX Triage. Alert has held for {alert.effective_age_text}.",
+    ])
+    payload = {
+        "fields": {
+            "project": {"key": JIRA_PROJECT},
+            "summary": f"{subject}: {diagnosis.verdict}"[:250],
+            "description": description,
+            "issuetype": {"name": "Task"},
+            "labels": ["cx-triage", f"alert-{alert.kind}", f"region-{alert.region or 'unknown'}"],
+        }
+    }
+    return Action(
+        id="jira", kind="jira",
+        label="Escalate to Infrastructure (Jira)",
+        summary=f"Create a {JIRA_PROJECT} issue for {subject}",
+        payload=payload,
+        enabled=jira_configured(),
+        blocked_reason="" if jira_configured() else "Jira is not configured.",
+    )
+
+
+def build_manual(diagnosis: Any) -> list[Action]:
+    """Steps a human must perform; surfaced as copyable commands, not buttons."""
+    out: list[Action] = []
+    alert = diagnosis.alert
+    osid = alert.openstack_id
+    region = alert.region
+    for step in diagnosis.actions:
+        if step.kind != "remediate" or step.status == "done":
+            continue
+        command = ""
+        low = step.text.lower()
+        if "delete the server in openstack" in low and osid and region:
+            command = f"{region} server delete {osid}"
+        elif "shelve" in low and osid and region:
+            command = f"{region} server shelve {osid}"
+        out.append(Action(
+            id=f"manual-{len(out)}", kind="manual", label=step.text,
+            summary=step.guide or "", payload={"command": command} if command else {},
+            enabled=False, blocked_reason="Perform manually - this tool is read-only.",
+            requires_confirmation=False,
+        ))
+    return out
+
+
+def build_all(diagnosis: Any) -> dict[str, Any]:
+    actions: list[Action] = []
+    for builder in (build_zendesk, build_jira):
+        action = builder(diagnosis)
+        if action:
+            actions.append(action)
+    actions.extend(build_manual(diagnosis))
+    return {
+        "actions": [a.to_json() for a in actions],
+        "zendesk_configured": zendesk_configured(),
+        "jira_configured": jira_configured(),
+    }

backend/triagelib/linkage.py

@@ -0,0 +1,233 @@
+"""Linkage analysis: Infrahub records that lost their OpenStack server, and
+OpenStack servers that no Infrahub record claims.
+
+A VM in ERROR is not always a failed build. Sometimes the build succeeded and
+only the *link* between Infrahub and OpenStack was never written - so Infrahub
+reports ERROR (or CREATING) with no usable openstack_id while a perfectly good
+server of the same name is running. Those look identical on an alert dashboard
+and are opposite problems: one needs a rebuild, the other needs a record fixed
+and is quietly billing nobody.
+
+This scans both sides in bulk and pairs them up by name.
+
+It also finds the reverse - OpenStack servers with no Infrahub record at all -
+which is what `Suspected Orphan VM` was meant to catch before its input metric
+went empty.
+"""
+from __future__ import annotations
+
+import difflib
+import threading
+import time
+from typing import Any, Optional
+
+from . import cxbridge
+
+REGIONS = ("ca1", "ca2", "us1", "no1")
+
+# Infrahub states where a missing OpenStack link is suspicious rather than normal.
+# HIBERNATED is excluded: shelved instances legitimately have no running server.
+UNLINKED_SUSPECT_STATES = {"ERROR", "CREATING", "BUILD", "ACTIVE", "REBOOTING", "RESTORING"}
+
+# Projects to ignore, matching the CX-Tools tempest suppression.
+def _is_ignorable(name: str) -> bool:
+    low = str(name or "").lower()
+    return "tempest" in low
+
+
+_REGION_ALIAS = {"canada-1": "ca1", "canada-2": "ca2", "us-1": "us1", "norway-1": "no1",
+                 "ca1": "ca1", "ca2": "ca2", "us1": "us1", "no1": "no1"}
+
+
+def _region_alias(region: str) -> str:
+    return _REGION_ALIAS.get(str(region or "").strip().lower(), "")
+
+
+def _norm(name: str) -> str:
+    return str(name or "").strip().lower()
+
+
+class Scan:
+    """One full cross-region scan. Slow (a server list per region), so cached."""
+
+    def __init__(self):
+        self.started = 0.0
+        self.finished = 0.0
+        self.state = "idle"        # idle | running | done | error
+        self.error = ""
+        self.progress = ""
+        self.result: dict[str, Any] = {}
+        self._lock = threading.Lock()
+
+    def to_json(self) -> dict[str, Any]:
+        return {
+            "state": self.state, "error": self.error, "progress": self.progress,
+            "started": self.started, "finished": self.finished,
+            "age_seconds": int(time.time() - self.finished) if self.finished else None,
+            "result": self.result,
+        }
+
+    def run(self, snapshot: Any, regions: tuple[str, ...] = REGIONS) -> None:
+        with self._lock:
+            if self.state == "running":
+                return
+            self.state = "running"
+            self.started = time.time()
+            self.error = ""
+            self.progress = "starting"
+        try:
+            self.result = self._scan(snapshot, regions)
+            self.state = "done"
+        except Exception as exc:  # surfaced in the UI rather than crashing the server
+            self.state = "error"
+            self.error = f"{type(exc).__name__}: {exc}"
+        finally:
+            self.finished = time.time()
+            self.progress = ""
+
+    # --- the analysis ------------------------------------------------------
+
+    def _scan(self, snapshot: Any, regions: tuple[str, ...]) -> dict[str, Any]:
+        # 1. Everything OpenStack has, per region.
+        os_by_id: dict[str, dict[str, str]] = {}
+        os_by_name: dict[str, list[dict[str, str]]] = {}
+        region_counts: dict[str, int] = {}
+        failures: dict[str, str] = {}
+
+        for region in regions:
+            self.progress = f"listing OpenStack servers in {region}"
+            ok, rows, raw = _server_list(region)
+            if not ok:
+                failures[region] = raw[:200]
+                continue
+            region_counts[region] = len(rows)
+            for row in rows:
+                sid = str(row.get("ID") or "")
+                if not sid:
+                    continue
+                rec = {
+                    "id": sid,
+                    "name": str(row.get("Name") or ""),
+                    "status": str(row.get("Status") or ""),
+                    "task": str(row.get("Task State") or ""),
+                    "host": str(row.get("Host") or ""),
+                    "project_id": str(row.get("Project ID") or ""),
+                    "flavor": str(row.get("Flavor") or ""),
+                    "region": region,
+                }
+                os_by_id[sid] = rec
+                os_by_name.setdefault(_norm(rec["name"]), []).append(rec)
+
+        # 2. Everything Infrahub has, from the bulk metric snapshot.
+        self.progress = "comparing against Infrahub"
+        infrahub = list(snapshot.by_openstack_id.values()) + list(snapshot.by_instance_name.values())
+        seen: set[str] = set()
+        ih_records: list[dict[str, str]] = []
+        for row in infrahub:
+            key = f"{row.get('openstack_id','')}|{row.get('instance_name','')}"
+            if key in seen:
+                continue
+            seen.add(key)
+            ih_records.append(row)
+
+        ih_osids = {str(r.get("openstack_id") or "") for r in ih_records if r.get("openstack_id")}
+        ih_osids.discard("")
+        ih_osids.discard("None")
+
+        # 3a. Infrahub records whose OpenStack server is missing or never linked.
+        #
+        # Only records in a region that was actually listed can be judged: if a
+        # region failed, every VM in it would look "missing from OpenStack".
+        scanned = set(region_counts)
+        skipped_unscanned = 0
+
+        broken_links: list[dict[str, Any]] = []
+        for row in ih_records:
+            status = str(row.get("status") or "").upper()
+            if status not in UNLINKED_SUSPECT_STATES:
+                continue
+            if _region_alias(str(row.get("region") or "")) not in scanned:
+                skipped_unscanned += 1
+                continue
+            osid = str(row.get("openstack_id") or "")
+            has_link = bool(osid) and osid != "None"
+            if has_link and osid in os_by_id:
+                continue  # properly linked, nothing to see
+
+            name = str(row.get("instance_name") or "")
+            if _is_ignorable(name):
+                continue
+
+            candidates = os_by_name.get(_norm(name), [])
+            # An exact-name server that nothing else claims is a very strong
+            # candidate for the link that was never written.
+            unclaimed = [c for c in candidates if c["id"] not in ih_osids]
+            match = unclaimed[0] if unclaimed else (candidates[0] if candidates else None)
+
+            broken_links.append({
+                "instance_name": name,
+                "infrahub_status": status,
+                "infrahub_openstack_id": osid or "(none)",
+                "organization": str(row.get("organization") or ""),
+                "region": str(row.get("region") or ""),
+                "flavor": str(row.get("flavor_name") or ""),
+                "gpus": str(row.get("_gpus") or ""),
+                "reason": (
+                    "Infrahub holds an OpenStack ID that OpenStack does not have"
+                    if has_link else "Infrahub never recorded an OpenStack ID"
+                ),
+                "candidate": match,
+                "candidate_claimed_by_other": bool(match and match["id"] in ih_osids),
+                "confidence": (
+                    "high" if match and not match["id"] in ih_osids and match["status"] not in ("", "ERROR")
+                    else "medium" if match else "none"
+                ),
+            })
+
+        # 3b. OpenStack servers no Infrahub record claims.
+        orphans: list[dict[str, Any]] = []
+        ih_names = {_norm(str(r.get("instance_name") or "")) for r in ih_records}
+        for sid, rec in os_by_id.items():
+            if sid in ih_osids or _is_ignorable(rec["name"]):
+                continue
+            orphans.append({**rec, "name_known_to_infrahub": _norm(rec["name"]) in ih_names})
+
+        linkable = [b for b in broken_links if b["candidate"] and not b["candidate_claimed_by_other"]]
+        return {
+            "scanned_regions": region_counts,
+            "region_failures": failures,
+            "openstack_servers": len(os_by_id),
+            "infrahub_records": len(ih_records),
+            "broken_links": sorted(broken_links, key=lambda b: (b["confidence"] != "high", b["instance_name"])),
+            "likely_linkage_failures": len(linkable),
+            "skipped_unscanned_regions": skipped_unscanned,
+            "orphans": sorted(orphans, key=lambda o: (not o["name_known_to_infrahub"], o["name"]))[:400],
+            "orphan_total": len(orphans),
+        }
+
+
+def _server_list(region: str) -> tuple[bool, list[dict[str, Any]], str]:
+    ok, data, raw = cxbridge.os_json(
+        region, ["server", "list", "--all-projects", "--long", "-f", "json"], timeout=180
+    )
+    if ok and isinstance(data, list):
+        return True, [x for x in data if isinstance(x, dict)], ""
+    return False, [], str(raw)
+
+
+def enrich(region: str, openstack_id: str) -> dict[str, Any]:
+    """Fetch created time and fault for one candidate, on demand."""
+    c = cxbridge.cx()
+    ok, srv, raw = c.server_show(cxbridge.config(), region, openstack_id)
+    if not ok:
+        return {"ok": False, "error": str(raw)[:200]}
+    fault = srv.get("fault")
+    return {
+        "ok": True,
+        "created": str(srv.get("created") or srv.get("Created") or ""),
+        "launched": str(srv.get("OS-SRV-USG:launched_at") or ""),
+        "status": str(srv.get("status") or ""),
+        "host": str(srv.get("OS-EXT-SRV-ATTR:host") or ""),
+        "project_id": str(srv.get("project_id") or ""),
+        "fault": (fault.get("message") if isinstance(fault, dict) else str(fault or "")) or "None",
+    }

backend/triagelib/prometheus.py

@@ -0,0 +1,545 @@
+"""Prometheus access.
+
+The alert Prometheus lives on the internal 10.11/8 network, which is reachable
+only from inside the CX-Tools VPN containers - the laptop itself routes 10.11.*
+out of its default gateway. So queries go the same way CX-Tools reaches
+OpenStack: `docker exec <region>-osc curl ...`. A direct HTTP transport is tried
+first so this still works from a host that does have a route.
+"""
+from __future__ import annotations
+
+import json
+import os
+import re
+import shlex
+import subprocess
+import threading
+import time
+import urllib.parse
+import urllib.request
+from typing import Any, Optional
+
+DEFAULT_BASE = os.environ.get("CX_PROMETHEUS_BASE", "http://10.11.254.250:9090")
+
+# Containers to try as an HTTP relay, in order. These are the CX-Tools
+# OpenStack client containers, which share the regional VPN network namespace.
+RELAY_CONTAINERS = ("ca1-osc", "us1-osc", "no1-osc", "ca2-osc")
+
+
+class PrometheusError(RuntimeError):
+    pass
+
+
+class PrometheusClient:
+    def __init__(self, base: str = DEFAULT_BASE, timeout: int = 20):
+        self.base = base.rstrip("/")
+        self.timeout = timeout
+        self._transport: Optional[tuple[str, str]] = None
+        self._lock = threading.Lock()
+
+    # --- transport selection ------------------------------------------------
+
+    def _try_direct(self) -> bool:
+        try:
+            req = urllib.request.Request(f"{self.base}/api/v1/status/buildinfo", headers={"User-Agent": "cx-triage"})
+            with urllib.request.urlopen(req, timeout=5) as resp:
+                return 200 <= getattr(resp, "status", 200) < 300
+        except Exception:
+            return False
+
+    def _try_relay(self, container: str) -> bool:
+        rc, out, _err = _run(
+            ["docker", "exec", "-i", container, "curl", "-sS", "-m", "6", f"{self.base}/api/v1/status/buildinfo"],
+            timeout=15,
+        )
+        return rc == 0 and '"status":"success"' in out
+
+    def transport(self) -> tuple[str, str]:
+        """Return (kind, detail) where kind is 'direct' or 'relay'."""
+        with self._lock:
+            if self._transport is not None:
+                return self._transport
+            forced = os.environ.get("CX_PROMETHEUS_RELAY", "").strip()
+            if forced:
+                self._transport = ("relay", forced)
+                return self._transport
+            if self._try_direct():
+                self._transport = ("direct", "host")
+                return self._transport
+            for container in RELAY_CONTAINERS:
+                if self._try_relay(container):
+                    self._transport = ("relay", container)
+                    return self._transport
+            raise PrometheusError(
+                f"Cannot reach Prometheus at {self.base}. The host has no route to the internal "
+                f"network and none of {', '.join(RELAY_CONTAINERS)} answered. Start the CX-Tools "
+                "VPN/OSC containers, or set CX_PROMETHEUS_RELAY to a container that has a route."
+            )
+
+    def describe_transport(self) -> str:
+        try:
+            kind, detail = self.transport()
+        except PrometheusError as exc:
+            return f"unavailable ({exc})"
+        return "direct from host" if kind == "direct" else f"relayed through {detail}"
+
+    # --- requests -----------------------------------------------------------
+
+    def _get(self, path: str, params: Optional[dict[str, str]] = None) -> Any:
+        url = f"{self.base}{path}"
+        if params:
+            url = f"{url}?{urllib.parse.urlencode(params)}"
+        kind, detail = self.transport()
+        if kind == "direct":
+            req = urllib.request.Request(url, headers={"User-Agent": "cx-triage"})
+            with urllib.request.urlopen(req, timeout=self.timeout) as resp:
+                body = resp.read().decode("utf-8", errors="replace")
+        else:
+            rc, out, err = _run(
+                ["docker", "exec", "-i", detail, "curl", "-sS", "-m", str(self.timeout), url],
+                timeout=self.timeout + 10,
+            )
+            if rc != 0:
+                raise PrometheusError(f"Prometheus relay via {detail} failed: {(err or out).strip()}")
+            body = out
+        try:
+            data = json.loads(body)
+        except json.JSONDecodeError as exc:
+            raise PrometheusError(f"Prometheus returned non-JSON for {path}: {body[:200]}") from exc
+        if data.get("status") != "success":
+            raise PrometheusError(f"Prometheus error for {path}: {data.get('error') or data}")
+        return data.get("data")
+
+    def alerts(self) -> list[dict[str, Any]]:
+        data = self._get("/api/v1/alerts") or {}
+        alerts = data.get("alerts")
+        return [a for a in alerts if isinstance(a, dict)] if isinstance(alerts, list) else []
+
+    def query(self, expr: str) -> list[dict[str, Any]]:
+        data = self._get("/api/v1/query", {"query": expr}) or {}
+        result = data.get("result")
+        return [r for r in result if isinstance(r, dict)] if isinstance(result, list) else []
+
+    def resources_by_floating_ip(self, floating_ip: str) -> list[dict[str, Any]]:
+        """The `Resources{floating_ip="..."}` query the Duplicated IPs runbook uses.
+
+        Unlike CX-Tools (production Infrahub only), this series covers every
+        environment, so it is how a PreProd/Staging claimant gets found.
+        """
+        expr = 'Resources{floating_ip="%s"}' % floating_ip.replace('"', "")
+        return [dict(r.get("metric") or {}) for r in self.query(expr)]
+
+    def query_range(self, expr: str, start: int, end: int, step: int) -> list[dict[str, Any]]:
+        data = self._get("/api/v1/query_range", {
+            "query": expr, "start": str(start), "end": str(end), "step": str(step),
+        }) or {}
+        result = data.get("result")
+        return [r for r in result if isinstance(r, dict)] if isinstance(result, list) else []
+
+    def rules(self) -> list[dict[str, Any]]:
+        data = self._get("/api/v1/rules") or {}
+        return [g for g in (data.get("groups") or []) if isinstance(g, dict)]
+
+    def series_count(self, metric: str) -> int:
+        rows = self.query(f"count({metric})")
+        if not rows:
+            return 0
+        try:
+            return int(float(rows[0]["value"][1]))
+        except (KeyError, IndexError, ValueError, TypeError):
+            return 0
+
+
+class RuleIndex:
+    """Maps alertname -> which rule file it came from and its `for` duration.
+
+    Keying off the rule file (not the alert name) is what lets node-exporter
+    alerts be separated reliably: two different files both use the group name
+    "Imported Rules".
+    """
+
+    def __init__(self, client: PrometheusClient, ttl: float = 600.0):
+        self.client = client
+        self.ttl = ttl
+        self._at = 0.0
+        self._by_name: dict[str, dict[str, Any]] = {}
+        self._error = ""
+        self._lock = threading.Lock()
+
+    def refresh(self) -> None:
+        groups = self.client.rules()
+        index: dict[str, dict[str, Any]] = {}
+        for group in groups:
+            source = str(group.get("file") or "").rsplit("/", 1)[-1]
+            for rule in group.get("rules") or []:
+                if rule.get("type") != "alerting":
+                    continue
+                index[str(rule.get("name") or "")] = {
+                    "group": str(group.get("name") or ""),
+                    "file": source,
+                    "for_seconds": int(rule.get("duration") or 0),
+                    "query": str(rule.get("query") or ""),
+                }
+        self._by_name = index
+
+    def ensure(self) -> None:
+        """Refresh if the cache is empty or stale."""
+        with self._lock:
+            if not self._by_name or time.monotonic() - self._at > self.ttl:
+                try:
+                    self.refresh()
+                    self._error = ""
+                except PrometheusError as exc:
+                    self._error = str(exc)
+                self._at = time.monotonic()
+
+    def get(self, alertname: str) -> dict[str, Any]:
+        self.ensure()
+        return self._by_name.get(alertname, {})
+
+    @property
+    def count(self) -> int:
+        return len(self._by_name)
+
+    @property
+    def error(self) -> str:
+        return self._error
+
+
+# Metrics the alert rules are built on. If one of these is empty, rules that
+# depend on it are broken rather than quiet - see StateSnapshot.broken_inputs.
+RULE_INPUT_METRICS = ("Resources", "In_Use_Gpus", "Total_Gpus", "openstack_nova_server_status")
+
+ROGUE_DELTA_EXPR = (
+    'sum by (instance) (In_Use_Gpus) - sum by (instance) '
+    '(Resources{organization!="3491 - luis.sarabando+runpod@nexgencloud.coms-Organization",'
+    'status=~"ACTIVE|SHUTOFF|PRE_ACTIVE"})'
+)
+
+
+def _episode(points: list[tuple[int, float]], median: float) -> dict[str, Any]:
+    return {
+        "start": points[0][0],
+        "end": points[-1][0],
+        "minutes": max(1, (points[-1][0] - points[0][0]) // 60 + 1),
+        "low": int(min(v for _, v in points)),
+        "normal": int(median),
+    }
+
+
+# Labels that identify one alert across time, for true-age lookup.
+TRUE_AGE_KEY_LABELS = ("alertname", "instance_name", "floating_ip", "instance", "openstack_id")
+
+
+def true_age_key(labels: dict[str, Any]) -> tuple:
+    """Identity of an alert, built from raw label values on both sides."""
+    return tuple(str((labels or {}).get(k, "")) for k in TRUE_AGE_KEY_LABELS)
+
+
+class TrueAgeIndex:
+    """How long each alert's condition has *actually* held.
+
+    Prometheus resets an alert's activeAt whenever the alert resolves, and the
+    Infrahub metric pipeline drops most of the `Resources` series for a few
+    minutes several times a day. Every alert alive during such a dip resolves and
+    re-fires, so activeAt collapses to "time since the last dip" and every alert
+    reports the same age.
+
+    This walks the `ALERTS` series backwards instead, bridging gaps shorter than
+    GAP_TOLERANCE, which recovers the real duration. Both pending and firing are
+    counted, so rules with a long `for:` are not reported as young.
+    """
+
+    WINDOW_DAYS = 7
+    STEP_SECONDS = 900
+    GAP_TOLERANCE = 2700  # 45 min: bridges pipeline dips, not genuine recoveries
+
+    def __init__(self, client: PrometheusClient, severity: str = "infrahub-critical", ttl: float = 300.0):
+        self.client = client
+        self.severity = severity
+        self.ttl = ttl
+        self._at = 0.0
+        self._lock = threading.Lock()
+        self._starts: dict[tuple, tuple[int, bool]] = {}
+        self.error = ""
+        self.window_start = 0
+
+    def refresh(self) -> None:
+        end = int(time.time())
+        start = end - self.WINDOW_DAYS * 86400
+        self.window_start = start
+        expr = (
+            "count by (%s) (ALERTS{severity=\"%s\"})"
+            % (", ".join(TRUE_AGE_KEY_LABELS), self.severity)
+        )
+        starts: dict[tuple, tuple[int, bool]] = {}
+        for series in self.client.query_range(expr, start, end, self.STEP_SECONDS):
+            stamps = []
+            for point in series.get("values") or []:
+                try:
+                    stamps.append(int(float(point[0])))
+                except (ValueError, TypeError, IndexError):
+                    continue
+            if not stamps:
+                continue
+            run_start = stamps[-1]
+            for earlier, later in list(zip(stamps, stamps[1:]))[::-1]:
+                if later - earlier > self.GAP_TOLERANCE:
+                    break
+                run_start = earlier
+            # A run that reaches the window edge is only a lower bound.
+            capped = run_start <= start + self.STEP_SECONDS
+            key = true_age_key(series.get("metric") or {})
+            existing = starts.get(key)
+            if existing is None or run_start < existing[0]:
+                starts[key] = (run_start, capped)
+        self._starts = starts
+
+    def get(self) -> "TrueAgeIndex":
+        with self._lock:
+            if not self._at or time.monotonic() - self._at > self.ttl:
+                try:
+                    self.refresh()
+                    self.error = ""
+                except PrometheusError as exc:
+                    self.error = str(exc)
+                self._at = time.monotonic()
+        return self
+
+    def lookup(self, labels: dict[str, Any]) -> tuple[Optional[int], bool]:
+        """Return (minutes the condition has held, whether that is a floor)."""
+        entry = self._starts.get(true_age_key(labels))
+        if not entry:
+            return None, False
+        start, capped = entry
+        return max(0, int((time.time() - start) // 60)), capped
+
+    @property
+    def loaded(self) -> bool:
+        return bool(self._starts)
+
+    @property
+    def count(self) -> int:
+        return len(self._starts)
+
+
+class StateSnapshot:
+    """A bulk read of current platform state, used to screen alerts cheaply.
+
+    Re-checking whether an alert's condition still holds is what separates a
+    real alert from one that already self-resolved. Doing it from these few
+    aggregate queries costs one Prometheus round trip for the whole queue,
+    instead of an Infrahub and OpenStack call per alert.
+    """
+
+    def __init__(self, client: PrometheusClient, ttl: float = 60.0):
+        self.client = client
+        self.ttl = ttl
+        self._at = 0.0
+        self._lock = threading.Lock()
+        self.error = ""
+        self.by_openstack_id: dict[str, dict[str, str]] = {}
+        self.by_instance_name: dict[str, dict[str, str]] = {}
+        self.fip_counts: dict[str, int] = {}
+        self.rogue_delta: dict[str, float] = {}
+        self.total_gpus: dict[str, float] = {}
+        self.in_use_gpus: dict[str, float] = {}
+        self.resources_by_host: dict[str, list[dict[str, str]]] = {}
+        self.broken_inputs: list[str] = []
+        # Infrahub VMs in a GPU-counted state with no host recorded. These are
+        # invisible to the per-host GPU sum the Rogue VM rule uses, so they can
+        # manufacture a gap on whichever host is actually running them.
+        self.unattributed_active: int = 0
+        self.unattributed_active_gpus: int = 0
+        # Episodes where the Resources series partially collapsed. Each one
+        # resets activeAt on every alert that was live at the time.
+        self.pipeline_dips: list[dict[str, Any]] = []
+
+    def refresh(self) -> None:
+        by_osid: dict[str, dict[str, str]] = {}
+        by_name: dict[str, dict[str, str]] = {}
+        fips: dict[str, int] = {}
+        by_host: dict[str, list[dict[str, str]]] = {}
+        counted_states = {"ACTIVE", "SHUTOFF", "PRE_ACTIVE"}
+        unattributed = 0
+        unattributed_gpus = 0
+
+        for row in self.client.query("Resources"):
+            metric = {str(k): str(v) for k, v in (row.get("metric") or {}).items()}
+            try:
+                metric["_gpus"] = str(int(float(row.get("value", [0, "0"])[1])))
+            except (ValueError, TypeError, IndexError):
+                metric["_gpus"] = "0"
+            osid = metric.get("openstack_id", "")
+            if osid and osid not in ("None", ""):
+                by_osid[osid] = metric
+            name = metric.get("instance_name", "")
+            if name:
+                by_name[name] = metric
+            fip = metric.get("floating_ip", "")
+            if fip and fip not in ("None", "NULL", ""):
+                fips[fip] = fips.get(fip, 0) + 1
+            host = metric.get("instance", "")
+            if host and host not in ("Unknown", "None"):
+                by_host.setdefault(host, []).append(metric)
+            elif metric.get("status", "").upper() in counted_states:
+                unattributed += 1
+                unattributed_gpus += int(metric["_gpus"] or 0)
+
+        self.by_openstack_id = by_osid
+        self.by_instance_name = by_name
+        self.fip_counts = fips
+        self.resources_by_host = by_host
+        self.unattributed_active = unattributed
+        self.unattributed_active_gpus = unattributed_gpus
+        # Summed by instance: these metrics are per (instance, gpu_name), so a
+        # host with two GPU models carries two series. Reading them unsummed
+        # would silently keep only one.
+        self.rogue_delta = self._scalar_by_instance(ROGUE_DELTA_EXPR)
+        self.total_gpus = self._scalar_by_instance("sum by (instance) (Total_Gpus)")
+        self.in_use_gpus = self._scalar_by_instance("sum by (instance) (In_Use_Gpus)")
+
+        self.broken_inputs = [m for m in RULE_INPUT_METRICS if self.client.series_count(m) == 0]
+        self.pipeline_dips = self._find_pipeline_dips()
+
+    def _find_pipeline_dips(self, hours: int = 24, drop_ratio: float = 0.8) -> list[dict[str, Any]]:
+        """Find episodes where most of the `Resources` series went missing."""
+        end = int(time.time())
+        start = end - hours * 3600
+        series = self.client.query_range("count(Resources)", start, end, 60)
+        if not series:
+            return []
+        points: list[tuple[int, float]] = []
+        for point in series[0].get("values") or []:
+            try:
+                points.append((int(float(point[0])), float(point[1])))
+            except (ValueError, TypeError, IndexError):
+                continue
+        if len(points) < 10:
+            return []
+        ordered = sorted(v for _, v in points)
+        median = ordered[len(ordered) // 2]
+        if median <= 0:
+            return []
+
+        episodes: list[dict[str, Any]] = []
+        current: list[tuple[int, float]] = []
+        for stamp, value in points:
+            if value < median * drop_ratio:
+                if current and stamp - current[-1][0] > 180:
+                    episodes.append(_episode(current, median))
+                    current = []
+                current.append((stamp, value))
+            elif current:
+                episodes.append(_episode(current, median))
+                current = []
+        if current:
+            episodes.append(_episode(current, median))
+        return episodes
+
+    def _scalar_by_instance(self, expr: str) -> dict[str, float]:
+        out: dict[str, float] = {}
+        for row in self.client.query(expr):
+            host = str((row.get("metric") or {}).get("instance") or "")
+            if not host:
+                continue
+            try:
+                out[host] = float(row.get("value", [0, "0"])[1])
+            except (ValueError, TypeError, IndexError):
+                continue
+        return out
+
+    def get(self) -> "StateSnapshot":
+        with self._lock:
+            if not self._at or time.monotonic() - self._at > self.ttl:
+                try:
+                    self.refresh()
+                    self.error = ""
+                except PrometheusError as exc:
+                    self.error = str(exc)
+                self._at = time.monotonic()
+        return self
+
+    @property
+    def loaded(self) -> bool:
+        return bool(self.by_openstack_id) or bool(self.total_gpus)
+
+
+def _run(cmd: list[str], timeout: int) -> tuple[int, str, str]:
+    try:
+        p = subprocess.run(cmd, text=True, capture_output=True, timeout=timeout)
+        return p.returncode, p.stdout, p.stderr
+    except subprocess.TimeoutExpired as exc:
+        return 124, exc.stdout or "", exc.stderr or f"timed out after {timeout}s"
+    except FileNotFoundError as exc:
+        return 127, "", str(exc)
+    except Exception as exc:  # pragma: no cover
+        return 1, "", str(exc)
+
+
+# --- parsing pasted alert text / URLs --------------------------------------
+
+_LABEL_RE = re.compile(r'(\w+)\s*=\s*"((?:[^"\\]|\\.)*)"')
+
+
+def parse_alert_text(text: str) -> list[dict[str, str]]:
+    """Parse pasted `ALERTS{...}` lines, or a Prometheus graph URL, into labels.
+
+    Accepts what a CX engineer actually copies: a raw series line from the
+    Prometheus console, or the `g0.expr=` URL from the Slack alert.
+    """
+    text = (text or "").strip()
+    if not text:
+        return []
+
+    found: list[dict[str, str]] = []
+
+    # A pasted graph URL: pull the expression out and parse its label matchers.
+    if text.startswith("http://") or text.startswith("https://"):
+        parsed = urllib.parse.urlparse(text)
+        params = urllib.parse.parse_qs(parsed.query)
+        exprs = [v for k, vs in params.items() if k.endswith("expr") for v in vs]
+        for expr in exprs:
+            labels = {k: _unescape(v) for k, v in _LABEL_RE.findall(expr)}
+            if labels:
+                found.append(labels)
+        return found
+
+    for block in re.findall(r"\{[^{}]*\}", text):
+        labels = {k: _unescape(v) for k, v in _LABEL_RE.findall(block)}
+        if labels:
+            found.append(labels)
+    if not found:
+        labels = {k: _unescape(v) for k, v in _LABEL_RE.findall(text)}
+        if labels:
+            found.append(labels)
+    return found
+
+
+def _unescape(value: str) -> str:
+    return value.replace('\\"', '"').replace("\\\\", "\\").replace("\\n", "\n")
+
+
+class AlertCache:
+    """Short-lived cache so the UI can poll without hammering Prometheus."""
+
+    def __init__(self, client: PrometheusClient, ttl: float = 30.0):
+        self.client = client
+        self.ttl = ttl
+        self._at = 0.0
+        self._alerts: list[dict[str, Any]] = []
+        self._error = ""
+        self._lock = threading.Lock()
+
+    def get(self, force: bool = False) -> tuple[list[dict[str, Any]], str, float]:
+        with self._lock:
+            age = time.monotonic() - self._at
+            if not force and self._at and age < self.ttl:
+                return self._alerts, self._error, age
+            try:
+                self._alerts = self.client.alerts()
+                self._error = ""
+            except PrometheusError as exc:
+                self._error = str(exc)
+            self._at = time.monotonic()
+            return self._alerts, self._error, 0.0

backend/triagelib/screening.py

@@ -0,0 +1,313 @@
+"""Noise-vs-real screening.
+
+An alert firing is not the same as work existing. Prometheus keeps an alert up
+until its expression stops matching on the next evaluation, and the CX rules sit
+on top of a metric pipeline that can go stale or empty. So before anything gets
+diagnosed, each alert's condition is re-checked against current state.
+
+The re-check is deliberately cheap: it reads the same Prometheus series the rules
+are built from (one bulk snapshot for the whole queue) rather than making an
+Infrahub or OpenStack call per alert. Anything it cannot settle is treated as
+real - screening only ever demotes an alert on positive evidence.
+"""
+from __future__ import annotations
+
+import time
+from typing import Any, Optional
+
+from .alerts import Alert
+
+# Beyond this, an alert is chronic: it is either already ticketed or nobody has
+# silenced it. Either way it is not today's queue.
+CHRONIC_DAYS = 3
+
+# Verdicts, most to least urgent.
+REAL = "real"
+OVERDUE = "overdue"
+UNVERIFIED = "unverified"
+CHRONIC = "chronic"
+LOW_IMPACT = "low_impact"
+PENDING = "pending"
+RESOLVED = "resolved"
+RULE_DEFECT = "rule_defect"
+SUPPRESSED = "suppressed"
+
+VERDICT_LABELS = {
+    REAL: "needs action",
+    OVERDUE: "overdue",
+    UNVERIFIED: "needs action (unverified)",
+    CHRONIC: "chronic",
+    LOW_IMPACT: "low impact",
+    PENDING: "not yet firing",
+    RESOLVED: "already resolved",
+    RULE_DEFECT: "invalid - alert rule defect",
+    SUPPRESSED: "hidden by a rule",
+}
+
+# Runbook commitments: past this age the customer contact is late, not chronic.
+# From "Instance in ERROR state": if a stock-failure instance is not deleted
+# within a day, contact the customer. SHUTOFF is here for a different reason -
+# the VM accrues full cost the entire time it is stopped, so an old one is a
+# customer who has been paying for nothing for longer, not a stale alert.
+SLA_HOURS = {"error": 24, "creating": 24, "restoring": 24, "rebooting": 24,
+             "build": 24, "shutoff": 48}
+
+SLA_REASON = {
+    "shutoff": "a SHUTOFF VM accrues full cost the whole time, so this customer has been paying "
+               "for a stopped instance that long and may never have been told",
+}
+
+# Verdicts that stay in the working queue by default.
+ACTIONABLE = {REAL, OVERDUE, UNVERIFIED}
+
+# Kept for the SLA check: an internal owner should not make an alert *overdue*.
+def _is_internal(alert) -> bool:
+    return bool(getattr(alert, 'is_internal_org', False))
+
+
+def _result(verdict: str, reason: str, *, current: str = "", detail: str = "") -> dict[str, Any]:
+    return {
+        "verdict": verdict,
+        "label": VERDICT_LABELS[verdict],
+        "reason": reason,
+        "actionable": verdict in ACTIONABLE,
+        "current_state": current,
+        "detail": detail,
+    }
+
+
+def _resources_row(alert: Alert, snap: Any) -> Optional[dict[str, str]]:
+    """Find the VM's current Infrahub row in the snapshot."""
+    if alert.openstack_id and alert.openstack_id in snap.by_openstack_id:
+        return snap.by_openstack_id[alert.openstack_id]
+    if alert.instance_name and alert.instance_name in snap.by_instance_name:
+        return snap.by_instance_name[alert.instance_name]
+    return None
+
+
+def _screen_state_alert(alert: Alert, snap: Any) -> dict[str, Any]:
+    """State alerts: does Infrahub still report the state that fired?"""
+    expected = alert.status.upper()
+    row = _resources_row(alert, snap)
+
+    if row is None:
+        if alert.kind == "creating" and not alert.openstack_id:
+            # A CREATING VM that never reached OpenStack has no Resources row to
+            # find; the alert stands on its own.
+            return _result(REAL, "Instance never got an OpenStack ID, so it cannot have recovered.")
+        return _result(
+            RESOLVED,
+            "No longer present in Infrahub's Resources series - the record has been deleted or cleaned up.",
+        )
+
+    current = row.get("status", "").upper()
+    if not expected:
+        return _result(UNVERIFIED, "Alert carried no status label, so its condition could not be re-checked.",
+                       current=current)
+    if current == expected:
+        return _result(REAL, f"Infrahub still reports {current}.", current=current)
+    return _result(
+        RESOLVED,
+        f"Fired on {expected} but Infrahub now reports {current} - it resolved on its own.",
+        current=current,
+    )
+
+
+def _screen_duplicate_ip(alert: Alert, snap: Any) -> dict[str, Any]:
+    count = snap.fip_counts.get(alert.floating_ip)
+    if count is None:
+        return _result(RESOLVED, f"No Infrahub VM currently holds {alert.floating_ip}.")
+    if count > 1:
+        return _result(REAL, f"{count} VMs still hold {alert.floating_ip}.", current=f"{count} claimants")
+    return _result(
+        RESOLVED,
+        f"Only 1 VM holds {alert.floating_ip} now - the duplicate is gone.",
+        current="1 claimant",
+    )
+
+
+def _screen_rogue_vm(alert: Alert, snap: Any) -> dict[str, Any]:
+    """Rogue VM fires on a per-host GPU accounting gap; re-evaluate the gap.
+
+    The rule subtracts Infrahub's allocated GPUs from `In_Use_Gpus`. On almost
+    every host `In_Use_Gpus` equals `Total_Gpus` - the physical GPU count - so
+    the expression reduces to "this host has at least one unallocated GPU" and
+    fires on ordinary spare capacity. Validated against OpenStack on 10 firing
+    hosts: Infrahub and OpenStack agreed exactly on all of them.
+    """
+    host = alert.host or alert.instance_name
+    delta = snap.rogue_delta.get(host)
+    if delta is None:
+        return _result(UNVERIFIED, f"No current GPU accounting data for {host}.")
+    if delta >= 1:
+        known = len(snap.resources_by_host.get(host, []))
+        in_use = snap.in_use_gpus.get(host)
+        total = snap.total_gpus.get(host)
+        if in_use is not None and total is not None and in_use == total:
+            return _result(
+                RULE_DEFECT,
+                f"Not a rogue VM: on {host} the rule's 'GPUs in use' reading ({int(in_use)}) is just the "
+                f"physical GPU count, so it is reporting {int(delta)} free GPU(s) as a discrepancy.",
+                current=f"{int(delta)} GPU(s) spare capacity",
+                detail=(
+                    "In_Use_Gpus == Total_Gpus on this host, so the rule expression reduces to "
+                    "'physical GPUs minus allocated GPUs', which is spare capacity rather than an "
+                    "Infrahub/OpenStack mismatch. The rule needs fixing at source."
+                ),
+            )
+        if total is None:
+            detail = (f"Infrahub records {known} VM(s) on this host. No Total_Gpus reading is available, so the "
+                      "spare-capacity explanation cannot be confirmed or ruled out from metrics alone.")
+        else:
+            detail = (f"Infrahub records {known} VM(s) on this host. In_Use_Gpus ({int(in_use)}) differs from "
+                      f"Total_Gpus ({int(total)}), so this is not simply spare capacity.")
+        return _result(
+            REAL,
+            f"{int(delta)} GPU(s) allocated on {host} are still unaccounted for in Infrahub.",
+            current=f"gap {int(delta)}",
+            detail=detail,
+        )
+    return _result(
+        RESOLVED,
+        f"GPU accounting for {host} now balances (delta {int(delta)}).",
+        current=f"delta {int(delta)}",
+    )
+
+
+def _screen_total_gpus(alert: Alert, snap: Any) -> dict[str, Any]:
+    host = alert.host or alert.instance_name
+    total = snap.total_gpus.get(host)
+    if total is None:
+        return _result(UNVERIFIED, f"No current Total_Gpus reading for {host}.")
+    # The rule fires on any count in 1,2,3,5,6,7,9 - i.e. not a full complement.
+    if int(total) in (0, 4, 8):
+        return _result(
+            RESOLVED,
+            f"{host} now reports {int(total)} GPUs, a valid complement.",
+            current=f"{int(total)} GPUs",
+        )
+    in_use = snap.in_use_gpus.get(host)
+    detail = f"{int(in_use)} GPU(s) currently allocated to instances." if in_use is not None else ""
+    return _result(
+        REAL,
+        f"{host} still reports {int(total)} GPUs - hardware is missing.",
+        current=f"{int(total)} GPUs",
+        detail=detail,
+    )
+
+
+_KIND_SCREENS = {
+    "duplicate_ip": _screen_duplicate_ip,
+    "rogue_vm": _screen_rogue_vm,
+    "orphan_vm": _screen_rogue_vm,
+    "total_gpus": _screen_total_gpus,
+}
+
+
+def screen(alert: Alert, snap: Any, user_settings: Any = None) -> dict[str, Any]:
+    """Decide whether an alert is worth a human's attention right now."""
+    # A rule the team wrote wins over anything inferred here.
+    if user_settings is not None:
+        from . import settings as settings_mod
+
+        rule = settings_mod.first_match(user_settings, alert)
+        if rule:
+            reason = rule.get("reason") or "Matched a suppression rule."
+            return _result(
+                SUPPRESSED,
+                f"Hidden by \u201c{rule.get('name')}\u201d - {reason}",
+                detail="Edit or disable this in Settings.",
+            )
+
+    # Prometheus has not committed to this alert yet.
+    if alert.state == "pending":
+        remaining = ""
+        if alert.for_seconds and alert.age_minutes is not None:
+            remaining = f" It needs {alert.for_seconds // 60} min of continuous firing; it has {alert.age_minutes} min."
+        return _result(PENDING, f"Prometheus still has this pending, not firing.{remaining}")
+
+    if snap is None or not getattr(snap, "loaded", False):
+        return _result(UNVERIFIED, "Current-state snapshot unavailable, so the condition could not be re-checked.")
+
+    screener = _KIND_SCREENS.get(alert.kind)
+    result = screener(alert, snap) if screener else _screen_state_alert(alert, snap)
+
+    # A still-valid alert can still be the wrong thing to spend time on.
+    if result["actionable"]:
+        # Uses the recovered duration: activeAt is reset by pipeline dips, which
+        # would make every chronic alert look hours old.
+        age = alert.effective_age_minutes
+        sla = SLA_HOURS.get(alert.kind)
+        if sla and age is not None and age > sla * 60 and not alert.is_internal_org:
+            # The runbook commits to contacting the customer inside this window,
+            # so age makes it more urgent, not less. Never demote these to chronic.
+            why = SLA_REASON.get(
+                alert.kind,
+                f"past the {sla}h point where the runbook says to contact the customer",
+            )
+            return _result(
+                OVERDUE,
+                f"Condition has held for {alert.effective_age_text} - {why}. Overdue, not chronic.",
+                current=result["current_state"],
+                detail=result["reason"],
+            )
+        if age is not None and age > CHRONIC_DAYS * 24 * 60:
+            note = result["reason"]
+            if alert.age_is_reset:
+                note += (f" Prometheus reports only {alert.age_text} because a metric-pipeline dip reset "
+                         "activeAt; the condition itself has held far longer.")
+            return _result(
+                CHRONIC,
+                f"Condition still holds, but it has held for {alert.effective_age_text} - "
+                "chronic, so it is likely already ticketed rather than new work.",
+                current=result["current_state"],
+                detail=note,
+            )
+    return result
+
+
+def screen_all(items: list[Alert], snap: Any, user_settings: Any = None) -> None:
+    for alert in items:
+        alert.screen = screen(alert, snap, user_settings)
+
+
+def summarize(items: list[Alert]) -> dict[str, Any]:
+    counts: dict[str, int] = {}
+    for alert in items:
+        verdict = alert.screen.get("verdict", UNVERIFIED)
+        counts[verdict] = counts.get(verdict, 0) + 1
+    return {
+        "counts": counts,
+        "actionable": sum(1 for a in items if a.screen.get("actionable")),
+        "screened_out": sum(1 for a in items if not a.screen.get("actionable")),
+        "labels": VERDICT_LABELS,
+    }
+
+
+def health_warnings(snap: Any) -> list[str]:
+    """Rules whose input metrics are empty are broken, not quiet."""
+    warnings: list[str] = []
+    for metric in getattr(snap, "broken_inputs", []) or []:
+        if metric == "openstack_nova_server_status":
+            warnings.append(
+                "The OpenStack server metric (openstack_nova_server_status) is currently empty. Any rule built on "
+                "it is unreliable: 'Exists in Infrahub but does not exist in OpenStack' matches every VM (which is "
+                "why it is excluded here), and 'Suspected Orphan VM' cannot fire at all. Worth raising with whoever "
+                "owns the exporter."
+            )
+        else:
+            warnings.append(
+                f"The metric '{metric}' is currently empty, so alert rules that depend on it are unreliable."
+            )
+
+    dips = getattr(snap, "pipeline_dips", []) or []
+    if dips:
+        latest = max(dips, key=lambda d: d["end"])
+        mins_ago = max(0, int((time.time() - latest["end"]) // 60))
+        warnings.append(
+            f"The Infrahub 'Resources' metric dropped most of its series {len(dips)} time(s) in the last 24h "
+            f"(most recently {mins_ago} min ago: {latest['low']} of ~{latest['normal']} series for "
+            f"{latest['minutes']} min). Every alert live during a dip resolves and re-fires, so Prometheus' own "
+            "alert ages all reset together. Ages shown here are recovered from ALERTS history instead."
+        )
+    return warnings

backend/triagelib/settings.py

@@ -0,0 +1,245 @@
+"""User settings: suppression rules and comms identity.
+
+Suppression rules replace hardcoded judgement calls. The internal-organisation
+check used to be baked into the screening code, which meant the one person who
+knew about it was whoever read the source. Now it ships as an editable default
+rule that says what it does and why, and anyone can add their own.
+
+A rule matches when *every* condition it sets matches (AND); within one
+condition, any listed value matches (OR). So "type is error AND organisation
+contains modal" is one rule with two conditions.
+"""
+from __future__ import annotations
+
+import json
+import os
+import re
+import threading
+import time
+import uuid
+from typing import Any, Optional
+
+SETTINGS_DIR = os.path.expanduser(os.environ.get("CX_TRIAGE_HOME", "~/.cx-triage"))
+SETTINGS_PATH = os.path.join(SETTINGS_DIR, "settings.json")
+
+# Conditions a rule can set. Every one is a substring match, case-insensitive,
+# except `kind` and `region` which are exact.
+CONDITIONS = {
+    "kind": "Alert type",
+    "organization": "Organisation contains",
+    "instance_name": "VM name contains",
+    "host": "Host contains",
+    "region": "Region",
+    "status": "Status is",
+}
+
+DEFAULT_RULES: list[dict[str, Any]] = [
+    {
+        "id": "builtin-internal-orgs",
+        "name": "Internal NexGen organisations",
+        "enabled": True,
+        "reason": "Owned by an internal test or platform organisation, not a customer.",
+        "conditions": {"organization": ["nexgencloud.com"]},
+    },
+    {
+        "id": "builtin-runpod-storage",
+        "name": "Runpod storage nodes (Luis)",
+        "enabled": True,
+        "reason": "Platform-owned storage nodes; SHUTOFF on these is expected and not customer-impacting.",
+        "conditions": {"kind": ["shutoff"], "instance_name": ["stor-runpod"]},
+    },
+]
+
+DEFAULTS: dict[str, Any] = {
+    "rules": DEFAULT_RULES,
+    "agent_name": "",
+    "chronic_days": 3,
+}
+
+
+def _blank(value: Any) -> bool:
+    return value is None or str(value).strip() == ""
+
+
+class Settings:
+    """Loaded once, written through on every change."""
+
+    def __init__(self, path: str = SETTINGS_PATH):
+        self.path = path
+        self._lock = threading.Lock()
+        self._data: dict[str, Any] = {}
+        self.load()
+
+    # --- persistence -------------------------------------------------------
+
+    def load(self) -> None:
+        data = dict(DEFAULTS)
+        try:
+            with open(self.path, encoding="utf-8") as handle:
+                stored = json.load(handle)
+            if isinstance(stored, dict):
+                data.update(stored)
+        except (OSError, json.JSONDecodeError):
+            pass
+        data["rules"] = [r for r in (data.get("rules") or []) if isinstance(r, dict)]
+        with self._lock:
+            self._data = data
+
+    def save(self) -> None:
+        with self._lock:
+            payload = json.dumps(self._data, indent=2, sort_keys=True)
+        try:
+            os.makedirs(os.path.dirname(self.path), exist_ok=True)
+            tmp = f"{self.path}.tmp"
+            with open(tmp, "w", encoding="utf-8") as handle:
+                handle.write(payload)
+            os.replace(tmp, self.path)
+        except OSError:
+            pass
+
+    # --- accessors ---------------------------------------------------------
+
+    @property
+    def rules(self) -> list[dict[str, Any]]:
+        with self._lock:
+            return [dict(r) for r in self._data.get("rules", [])]
+
+    @property
+    def agent_name(self) -> str:
+        with self._lock:
+            return str(self._data.get("agent_name") or "")
+
+    @property
+    def chronic_days(self) -> int:
+        with self._lock:
+            try:
+                return max(1, int(self._data.get("chronic_days") or 3))
+            except (TypeError, ValueError):
+                return 3
+
+    def to_json(self) -> dict[str, Any]:
+        with self._lock:
+            return {
+                "rules": [dict(r) for r in self._data.get("rules", [])],
+                "agent_name": self._data.get("agent_name") or "",
+                "chronic_days": self._data.get("chronic_days", 3),
+                "conditions": CONDITIONS,
+                "path": self.path,
+            }
+
+    # --- mutations ---------------------------------------------------------
+
+    def set_general(self, agent_name: Optional[str] = None, chronic_days: Optional[Any] = None) -> None:
+        with self._lock:
+            if agent_name is not None:
+                self._data["agent_name"] = str(agent_name).strip()
+            if chronic_days is not None:
+                try:
+                    self._data["chronic_days"] = max(1, int(chronic_days))
+                except (TypeError, ValueError):
+                    pass
+        self.save()
+
+    def upsert_rule(self, rule: dict[str, Any]) -> dict[str, Any]:
+        clean = _normalize_rule(rule)
+        with self._lock:
+            rules = self._data.setdefault("rules", [])
+            for idx, existing in enumerate(rules):
+                if existing.get("id") == clean["id"]:
+                    rules[idx] = clean
+                    break
+            else:
+                rules.append(clean)
+        self.save()
+        return clean
+
+    def delete_rule(self, rule_id: str) -> None:
+        with self._lock:
+            self._data["rules"] = [r for r in self._data.get("rules", []) if r.get("id") != rule_id]
+        self.save()
+
+    def toggle_rule(self, rule_id: str, enabled: bool) -> None:
+        with self._lock:
+            for rule in self._data.get("rules", []):
+                if rule.get("id") == rule_id:
+                    rule["enabled"] = bool(enabled)
+        self.save()
+
+
+def _normalize_rule(rule: dict[str, Any]) -> dict[str, Any]:
+    conditions: dict[str, list[str]] = {}
+    for field, values in (rule.get("conditions") or {}).items():
+        if field not in CONDITIONS:
+            continue
+        if isinstance(values, str):
+            values = [v.strip() for v in values.split(",")]
+        cleaned = [str(v).strip() for v in (values or []) if str(v).strip()]
+        if cleaned:
+            conditions[field] = cleaned
+    return {
+        "id": str(rule.get("id") or f"rule-{uuid.uuid4().hex[:8]}"),
+        "name": str(rule.get("name") or "Untitled rule").strip(),
+        "enabled": bool(rule.get("enabled", True)),
+        "reason": str(rule.get("reason") or "").strip(),
+        "conditions": conditions,
+        "created": rule.get("created") or time.strftime("%Y-%m-%d"),
+    }
+
+
+# --- matching ---------------------------------------------------------------
+
+def _alert_field(alert: Any, field: str) -> str:
+    if field == "kind":
+        return str(getattr(alert, "kind", ""))
+    if field == "organization":
+        return f"{getattr(alert, 'org_id', '')} {getattr(alert, 'org_name', '')}"
+    if field == "instance_name":
+        return str(getattr(alert, "instance_name", ""))
+    if field == "host":
+        return str(getattr(alert, "host", ""))
+    if field == "region":
+        return f"{getattr(alert, 'region', '')} {getattr(alert, 'region_label', '')}"
+    if field == "status":
+        return str(getattr(alert, "status", ""))
+    return ""
+
+
+def _condition_matches(field: str, values: list[str], alert: Any) -> bool:
+    actual = _alert_field(alert, field).lower()
+    if field in ("kind", "status"):
+        return any(actual == str(v).strip().lower() for v in values)
+    if field == "region":
+        return any(str(v).strip().lower() in actual for v in values)
+    return any(str(v).strip().lower() in actual for v in values)
+
+
+def rule_matches(rule: dict[str, Any], alert: Any) -> bool:
+    """Every condition in the rule must match (AND)."""
+    conditions = rule.get("conditions") or {}
+    if not conditions:
+        return False  # an empty rule would swallow the whole queue
+    return all(_condition_matches(f, v, alert) for f, v in conditions.items())
+
+
+def first_match(settings: Settings, alert: Any) -> Optional[dict[str, Any]]:
+    for rule in settings.rules:
+        if rule.get("enabled") and rule_matches(rule, alert):
+            return rule
+    return None
+
+
+def preview(settings: Settings, rule: dict[str, Any], alerts: list[Any]) -> list[dict[str, Any]]:
+    """Which currently-firing alerts a rule would hide - shown before saving."""
+    clean = _normalize_rule(rule)
+    hits = []
+    for alert in alerts:
+        if rule_matches(clean, alert):
+            hits.append({
+                "kind": alert.kind,
+                "title": alert.title,
+                "instance_name": alert.instance_name,
+                "host": alert.host,
+                "org_name": alert.org_name,
+                "region": alert.region,
+            })
+    return hits