parham.monfared/cx-ui
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Split into a FastAPI backend and a React frontend, add case state and SSO
Some checks failed
build-and-deploy / test (push) Has been cancelled
Details
build-and-deploy / image (push) Has been cancelled
Details
build-and-deploy / deploy (push) Has been cancelled
Details

Browse Source 
The single-file stdlib server became the limit: no way to track what had been
done about an alert, no accounts, and a UI that had to be hand-rolled in
template strings. This restructures it into something deployable.

Backend (FastAPI)
- app/ holds config, database, auth, delivery and the routers; triagelib keeps
  the triage engine unchanged, so the validated screening and runbook logic is
  untouched.
- Cases persist per alert fingerprint with a status workflow (investigating,
  customer contacted, escalated to Infra, waiting, remediated, resolved, won't
  fix, false positive), an assignee, notes and an append-only history. An alert
  that stops and re-fires lands back on the same case and counts as a reopen.
- Suppression rules move from a JSON file into the database.

Auth
- Signed session cookies over PBKDF2 local accounts, plus an OIDC flow ready for
  Authentik: users are created on first login and admin follows a group claim.
  Local login can be switched off entirely once SSO is live.

Zendesk and Jira
- Delivery is now implemented, behind three gates: the integration must be
  configured, its feature flag on, and CX_FEATURE_SEND_ENABLED on. A demo
  instance leaves the last off and cannot mail anyone. Both search before
  creating, so re-diagnosing an alert updates one ticket rather than opening
  several, and a rolling daily cap stops a loop mailing everybody.

Deployment
- Multi-stage Dockerfile builds the bundle and serves it from the API origin.
- docker-compose for local and single-host use; Gitea Actions runs the tests,
  builds the image and renders deploy/k8s with envsubst.

Two fixes found while testing: assigning a case returned a null assignee, and
add_event could leave an already-loaded history collection stale.

Known gap: the engine reaches OpenStack via `docker exec <region>-osc`, which
does not work in a pod without the CX-Tools containers alongside it.
docs/DEPLOYMENT.md sets out the three ways to close that.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Parham Monfared
2026-08-06 07:11:28 +01:00
parent a039e0b5fd
commit 1262690276
68 changed files with 3839 additions and 2223 deletions
Download Patch File Download Diff File Expand all files Collapse all files

124
backend/tests/test_api.py Normal file
View File

@@ -0,0 +1,124 @@
"""API-level tests: auth gates, case lifecycle, suppression rules, send gating.
Runs against an in-memory database with CX-Tools stubbed out, so it needs no
credentials. Prometheus is only touched for cache warming, which is tolerant of
being unreachable.
"""
import os
import sys
import tempfile
sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
os.environ.update({
"CX_DATABASE_URL": f"sqlite:///{tempfile.mkdtemp()}/test.db",
"CX_BOOTSTRAP_ADMIN_EMAIL": "admin@localhost",
"CX_BOOTSTRAP_ADMIN_PASSWORD": "test-password",
"CX_SECRET_KEY": "test-secret",
"CX_FEATURE_SEND_ENABLED": "false",
})
from triagelib import cxbridge # noqa: E402
cxbridge.bootstrap = lambda: (_ for _ in ()).throw(cxbridge.BridgeError("stubbed"))
from fastapi.testclient import TestClient # noqa: E402
from app import auth as auth_mod # noqa: E402
from app.main import app # noqa: E402
FAILS = []
def expect(label, cond, got=""):
print((" PASS " if cond else " FAIL ") + label + ("" if cond else f" <- {got}"))
if not cond:
FAILS.append(label)
print("\nPASSWORDS AND SESSIONS")
h = auth_mod.hash_password("hunter2")
expect("correct password verifies", auth_mod.verify_password("hunter2", h))
expect("wrong password rejected", not auth_mod.verify_password("hunter3", h))
expect("hash is salted (two hashes differ)", auth_mod.hash_password("x") != auth_mod.hash_password("x"))
token = auth_mod.issue_session(7)
expect("session round-trips", auth_mod.read_session(token) == 7)
expect("tampered session rejected", auth_mod.read_session(token[:-4] + "aaaa") is None)
expect("garbage session rejected", auth_mod.read_session("not-a-token") is None)
expect("oidc state verifies", auth_mod.oidc_state_valid(auth_mod.oidc_state()))
expect("forged oidc state rejected", not auth_mod.oidc_state_valid("aaa.bbb"))
with TestClient(app) as client:
print("\nAUTH GATES")
expect("health is public", client.get("/api/health").status_code == 200)
expect("alerts need a session", client.get("/api/alerts").status_code == 401)
expect("cases need a session", client.get("/api/cases").status_code == 401)
expect("wrong password is 401", client.post(
"/api/auth/login", json={"email": "admin@localhost", "password": "no"}).status_code == 401)
login = client.post("/api/auth/login", json={"email": "admin@localhost", "password": "test-password"})
expect("login succeeds", login.status_code == 200, login.text[:120])
expect("bootstrap user is admin", login.json()["user"]["is_admin"])
expect("session works after login", client.get("/api/cases").status_code == 200)
print("\nCONFIG EXPOSURE")
cfg = client.get("/api/auth/me").json()["config"]
expect("send disabled by default", cfg["send_enabled"] is False, cfg)
expect("no secret leaks into public config",
not any("token" in k.lower() or "secret" in k.lower() for k in cfg), list(cfg))
print("\nSUPPRESSION RULES")
rule = {"name": "Modal ERROR churn", "reason": "known batch churn",
"conditions": {"kind": ["error"], "organization": ["modal"]}}
expect("admin can save a rule", client.post("/api/settings/rules", json=rule).status_code == 200)
expect("rule is persisted", len(client.get("/api/settings").json()["rules"]) == 1)
expect("a rule with no conditions is refused", client.post(
"/api/settings/rules", json={"name": "catch all", "conditions": {}}).status_code == 400)
expect("unknown condition fields are dropped", client.post(
"/api/settings/rules", json={"name": "bogus", "conditions": {"nope": ["x"]}}).status_code == 400)
print("\nSEND GATING")
status = client.get("/api/actions/status").json()
expect("send reported as disabled", status["send_enabled"] is False)
expect("zendesk reported as not ready", status["zendesk_ready"] is False)
blocked = client.post("/api/actions/zendesk", json={
"fingerprint": "does-not-exist", "to": "a@b.c", "subject": "s", "body": "b"})
expect("sending on an untracked alert is refused", blocked.status_code == 404, blocked.text[:120])
print("\nCASE LIFECYCLE")
from app.db import SessionLocal
from app.models import Case, CaseStatus
from app.services import add_event, set_status
db = SessionLocal()
case = Case(fingerprint="test-fp", kind="error", title="Instance in ERROR state", subject="vm-1")
db.add(case)
db.commit()
expect("new case starts open", case.is_open and case.status == CaseStatus.NEW)
set_status(db, case, CaseStatus.ESCALATED_INFRA, None, "INFRA-1")
expect("escalated is still open", case.is_open)
expect("status change is recorded", any(e.action == "status_changed" for e in case.events))
set_status(db, case, CaseStatus.RESOLVED, None)
expect("resolved closes the case", not case.is_open and case.closed_at is not None)
add_event(db, case, None, "note", "manual note")
db.commit()
expect("history is append-only and ordered newest first",
case.events[0].action in ("note", "status_changed"), [e.action for e in case.events])
payload = case.to_json(with_events=True)
expect("serialises for the API", payload["status"] == "resolved" and len(payload["events"]) >= 3)
db.close()
expect("bad status is rejected", client.post(
"/api/cases/test-fp/status", json={"status": "banana"}).status_code == 400)
expect("unknown case is 404", client.get("/api/cases/nope").status_code == 404)
print("\nLOGOUT")
client.post("/api/auth/logout")
expect("session is cleared", client.get("/api/cases").status_code == 401)
print("\n" + ("ALL CHECKS PASSED" if not FAILS else f"{len(FAILS)} CHECK(S) FAILED: {FAILS}"))
sys.exit(1 if FAILS else 0)

268
backend/tests/test_runbooks.py Normal file
View File

@@ -0,0 +1,268 @@
"""Runbook decision tests: fixtures shaped like real CX-Tools collector output."""
import sys, os
sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
from triagelib import alerts as A, cxbridge, runbooks
# --- stub the CX-Tools boundary so the decision logic can be tested alone ---
class FakeCx:
def status_pair_ok(self, ih, os_, task):
ih, os_ = (ih or "").upper(), (os_ or "").upper()
if ih == os_: return True
return (ih, os_) in {("HIBERNATED","SHELVED_OFFLOADED"), ("REBOOTING","HARD_REBOOT")}
def mismatch_parts(self, item):
return str(item.get("check","Mismatch")), str(item.get("detail",""))
def normalize_empty(self, v): return "" if v in (None,"","None") else str(v)
def first_present(self, m, *keys, default=""):
for k in keys:
if isinstance(m, dict) and k in m: return m[k]
return default
def public_ip_from_server(self, s): return (s or {}).get("_public_ip","")
def json_safe(self, o): return o
def map_region(self, r): return {"CANADA-1":"ca1","US-1":"us1","NORWAY-1":"no1"}.get(r,"")
STUB = {"host_health": {}, "gpu_census": {}, "failed_event": {}, "vm": {}, "host": {}}
cxbridge.cx = lambda: FakeCx()
cxbridge.host_health = lambda region, host: STUB["host_health"]
cxbridge.host_gpu_census = lambda region, host: STUB["gpu_census"]
cxbridge.failed_openstack_event = lambda region, osid, scan=5: STUB["failed_event"]
cxbridge.collect_vm = lambda t, **k: STUB["vm"]
cxbridge.collect_host = lambda h, **k: STUB["host"]
cxbridge.json_safe = lambda o: o
HEALTHY = {"ok": True, "nova_state": "up", "nova_status": "enabled", "ovs_alive": True,
"ovs_state": "UP", "uptime": "17:54", "aggregates": "agg", "bad_signals": []}
SICK = {**HEALTHY, "nova_state": "down", "ovs_state": "DOWN", "ovs_alive": False,
"bad_signals": ["Nova state is down", "OVS state is DOWN"]}
def vm(**over):
base = {"ok": True, "exit_code": 0, "mode": "vm", "infrahub_id": "123456",
"openstack_id": "9ec7a021-d741-484f-9387-4eaf8879fd77", "name": "test-vm",
"region": "ca1", "region_display": "CANADA-1", "ih_status": "ACTIVE", "os_status": "ACTIVE",
"task_state": "None", "host": "CA1-ESC8-040", "flavor": "n3-H100x8", "gpu_count": "8",
"floating_ip": "69.19.140.110", "created": "2026-07-01 10:00:00 UTC", "ssh_text": "reachable",
"ssh_raw": "reachable", "volumes_summary": "None", "openstack_fault": "None",
"faults": [], "ih_events": [], "ih_events_all": [], "mismatches": [], "warn_reasons": [],
"info_notes": [], "org_value": "8463 - Acme Corp", "owners": ["Lars <lars@simli.com>"],
"server": {"status": "ACTIVE"}, "infrahub": {"floating_ip": "69.19.140.110"}}
base.update(over); return base
def alert(name, **labels):
return A.from_labels({"alertname": name, **labels})
def run(name, labels, vmdata=None, hostdata=None, health=None, census=None, fevent=None, prom=None):
STUB.update({"vm": vmdata or {}, "host": hostdata or {}, "host_health": health or HEALTHY,
"gpu_census": census or {}, "failed_event": fevent or {}})
return runbooks.diagnose(alert(name, **labels), prom)
def check(label, cond, extra=""):
print(f" {'PASS' if cond else 'FAIL'} {label}" + (f" <- {extra}" if not cond and extra else ""))
return cond
fails = 0
def expect(label, cond, extra=""):
global fails
if not check(label, cond, extra): fails += 1
L_ERR = dict(openstack_id="9ec7a021-d741-484f-9387-4eaf8879fd77", region="CANADA-1",
instance_name="test-vm", organization="8463 - Acme Corp", status="ERROR")
print("\n[1] ERROR - creation failed, never reached ACTIVE, insufficient stock")
d = run("Instance in ERROR state in :flag-ca:CA-1", L_ERR,
vm(host="N/A", ih_status="ERROR", os_status="ERROR", server={"status": "ERROR"},
openstack_fault="No valid host was found. There are not enough hosts available"))
expect("matched no_valid_host fault", "scheduler could not place" in d.verdict.lower(), d.verdict)
expect("identified as never-ACTIVE", any("never placed on a host" in f.value for f in d.findings))
expect("chose the insufficient-stock template", any(x.template_id == "error_never_active" for x in d.drafts),
[x.template_id for x in d.drafts])
expect("mentions 7-day outreach window", any("7 days" in x.when for x in d.drafts))
expect("escalates to Infrastructure", any(a.owner == runbooks.INFRA for a in d.actions))
expect("contacts resolved", d.contacts.get("resolved"))
print("\n[2] ERROR - was ACTIVE, stale LVM on host")
d = run("Instance in ERROR state in :flag-ca:CA-1", L_ERR,
vm(ih_status="ERROR", os_status="ERROR",
faults=[["500", "Build of instance aborted: Failed to remove volume(s): lvremove -f /dev/nova-vg/x_disk", "2026-07-29"]]))
expect("matched lvremove fault", "stale LVM" in d.verdict, d.verdict)
expect("identified as previously ACTIVE", any("hypervisor is recorded" in f.value for f in d.findings))
expect("chose the was-ACTIVE template", any(x.template_id == "error_was_active" for x in d.drafts),
[x.template_id for x in d.drafts])
expect("assessment flags possible customer data", "customer data" in d.assessment)
print("\n[3] ERROR - NUMA/PCI fault, host is FULL (proves host fine)")
d = run("Instance in ERROR state in :flag-ca:CA-1", L_ERR,
vm(ih_status="ERROR", os_status="ERROR",
openstack_fault="Insufficient compute resources: Requested instance NUMA topology together with requested PCI devices cannot fit the given host NUMA topology; Claim pci failed."),
census={"ok": True, "total_gpus": 8, "instances": [{"name":"a"}]*4})
expect("ran the GPU census", any("GPUs allocated on host" in f.label for f in d.findings))
expect("concluded host is FULL", any("FULL" in f.detail for f in d.findings))
expect("marked the capacity check as already done", any(a.status == "done" for a in d.actions))
print("\n[3b] same fault, host NOT full -> must escalate")
d = run("Instance in ERROR state in :flag-ca:CA-1", L_ERR,
vm(ih_status="ERROR", os_status="ERROR",
openstack_fault="Claim pci failed."),
census={"ok": True, "total_gpus": 4, "instances": [{"name":"a"}]*2})
expect("raises a Jira for Infra", any("Jira" in a.text and a.owner == runbooks.INFRA for a in d.actions))
print("\n[4] SHUTOFF - billing notice only")
d = run("Instance in SHUTOFF state in :flag-ca:CA-1 for greater than 30 min",
dict(openstack_id="dc156762-3fa8-481d-89e0-12c2fb55e046", region="CANADA-1",
instance_name="energetic-galileo", organization="27358 - Zyra", status="SHUTOFF"),
vm(name="energetic-galileo", ih_status="SHUTOFF", os_status="SHUTOFF", server={"status":"SHUTOFF"}))
expect("verdict is customer-initiated", "customer-initiated" in d.verdict.lower(), d.verdict)
expect("used the shutoff snippet", any(x.template_id == "shutoff" for x in d.drafts))
expect("substituted the VM name", any("energetic-galileo" in x.body for x in d.drafts))
expect("no Infra escalation", not any(a.owner == runbooks.INFRA for a in d.actions))
print("\n[5] DELETING - server still in OpenStack, delete request found")
d = run("Instance in DELETING state in :flag-ca:CA-1 for greater than 30 min",
dict(openstack_id="13a855a4-c563-4350-9a8e-ffa9efa27d9e", region="CANADA-1",
instance_name="external-hs-h100", organization="10420 - Inceptions AI", status="DELETING"),
vm(ih_status="DELETING", os_status="ACTIVE", server={"status": "ACTIVE"},
ih_events_all=[["2026-07-29 21:40:00", "InstanceDeleteRequest", "Delete Instance Request Sent."]]))
expect("confirmed customer intent from events", any(a.status == "done" and "intent" in a.text for a in d.actions))
expect("tells CX to delete in OpenStack", any("Delete the server in OpenStack" in a.text for a in d.actions))
expect("offers both the notice and the ticket-closing reply",
sorted(x.template_id for x in d.drafts) == ["deleting", "deleting_resolved"],
[x.template_id for x in d.drafts])
expect("tells CX to close the Infrahub record too, not just the server",
any("InfraInsight" in a.text for a in d.actions), [a.text for a in d.actions])
print("\n[5c] DELETING - server exists but never reached a host")
d = run("Instance in DELETING state in :flag-ca:CA-1 for greater than 30 min",
dict(openstack_id="x", region="CANADA-1", status="DELETING"),
vm(ih_status="DELETING", os_status="ERROR", host="N/A",
server={"status": "ERROR"}, openstack_fault="No valid host was found."))
expect("flags that the build never completed", any("build never completed" in f.value for f in d.findings))
expect("still requires the InfraInsight close-out", any("InfraInsight" in a.text for a in d.actions))
print("\n[5b] DELETING - already gone from OpenStack, no delete event")
d = run("Instance in DELETING state in :flag-ca:CA-1 for greater than 30 min",
dict(openstack_id="x", region="CANADA-1", status="DELETING"),
vm(ih_status="DELETING", os_status="N/A", server={}, ih_events_all=[]))
expect("notes the server is already gone", any("already gone" in f.value for f in d.findings))
expect("routes attribution to DevOps", any(a.owner == runbooks.DEVOPS for a in d.actions))
print("\n[6] CREATING - never got an OpenStack ID")
d = run("Instance in CREATING state in :flag-ca:CA-1 for greater than 30min",
dict(openstack_id="None", region="CANADA-1", instance_name="vm-28070520-5d1f2",
organization="5574 - Nexgen", status="CREATING"),
vm(openstack_id="N/A", ih_status="CREATING", os_status="N/A", server={}, host="N/A"))
expect("verdict says never got an OpenStack ID", "never got an OpenStack ID" in d.verdict, d.verdict)
expect("used the creating template", any(x.template_id == "creating" for x in d.drafts))
expect("instructs deletion", any("Delete the stuck instance" in a.text for a in d.actions))
print("\n[7] HIBERNATING - sick host")
d = run("Instance in HIBERNATING state in :flag-ca:CA-1 for greater than 120min",
dict(openstack_id="28171e6f", region="CANADA-1", instance="CA1-ESC812-211",
instance_name="apt25-prod", status="HIBERNATING"),
vm(ih_status="HIBERNATING", os_status="ACTIVE"), health=SICK)
expect("verdict blames the host", "host problem" in d.verdict, d.verdict)
expect("escalates to Infra with the bad signals", any(a.owner == runbooks.INFRA and "OVS" in a.text for a in d.actions))
expect("still drives the shelve", any("shelve" in a.text.lower() for a in d.actions))
print("\n[8] Suspected Rogue VM - host with two different mismatches")
host_result = {"ok": True, "mode": "host", "host": "CA1-ESC8-068", "region": "ca1", "server_count": 3,
"hypervisor": {"state": "up", "status": "enabled"}, "ovs": {"alive": True, "state": "UP"},
"instances": [
{"idx": 1, "name": "vm-hib-shutoff", "infrahub_id": "1", "openstack_id": "a", "ih_status": "HIBERNATED",
"os_status": "SHUTOFF", "host": "CA1-ESC8-068", "mismatches": [{"check": "State", "detail": "IH HIBERNATED vs OS SHUTOFF"}],
"warn_reasons": ["mismatch detected"], "org_value": "1 - A", "owners": ["a@x.com"]},
{"idx": 2, "name": "vm-hib-active", "infrahub_id": "2", "openstack_id": "b", "ih_status": "HIBERNATED",
"os_status": "ACTIVE", "host": "CA1-ESC8-068", "mismatches": [{"check": "State", "detail": "IH HIBERNATED vs OS ACTIVE"}],
"warn_reasons": ["mismatch detected"], "org_value": "2 - B", "owners": ["b@x.com"]},
{"idx": 3, "name": "tempest-thing", "tempest": True, "ih_status": "N/A", "os_status": "ACTIVE",
"mismatches": [{"check": "Infrahub Missing", "detail": "not in Infrahub"}], "warn_reasons": []},
]}
d = run(":ninja:Suspected Rogue VM", dict(instance="CA1-ESC8-068"), hostdata=host_result)
expect("counted 2 of 3 as mismatched", "2 of 3" in d.verdict, d.verdict)
expect("ignored the tempest instance", d.evidence.get("ignored_tempest") == 1)
expect("HIBERNATED/SHUTOFF -> Windmill stale-image cleanup", any("Windmill" in a.text for a in d.actions))
expect("HIBERNATED/ACTIVE -> sync-error comms", any(x.template_id == "sync_state" for x in d.drafts))
expect("actions are scoped per instance", any(a.text.startswith("[vm-hib-active]") for a in d.actions))
print("\n[9] Suspected Rogue VM - clean host")
d = run(":ninja:Suspected Rogue VM", dict(instance="CA1-ESC8-068"),
hostdata={**host_result, "instances": [{"idx":1,"name":"ok","ih_status":"ACTIVE","os_status":"ACTIVE",
"mismatches": [], "warn_reasons": []}]})
expect("verdict says no mismatch", "No Infrahub/OpenStack mismatch" in d.verdict, d.verdict)
expect("redirects to the InfraInsight host query", any("InfraInsight" in a.text for a in d.actions))
print("\n[10] Duplicated IPs - one DELETING claimant, one with a wrong Infrahub IP")
multi = {"ok": True, "mode": "multi_vm", "instances": [
{"name": "old-vm", "infrahub_id": "1", "openstack_id": "a", "ih_status": "DELETING", "os_status": "ACTIVE",
"server": {"status": "ACTIVE", "_public_ip": "69.19.137.135"}, "infrahub": {"floating_ip": "69.19.137.135"},
"org_value": "1 - A", "owners": ["a@x.com"]},
{"name": "new-vm", "infrahub_id": "2", "openstack_id": "b", "ih_status": "ACTIVE", "os_status": "ACTIVE",
"server": {"status": "ACTIVE", "_public_ip": "69.19.140.9"}, "infrahub": {"floating_ip": "69.19.137.135"},
"org_value": "2 - B", "owners": ["b@x.com"]},
]}
class FakeProm:
def resources_by_floating_ip(self, fip):
return [{"instance_name": "preprod-vm", "status": "ACTIVE", "region": "CANADA-1", "environment": "preprod"}]
d = run(":awkward:Duplicated IPs", dict(floating_ip="69.19.137.135"), vmdata=multi, prom=FakeProm())
expect("found 2 claimants needing correction", "2 of 2" in d.verdict, d.verdict)
expect("DELETING claimant -> delete it", any("[old-vm]" in a.text and "stuck DELETING" in a.text for a in d.actions))
expect("wrong-IP claimant -> Scenario #2", any("Scenario #2" in f.value for f in d.findings))
expect("Scenario #2 comms filled with the real IP", any(
x.template_id == "dupip_corrected" and "69.19.140.9" in x.body for x in d.drafts))
expect("an unset sign-off name is reported rather than left as a placeholder", any(
"AGENT_NAME" in x.unfilled for x in d.drafts if x.template_id == "dupip_corrected"))
expect("surfaced the PreProd claimant from Prometheus", any("preprod-vm" in f.label for f in d.findings))
expect("asks for a re-check after 5-10 min", any("5-10 minutes" in a.text for a in d.actions))
print("\n[11] Problem with Total GPUs - customers on host")
d = run("Problem with Total GPUs in a System", dict(instance="CA1-ESC8-111", region="CANADA-1", gpu_name="B200-SXM"),
census={"ok": True, "total_gpus": 6, "instances": [
{"name": "cust-vm", "status": "ACTIVE", "flavor": "n3-B200x6", "gpus": "6", "openstack_id": "z"}]})
expect("verdict names the host", "CA1-ESC8-111" in d.verdict, d.verdict)
expect("lists the affected instance", any("cust-vm" in f.label for f in d.findings))
expect("flags host-maintenance comms", any(a.kind == "comms" for a in d.actions))
expect("escalates a Jira to Infra", any("Jira" in a.text and a.owner == runbooks.INFRA for a in d.actions))
expect("notes there is no approved template", any("no approved customer template" in n for n in d.notes))
expect("drafts nothing", not d.drafts)
print("\n[12] K8s instance name is flagged")
d = run("Instance in ERROR state in :flag-ca:CA-1", {**L_ERR, "instance_name": "hyperstack-minion-lydhjev"},
vm(host="N/A", ih_status="ERROR", os_status="ERROR"))
expect("noted the likely K8s node", any("Kubernetes" in n for n in d.notes))
print("\n[13] GPU sockets - every physical GPU accounted for")
from triagelib.runbooks import _gpu_slots
import collections as _c
R289 = [{"name": "luminous-hubble", "gpus": "1", "linked": True, "match": True, "ih_status": "ACTIVE", "os_status": "ACTIVE"},
{"name": "vm832adbe203242", "gpus": "1", "linked": True, "match": True, "ih_status": "ACTIVE", "os_status": "ACTIVE"},
{"name": "noble-maxwell", "gpus": "1", "linked": True, "match": True, "ih_status": "ACTIVE", "os_status": "ACTIVE"},
{"name": "clever-schrodinger", "gpus": "2", "linked": True, "match": True, "ih_status": "ACTIVE", "os_status": "ACTIVE"}]
# Real numbers from CA1-ESC812-289: 8 physical, 7 reported in use, 4 VMs on 5 GPUs.
s = _gpu_slots({"physical": 8, "in_use_metric": 7, "spare_capacity_artifact": False}, R289)
c = _c.Counter(x["kind"] for x in s)
expect("8 sockets drawn for an 8-GPU host", len(s) == 8, len(s))
expect("5 named + 2 unaccounted + 1 free", (c["vm"], c["unaccounted"], c["free"]) == (5, 2, 1), dict(c))
s = _gpu_slots({"physical": 8, "in_use_metric": 8, "spare_capacity_artifact": True}, R289)
c = _c.Counter(x["kind"] for x in s)
expect("artifact host shows spare sockets as free, not unaccounted",
(c["vm"], c["unaccounted"], c["free"]) == (5, 0, 3), dict(c))
s = _gpu_slots({"physical": None, "in_use_metric": 2, "spare_capacity_artifact": False},
[{"name": "basilica", "gpus": "1", "linked": True, "match": True, "ih_status": "ACTIVE", "os_status": "ACTIVE"}])
c = _c.Counter(x["kind"] for x in s)
expect("unknown socket count degrades gracefully", (c["vm"], c["unaccounted"]) == (1, 1), dict(c))
s = _gpu_slots({"physical": 8, "in_use_metric": 8, "spare_capacity_artifact": False},
[{"name": "ghost", "gpus": "8", "linked": False, "match": False,
"ih_status": "not in Infrahub", "os_status": "ACTIVE"}])
expect("a VM with no Infrahub record still fills its sockets and is flagged",
len(s) == 8 and all(x["kind"] == "vm" and not x["linked"] for x in s))
s = _gpu_slots({"physical": 8, "in_use_metric": 5, "spare_capacity_artifact": False}, R289)
expect("no negative slots when in_use is below what VMs claim", len(s) >= 5 and all(
x["kind"] in ("vm", "free", "unaccounted") for x in s), len(s))
print(f"\n{'ALL CHECKS PASSED' if not fails else str(fails) + ' CHECK(S) FAILED'}")
sys.exit(1 if fails else 0)

290
backend/tests/test_screening.py Normal file
View File

@@ -0,0 +1,290 @@
"""Screening, exclusion, categorisation and ordering tests.
These cover the noise-vs-real decisions, which are what keeps the queue small.
No network and no CX-Tools: snapshots are synthetic.
"""
import datetime as dt
import os
import sys
sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
from triagelib import alerts as A, screening
class Snap:
"""Stands in for prometheus.StateSnapshot."""
loaded = True
def __init__(self, **kw):
self.by_openstack_id = {}
self.by_instance_name = {}
self.fip_counts = {}
self.rogue_delta = {}
self.total_gpus = {}
self.in_use_gpus = {}
self.resources_by_host = {}
self.broken_inputs = []
self.unattributed_active = 0
self.unattributed_active_gpus = 0
self.__dict__.update(kw)
def al(name, **labels):
return A.from_labels({"alertname": name, **labels})
def aged(alert, minutes):
alert.active_at = dt.datetime.now(dt.timezone.utc) - dt.timedelta(minutes=minutes)
return alert
FAILS = []
def expect(label, cond, got=""):
print((" PASS " if cond else " FAIL ") + label + ("" if cond else f" <- {got}"))
if not cond:
FAILS.append(label)
OSID = "abc-123"
ERROR_ALERT = "Instance in ERROR state in :flag-ca:CA-1"
SHUTOFF_ALERT = "Instance in SHUTOFF state in :flag-ca:CA-1 for greater than 30 min"
print("\nSTATE ALERTS - does the condition still hold?")
a = al(ERROR_ALERT, openstack_id=OSID, status="ERROR", region="CANADA-1")
a.screen = screening.screen(a, Snap(by_openstack_id={OSID: {"status": "ERROR"}}))
expect("Infrahub still ERROR -> real", a.screen["verdict"] == screening.REAL, a.screen)
a = al(ERROR_ALERT, openstack_id=OSID, status="ERROR", region="CANADA-1")
a.screen = screening.screen(a, Snap(by_openstack_id={OSID: {"status": "ACTIVE"}}))
expect("recovered to ACTIVE -> resolved",
a.screen["verdict"] == screening.RESOLVED and "ACTIVE" in a.screen["reason"], a.screen)
a = al(SHUTOFF_ALERT, openstack_id=OSID, status="SHUTOFF")
a.screen = screening.screen(a, Snap())
expect("record gone from Infrahub -> resolved", a.screen["verdict"] == screening.RESOLVED, a.screen)
a = al("Instance in CREATING state in :flag-ca:CA-1 for greater than 30min",
openstack_id="None", instance_name="vm-x", status="CREATING")
a.screen = screening.screen(a, Snap())
expect("CREATING with no OpenStack ID -> real, not 'resolved'",
a.screen["verdict"] == screening.REAL, a.screen)
a = al(ERROR_ALERT, openstack_id=OSID)
a.screen = screening.screen(a, Snap(by_openstack_id={OSID: {"status": "ERROR"}}))
expect("no status label -> unverified but kept", a.screen["verdict"] == screening.UNVERIFIED, a.screen)
print("\nDUPLICATED IPs")
a = al(":awkward:Duplicated IPs", floating_ip="1.2.3.4")
a.screen = screening.screen(a, Snap(fip_counts={"1.2.3.4": 1}))
expect("one claimant left -> resolved", a.screen["verdict"] == screening.RESOLVED, a.screen)
a.screen = screening.screen(a, Snap(fip_counts={"1.2.3.4": 3}))
expect("three claimants -> real", a.screen["verdict"] == screening.REAL and "3 VMs" in a.screen["reason"], a.screen)
a.screen = screening.screen(a, Snap(fip_counts={"9.9.9.9": 2}))
expect("IP held by nobody -> resolved", a.screen["verdict"] == screening.RESOLVED, a.screen)
print("\nSUSPECTED ROGUE VM - per-host GPU accounting gap")
a = al(":ninja:Suspected Rogue VM", instance="CA1-ESC8-068")
a.screen = screening.screen(a, Snap(rogue_delta={"CA1-ESC8-068": 0.0}))
expect("gap closed -> resolved", a.screen["verdict"] == screening.RESOLVED, a.screen)
a.screen = screening.screen(a, Snap(rogue_delta={"CA1-ESC8-068": 4.0},
resources_by_host={"CA1-ESC8-068": [{}] * 4}))
expect("gap of 4 GPUs -> real", a.screen["verdict"] == screening.REAL, a.screen)
expect("reason quantifies the gap", "4 GPU(s)" in a.screen["reason"], a.screen["reason"])
a.screen = screening.screen(a, Snap(rogue_delta={"other-host": 4.0}, total_gpus={"x": 8}))
expect("no data for the host -> unverified, still actionable",
a.screen["verdict"] == screening.UNVERIFIED and a.screen["actionable"], a.screen)
print("\nTOTAL GPUs")
a = al("Problem with Total GPUs in a System", instance="h1", gpu_name="B200-SXM")
a.screen = screening.screen(a, Snap(total_gpus={"h1": 8}))
expect("full complement of 8 -> resolved", a.screen["verdict"] == screening.RESOLVED, a.screen)
a.screen = screening.screen(a, Snap(total_gpus={"h1": 6}, in_use_gpus={"h1": 6}))
expect("6 GPUs -> real", a.screen["verdict"] == screening.REAL, a.screen)
print("\nSUPPRESSION RULES - what used to be hardcoded is now user-editable")
from triagelib import settings as settings_mod
import tempfile, os as _os
_tmp = _os.path.join(tempfile.mkdtemp(), "settings.json")
CFG = settings_mod.Settings(_tmp)
a = al(SHUTOFF_ALERT, openstack_id=OSID, status="SHUTOFF",
organization="3491 - luis.sarabando+runpod@nexgencloud.coms-Organization")
a.screen = screening.screen(a, Snap(by_openstack_id={OSID: {"status": "SHUTOFF"}}), CFG)
expect("default rule hides internal nexgencloud orgs",
a.screen["verdict"] == screening.SUPPRESSED, a.screen)
expect("suppression names the rule that did it", "Internal NexGen" in a.screen["reason"], a.screen["reason"])
a = al("Instance in SHUTOFF state in :flag-no:NO-1 for greater than 30 min", openstack_id=OSID,
status="SHUTOFF", instance="no1-stor-runpod03", instance_name="no1-stor-runpod03",
organization="99 - Real Customer")
a.screen = screening.screen(a, Snap(by_openstack_id={OSID: {"status": "SHUTOFF"}}), CFG)
expect("default rule hides runpod storage nodes", a.screen["verdict"] == screening.SUPPRESSED, a.screen)
# The combinational case the team asked for: type AND organisation.
CFG.upsert_rule({"name": "Modal ERROR churn", "reason": "known batch churn",
"conditions": {"kind": ["error"], "organization": ["modal"]}})
hit = al(ERROR_ALERT, openstack_id=OSID, status="ERROR", organization="19417 - colin@modal.coms-Organization")
hit.screen = screening.screen(hit, Snap(by_openstack_id={OSID: {"status": "ERROR"}}), CFG)
expect("error + modal is suppressed", hit.screen["verdict"] == screening.SUPPRESSED, hit.screen)
miss = al(ERROR_ALERT, openstack_id=OSID, status="ERROR", organization="123 - Someone Else")
miss.screen = screening.screen(miss, Snap(by_openstack_id={OSID: {"status": "ERROR"}}), CFG)
expect("error from another org is NOT suppressed", miss.screen["verdict"] != screening.SUPPRESSED, miss.screen)
other = al(SHUTOFF_ALERT, openstack_id=OSID, status="SHUTOFF",
organization="19417 - colin@modal.coms-Organization")
other.screen = screening.screen(other, Snap(by_openstack_id={OSID: {"status": "SHUTOFF"}}), CFG)
expect("modal SHUTOFF is NOT suppressed - both conditions must match",
other.screen["verdict"] != screening.SUPPRESSED, other.screen)
empty = {"name": "catch all", "conditions": {}}
expect("a rule with no conditions never matches", not settings_mod.rule_matches(
settings_mod._normalize_rule(empty), hit))
expect("rules survive a reload", settings_mod.Settings(_tmp).rules and any(
r["name"] == "Modal ERROR churn" for r in settings_mod.Settings(_tmp).rules))
print("\nAGE DEMOTIONS")
a = aged(al("Problem with Total GPUs in a System", instance="h1"), 7 * 24 * 60)
a.screen = screening.screen(a, Snap(total_gpus={"h1": 6}))
expect("firing 7 days -> chronic", a.screen["verdict"] == screening.CHRONIC, a.screen)
a = aged(al("Problem with Total GPUs in a System", instance="h1"), 60)
a.screen = screening.screen(a, Snap(total_gpus={"h1": 6}))
expect("firing 1 hour -> stays real", a.screen["verdict"] == screening.REAL, a.screen)
print("\nFAIL-SAFE BEHAVIOUR")
a = al(ERROR_ALERT, openstack_id=OSID, status="ERROR")
a.state = "pending"
a.for_seconds = 1800
a.screen = screening.screen(a, Snap(by_openstack_id={OSID: {"status": "ERROR"}}))
expect("pending -> screened out", a.screen["verdict"] == screening.PENDING and not a.screen["actionable"], a.screen)
a = al(ERROR_ALERT, openstack_id=OSID, status="ERROR")
a.screen = screening.screen(a, None)
expect("no snapshot -> unverified but NOT hidden",
a.screen["verdict"] == screening.UNVERIFIED and a.screen["actionable"], a.screen)
warnings = screening.health_warnings(Snap(broken_inputs=["openstack_nova_server_status"]))
expect("empty nova metric raises a monitoring warning",
len(warnings) == 1 and "openstack_nova_server_status" in warnings[0], warnings)
print("\nEXCLUSION AND TAB ROUTING")
ex = al("Exists in Infrahub but does not exist in OpenStack", openstack_id=OSID)
expect("orphan spam excluded outright", A.is_excluded(ex) and not A.cx_relevant(ex))
expect("node-exporter -> Infrastructure tab",
A.category("HostSwapIsFillingUp", "node-exporter-rules.yml") == "node")
expect("ceph -> Infrastructure tab", A.category("CephOsdDown", "ceph-rules.yml") == "infra")
expect("regional Infrahub rule -> CX tab",
A.category(ERROR_ALERT, "infrahub-rules-CA1.yml") == "cx")
expect("main Infrahub rule -> CX tab",
A.category(":ninja:Suspected Rogue VM", "infrahub-rules.yml") == "cx")
expect("status-mismatch rule classified",
A.classify("Openstack status=ACTIVE and Infrahub status!=ACTIVE in :flag-ca:CANADA-1 for greater "
"than 30min") == "status_mismatch")
expect("hibernation-failure rule maps to HIBERNATING",
A.classify("Failure of Hibernation on Infrahub in :flag-ca:CANADA-1 for greater than 30min") == "hibernating")
expect("orphan VM rule classified", A.classify(":pirate_flag:Suspected Orphan VM") == "orphan_vm")
print("\nORDERING AND GROUPING")
def real(minutes, name=ERROR_ALERT, **labels):
x = aged(al(name, openstack_id="o%d" % minutes, status="ERROR", **labels), minutes)
x.screen = {"actionable": True, "verdict": "real", "label": "needs action", "reason": ""}
return x
ages = [x["age_minutes"] for x in A.group_alerts([real(500), real(10), real(100), real(9331)])[0]["alerts"]]
expect("newest first, oldest at the bottom", ages == [10, 100, 500, 9331], ages)
unknown = real(50)
unknown.active_at = None
ages = [x["age_minutes"] for x in A.group_alerts([unknown, real(200), real(5)])[0]["alerts"]]
expect("unknown start time sorts last", ages == [5, 200, None], ages)
groups = A.group_alerts([real(5), real(6, ":ninja:Suspected Rogue VM", instance="h1")])
expect("focus order puts rogue VM before ERROR", [g["kind"] for g in groups][0] == "rogue_vm",
[g["kind"] for g in groups])
noisy = real(7)
noisy.screen = {"actionable": False, "verdict": "resolved", "label": "already resolved", "reason": ""}
group = A.group_alerts([real(5), noisy])[0]
expect("group counts action vs noise separately",
group["actionable"] == 1 and group["noise"] == 1, group)
expect("age_text renders days", real(9331).age_text == "6d 11h", real(9331).age_text)
expect("age_text renders hours", real(431).age_text == "7h 11m", real(431).age_text)
expect("age_text renders minutes", real(7).age_text == "7m", real(7).age_text)
print("\nTRUE AGE - activeAt reset by pipeline dips")
# activeAt says 7h; ALERTS history says 7 days. The true value must win.
a = real(431)
a.true_age_minutes, a.true_age_capped = 7 * 24 * 60, False
expect("effective age prefers the recovered duration", a.effective_age_minutes == 10080, a.effective_age_minutes)
expect("reset is detected", a.age_is_reset)
expect("raw activeAt still reported", a.age_text == "7h 11m", a.age_text)
expect("effective text renders days", a.effective_age_text == "7d", a.effective_age_text)
a.screen = screening.screen(a, Snap(by_openstack_id={"o431": {"status": "ERROR"}}))
expect("7-day ERROR -> overdue (runbook says contact within 24h), not chronic",
a.screen["verdict"] == screening.OVERDUE, a.screen)
expect("overdue stays in the actionable queue", a.screen["actionable"])
# A kind with no runbook SLA still demotes to chronic, and explains the reset.
g = al("Problem with Total GPUs in a System", instance="h9")
g.active_at = dt.datetime.now(dt.timezone.utc) - dt.timedelta(minutes=431)
g.true_age_minutes, g.true_age_capped = 7 * 24 * 60, False
g.screen = screening.screen(g, Snap(total_gpus={"h9": 6}))
expect("no-SLA kind, 7 days -> chronic", g.screen["verdict"] == screening.CHRONIC, g.screen)
expect("chronic reason explains the activeAt reset", "pipeline dip" in g.screen["detail"], g.screen["detail"])
print("\nVALIDATION FINDINGS - rogue VM rule defect")
r = al(":ninja:Suspected Rogue VM", instance="CA1-ESC8-057")
r.screen = screening.screen(r, Snap(rogue_delta={"CA1-ESC8-057": 1.0},
in_use_gpus={"CA1-ESC8-057": 8.0},
total_gpus={"CA1-ESC8-057": 8.0},
resources_by_host={"CA1-ESC8-057": [{}] * 5}))
expect("In_Use == Total -> rule defect, not a rogue VM",
r.screen["verdict"] == screening.RULE_DEFECT, r.screen)
expect("rule defect is screened out of the queue", not r.screen["actionable"])
expect("reason names it as spare capacity", "free GPU" in r.screen["reason"], r.screen["reason"])
r2 = al(":ninja:Suspected Rogue VM", instance="CA1-ESC812-289")
r2.screen = screening.screen(r2, Snap(rogue_delta={"CA1-ESC812-289": 2.0},
in_use_gpus={"CA1-ESC812-289": 7.0},
resources_by_host={"CA1-ESC812-289": [{}] * 4}))
expect("In_Use with no Total reading -> still a real gap",
r2.screen["verdict"] == screening.REAL, r2.screen)
r3 = al(":ninja:Suspected Rogue VM", instance="h3")
r3.screen = screening.screen(r3, Snap(rogue_delta={"h3": 3.0}, in_use_gpus={"h3": 9.0},
total_gpus={"h3": 8.0}, resources_by_host={"h3": [{}]}))
expect("In_Use != Total -> real gap", r3.screen["verdict"] == screening.REAL, r3.screen)
b = real(431)
b.true_age_minutes, b.true_age_capped = 7 * 24 * 60, True
expect("window-capped age marked with +", b.effective_age_text == "7d+", b.effective_age_text)
c = real(120)
c.true_age_minutes, c.true_age_capped = 130, False
expect("small drift is not flagged as a reset", not c.age_is_reset)
expect("no true age -> falls back to activeAt", real(90).effective_age_minutes == 90)
# Ordering must use the recovered duration, not activeAt.
old, new = real(431), real(430)
old.true_age_minutes = 7 * 24 * 60
new.true_age_minutes = 30
order = [x["true_age_minutes"] for x in A.group_alerts([old, new])[0]["alerts"]]
expect("true age drives ordering, not activeAt", order == [30, 10080], order)
dips = screening.health_warnings(Snap(pipeline_dips=[
{"start": 0, "end": __import__("time").time() - 600, "minutes": 6, "low": 1359, "normal": 4374}]))
expect("pipeline dip raises a warning", len(dips) == 1 and "1359 of ~4374" in dips[0], dips)
print("\n" + ("ALL CHECKS PASSED" if not FAILS else f"{len(FAILS)} CHECK(S) FAILED: {FAILS}"))
sys.exit(1 if FAILS else 0)