Parham Monfared
8892144e0a
Handover - The Confluence shift doc becomes the landing page: shift metadata, the top-of-page checks, key updates with their Zendesk/Jira refs and status, and the free-text comments. "Hand over shift" closes the shift, opens the next one and carries the live items across, dropping anything done or marked "remove at end of shift" - the retyping this replaces. - The RunPod table on that page is read from live host state instead of being copied in by hand, with the six-colour key preserved. RunPod - GraphQL client keyed on CX_RUNPOD_API_KEY. The old console login is kept as a fallback but cannot run unattended: the account has 2FA, so Clerk verifies the password and then asks for an emailed code and never issues a session. That is the real cause of the "No active session found" failure, and the client now says so instead of failing opaquely. TOTP is supported if the account moves to an authenticator app. - Hosts and their listing history are persisted, so "most problematic hosts" can be ranked and each machine has a timeline of who listed or unlisted it, with the Zendesk comment and the error hint. - The unlisting emails are parsed for the error block (they arrive quoted-printable) and classified into a likely cause and a next step. Zendesk and Jira - Unlisting raises a Zendesk ticket that follows the format of RunPod's own email, keyed on the machine so one machine keeps one thread, posted as an internal note. - Jira is split in two: the Infrahub/OIE instance and the RunPod/RMA one, which may be a different Atlassian site. Blank RunPod values fall back to the defaults rather than failing. Running in a container - CX-Tools reads its keys from 1Password, which needs a desktop app. Config is a dataclass whose lookups live in per-field default factories, so passing CX_INFRAHUB_TOKEN/CX_INFRAINSIGHT_TOKEN in means those factories never run and CX-Tools itself stays unmodified. - CX-Tools reaches OpenStack with `docker exec <region>-osc`, so the image now carries the Docker client (the static binary, not the docker.io package) and compose mounts the host socket with group_add for it. Verified from inside the container: live OpenStack and Infrahub calls both succeed. Also fixes Settings, which read the environment at class-definition time and so ignored anything set afterwards; a fresh Settings() silently returned stale values. Caught by the Jira scoping tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CX Triage
Turns the Infrahub error-alert firehose into a short list of things that actually need doing — then helps you do them.
Python/FastAPI backend, React frontend, PostgreSQL for case state.
What it does
- Pulls the alert queue from Prometheus.
- Screens every alert by re-checking its condition against live state, so noise and already-resolved alerts drop out.
- Diagnoses what is left using the CX runbooks, reconciling Infrahub against OpenStack through the CX-Tools collectors.
- Drafts the customer email with contacts resolved from Infrahub, and the Jira escalation with the evidence attached.
- Tracks each case — who owns it, what was done, what was sent.
On live data this takes roughly 2,650 firing alerts down to ~20 that need a decision.
Findings that shaped it
Validated against production, not assumed:
Suspected Rogue VMis measuring spare capacity.In_Use_Gpusequals the physical GPU count on 71 of 75 firing hosts, so the rule reduces to "this host has a free GPU". Checked against OpenStack on 10 hosts: Infrahub and OpenStack agreed exactly on all of them. Those alerts are flagged as a rule defect.Exists in Infrahub but does not exist in OpenStackmatches every VM, becauseopenstack_nova_server_statusreturns no series. Excluded outright. The same gap meansSuspected Orphan VMcannot fire at all.- Prometheus alert ages are unreliable here. The
Resourcesmetric drops most of its series several times a day; every alert alive at the time resolves and re-fires, resettingactiveAt. Ages are recovered fromALERTShistory instead.
Read-only by design
The app queries and advises. It never deletes, shelves, or edits a VM — those stay copy-a-command. The only thing it can send is a Zendesk ticket or a Jira issue, behind three gates and a confirm step. See docs/INTEGRATIONS.md.
Run it
cp .env.example .env
docker compose up --build
More: docs/DEPLOYMENT.md · docs/LINKAGE.md · docs/PLAN.md
Layout
backend/
app/ FastAPI: config, db, models, auth, delivery, routers
triagelib/ the triage engine (screening, runbooks, comms, linkage)
tests/ pure-logic tests — no network, no CX-Tools
frontend/ React + TypeScript + Vite
deploy/k8s/ manifests, rendered by the pipeline
.gitea/workflows/ test → build → deploy
Tests
cd backend && python tests/test_runbooks.py && python tests/test_screening.py