Handover
- The Confluence shift doc becomes the landing page: shift metadata, the
top-of-page checks, key updates with their Zendesk/Jira refs and status, and
the free-text comments. "Hand over shift" closes the shift, opens the next
one and carries the live items across, dropping anything done or marked
"remove at end of shift" - the retyping this replaces.
- The RunPod table on that page is read from live host state instead of being
copied in by hand, with the six-colour key preserved.
RunPod
- GraphQL client keyed on CX_RUNPOD_API_KEY. The old console login is kept as a
fallback but cannot run unattended: the account has 2FA, so Clerk verifies the
password and then asks for an emailed code and never issues a session. That is
the real cause of the "No active session found" failure, and the client now
says so instead of failing opaquely. TOTP is supported if the account moves to
an authenticator app.
- Hosts and their listing history are persisted, so "most problematic hosts" can
be ranked and each machine has a timeline of who listed or unlisted it, with
the Zendesk comment and the error hint.
- The unlisting emails are parsed for the error block (they arrive
quoted-printable) and classified into a likely cause and a next step.
Zendesk and Jira
- Unlisting raises a Zendesk ticket that follows the format of RunPod's own
email, keyed on the machine so one machine keeps one thread, posted as an
internal note.
- Jira is split in two: the Infrahub/OIE instance and the RunPod/RMA one, which
may be a different Atlassian site. Blank RunPod values fall back to the
defaults rather than failing.
Running in a container
- CX-Tools reads its keys from 1Password, which needs a desktop app. Config is a
dataclass whose lookups live in per-field default factories, so passing
CX_INFRAHUB_TOKEN/CX_INFRAINSIGHT_TOKEN in means those factories never run and
CX-Tools itself stays unmodified.
- CX-Tools reaches OpenStack with `docker exec <region>-osc`, so the image now
carries the Docker client (the static binary, not the docker.io package) and
compose mounts the host socket with group_add for it. Verified from inside the
container: live OpenStack and Infrahub calls both succeed.
Also fixes Settings, which read the environment at class-definition time and so
ignored anything set afterwards; a fresh Settings() silently returned stale
values. Caught by the Jira scoping tests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The single-file stdlib server became the limit: no way to track what had been
done about an alert, no accounts, and a UI that had to be hand-rolled in
template strings. This restructures it into something deployable.
Backend (FastAPI)
- app/ holds config, database, auth, delivery and the routers; triagelib keeps
the triage engine unchanged, so the validated screening and runbook logic is
untouched.
- Cases persist per alert fingerprint with a status workflow (investigating,
customer contacted, escalated to Infra, waiting, remediated, resolved, won't
fix, false positive), an assignee, notes and an append-only history. An alert
that stops and re-fires lands back on the same case and counts as a reopen.
- Suppression rules move from a JSON file into the database.
Auth
- Signed session cookies over PBKDF2 local accounts, plus an OIDC flow ready for
Authentik: users are created on first login and admin follows a group claim.
Local login can be switched off entirely once SSO is live.
Zendesk and Jira
- Delivery is now implemented, behind three gates: the integration must be
configured, its feature flag on, and CX_FEATURE_SEND_ENABLED on. A demo
instance leaves the last off and cannot mail anyone. Both search before
creating, so re-diagnosing an alert updates one ticket rather than opening
several, and a rolling daily cap stops a loop mailing everybody.
Deployment
- Multi-stage Dockerfile builds the bundle and serves it from the API origin.
- docker-compose for local and single-host use; Gitea Actions runs the tests,
builds the image and renders deploy/k8s with envsubst.
Two fixes found while testing: assigning a case returned a null assignee, and
add_event could leave an already-loaded history collection stale.
Known gap: the engine reaches OpenStack via `docker exec <region>-osc`, which
does not work in a pod without the CX-Tools containers alongside it.
docs/DEPLOYMENT.md sets out the three ways to close that.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Read-only triage for the Infrahub error alerts. Pulls the Prometheus alert
queue, re-checks each alert's condition against live state to separate real
work from noise, diagnoses it using the CX runbooks, and drafts the customer
comms with contacts resolved from Infrahub.
Findings from validating against production:
- "Suspected Rogue VM" fires on spare GPU capacity, not rogue VMs: In_Use_Gpus
equals the physical count on 71 of 75 firing hosts, so the rule reduces to
"this host has a free GPU". Verified against OpenStack on 10 hosts.
- "Exists in Infrahub but does not exist in OpenStack" matches every VM because
openstack_nova_server_status returns no series; excluded as a rule defect.
- Prometheus activeAt is reset several times a day by dips in the Resources
metric, so alert ages are recovered from ALERTS history instead.
Takes ~2,650 firing alerts down to ~20 that need a decision.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
