Handover
- The Confluence shift doc becomes the landing page: shift metadata, the
top-of-page checks, key updates with their Zendesk/Jira refs and status, and
the free-text comments. "Hand over shift" closes the shift, opens the next
one and carries the live items across, dropping anything done or marked
"remove at end of shift" - the retyping this replaces.
- The RunPod table on that page is read from live host state instead of being
copied in by hand, with the six-colour key preserved.
RunPod
- GraphQL client keyed on CX_RUNPOD_API_KEY. The old console login is kept as a
fallback but cannot run unattended: the account has 2FA, so Clerk verifies the
password and then asks for an emailed code and never issues a session. That is
the real cause of the "No active session found" failure, and the client now
says so instead of failing opaquely. TOTP is supported if the account moves to
an authenticator app.
- Hosts and their listing history are persisted, so "most problematic hosts" can
be ranked and each machine has a timeline of who listed or unlisted it, with
the Zendesk comment and the error hint.
- The unlisting emails are parsed for the error block (they arrive
quoted-printable) and classified into a likely cause and a next step.
Zendesk and Jira
- Unlisting raises a Zendesk ticket that follows the format of RunPod's own
email, keyed on the machine so one machine keeps one thread, posted as an
internal note.
- Jira is split in two: the Infrahub/OIE instance and the RunPod/RMA one, which
may be a different Atlassian site. Blank RunPod values fall back to the
defaults rather than failing.
Running in a container
- CX-Tools reads its keys from 1Password, which needs a desktop app. Config is a
dataclass whose lookups live in per-field default factories, so passing
CX_INFRAHUB_TOKEN/CX_INFRAINSIGHT_TOKEN in means those factories never run and
CX-Tools itself stays unmodified.
- CX-Tools reaches OpenStack with `docker exec <region>-osc`, so the image now
carries the Docker client (the static binary, not the docker.io package) and
compose mounts the host socket with group_add for it. Verified from inside the
container: live OpenStack and Infrahub calls both succeed.
Also fixes Settings, which read the environment at class-definition time and so
ignored anything set afterwards; a fresh Settings() silently returned stale
values. Caught by the Jira scoping tests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Read-only triage for the Infrahub error alerts. Pulls the Prometheus alert
queue, re-checks each alert's condition against live state to separate real
work from noise, diagnoses it using the CX runbooks, and drafts the customer
comms with contacts resolved from Infrahub.
Findings from validating against production:
- "Suspected Rogue VM" fires on spare GPU capacity, not rogue VMs: In_Use_Gpus
equals the physical count on 71 of 75 firing hosts, so the rule reduces to
"this host has a free GPU". Verified against OpenStack on 10 hosts.
- "Exists in Infrahub but does not exist in OpenStack" matches every VM because
openstack_nova_server_status returns no series; excluded as a rule defect.
- Prometheus activeAt is reset several times a day by dips in the Resources
metric, so alert ages are recovered from ALERTS history instead.
Takes ~2,650 firing alerts down to ~20 that need a decision.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
