Add shift handover and RunPod, and make CX-Tools work in a container

Handover
- The Confluence shift doc becomes the landing page: shift metadata, the
  top-of-page checks, key updates with their Zendesk/Jira refs and status, and
  the free-text comments. "Hand over shift" closes the shift, opens the next
  one and carries the live items across, dropping anything done or marked
  "remove at end of shift" - the retyping this replaces.
- The RunPod table on that page is read from live host state instead of being
  copied in by hand, with the six-colour key preserved.

RunPod
- GraphQL client keyed on CX_RUNPOD_API_KEY. The old console login is kept as a
  fallback but cannot run unattended: the account has 2FA, so Clerk verifies the
  password and then asks for an emailed code and never issues a session. That is
  the real cause of the "No active session found" failure, and the client now
  says so instead of failing opaquely. TOTP is supported if the account moves to
  an authenticator app.
- Hosts and their listing history are persisted, so "most problematic hosts" can
  be ranked and each machine has a timeline of who listed or unlisted it, with
  the Zendesk comment and the error hint.
- The unlisting emails are parsed for the error block (they arrive
  quoted-printable) and classified into a likely cause and a next step.

Zendesk and Jira
- Unlisting raises a Zendesk ticket that follows the format of RunPod's own
  email, keyed on the machine so one machine keeps one thread, posted as an
  internal note.
- Jira is split in two: the Infrahub/OIE instance and the RunPod/RMA one, which
  may be a different Atlassian site. Blank RunPod values fall back to the
  defaults rather than failing.

Running in a container
- CX-Tools reads its keys from 1Password, which needs a desktop app. Config is a
  dataclass whose lookups live in per-field default factories, so passing
  CX_INFRAHUB_TOKEN/CX_INFRAINSIGHT_TOKEN in means those factories never run and
  CX-Tools itself stays unmodified.
- CX-Tools reaches OpenStack with `docker exec <region>-osc`, so the image now
  carries the Docker client (the static binary, not the docker.io package) and
  compose mounts the host socket with group_add for it. Verified from inside the
  container: live OpenStack and Infrahub calls both succeed.

Also fixes Settings, which read the environment at class-definition time and so
ignored anything set afterwards; a fresh Settings() silently returned stale
values. Caught by the Jira scoping tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

backend/app/runpod/client.py

@@ -0,0 +1,285 @@
+"""RunPod access.
+
+Two transports, deliberately unequal:
+
+* **API key (default).** One header against the documented GraphQL endpoint.
+  Nothing to expire, nothing to refresh, works from a container.
+* **Console login (fallback).** The old flow: sign in to Clerk with an email and
+  password, then reuse the session JWT. Kept because it is the only path that
+  works if the API key is revoked, but it cannot run unattended.
+
+The "No active session found" failure in the old script was not a UI change:
+the account now has two-factor authentication on. Clerk verifies the password
+and then answers ``status: needs_second_factor`` with an emailed code, so no
+session is ever created. That cannot be automated without reading the mailbox.
+If the account is switched from email codes to an authenticator app, set
+CX_RUNPOD_TOTP_SECRET and this fallback can complete on its own; otherwise
+CX_RUNPOD_2FA_CODE accepts a code for a single manual run.
+
+Set CX_RUNPOD_API_KEY and this module never touches the console at all.
+"""
+from __future__ import annotations
+
+import os
+import threading
+import time
+from typing import Any, Optional
+
+import httpx
+
+GRAPHQL_URL = os.environ.get("CX_RUNPOD_GRAPHQL_URL", "https://api.runpod.io/graphql")
+CLERK_BASE = os.environ.get("CX_RUNPOD_CLERK_BASE", "https://clerk.runpod.io")
+CONSOLE_URL = "https://console.runpod.io"
+CLERK_QS = "__clerk_api_version=2025-11-10&_clerk_js_version=5.125.12"
+
+
+class RunPodError(RuntimeError):
+    pass
+
+
+# --- queries ----------------------------------------------------------------
+
+MACHINES_QUERY = """
+query getMachinesForHostDashboard {
+  myself {
+    machineQuota
+    machines {
+      id name listed registered verified
+      gpuTypeId gpuReserved gpuTotal
+      dataCenterId machineType
+      hostPricePerGpu margin
+      uptimePercentListedOneWeek uptimePercentListedFourWeek
+      maintenanceStart maintenanceEnd
+      gpuType { displayName manufacturer }
+      machineSystem { os cudaVersion kernelVersion }
+    }
+  }
+}
+"""
+
+SUMMARY_QUERY = """
+query getMyMachines {
+  myself {
+    machinesSummary {
+      id displayName listed machineType
+      gpuTypeId gpuRented gpuTotal
+      podProfitPerHr diskProfitPerHr
+      onDemandPods spotPods
+    }
+  }
+}
+"""
+
+LIST_MUTATION = """
+mutation listMachineBulk($input: MachineListBulkInput) { machineListBulk(input: $input) }
+"""
+
+UNLIST_MUTATION = """
+mutation unlistMachineBulk($input: MachineUnlistBulkInput) { machineUnlistBulk(input: $input) }
+"""
+
+MAINTENANCE_MUTATION = """
+mutation machineScheduleMaintenance($input: MachineScheduleMaintenanceInput) {
+  machineScheduleMaintenance(input: $input)
+}
+"""
+
+
+class RunPodClient:
+    def __init__(self, api_key: str = "", email: str = "", password: str = "",
+                 team_id: str = "", totp_secret: str = "", otp_code: str = "",
+                 timeout: int = 30):
+        self.api_key = (api_key or "").strip()
+        self.email = (email or "").strip()
+        self.password = password or ""
+        self.team_id = (team_id or "").strip()
+        self.totp_secret = (totp_secret or "").strip()
+        self.otp_code = (otp_code or "").strip()
+        self.timeout = timeout
+        self._jwt: Optional[str] = None
+        self._jwt_at = 0.0
+        self._lock = threading.Lock()
+
+    # --- auth ---------------------------------------------------------------
+
+    @property
+    def mode(self) -> str:
+        if self.api_key:
+            return "api_key"
+        if self.email and self.password:
+            return "console_login"
+        return "unconfigured"
+
+    def _auth_header(self) -> dict[str, str]:
+        if self.api_key:
+            return {"Authorization": f"Bearer {self.api_key}"}
+        jwt = self._console_jwt()
+        header = {"Authorization": f"Bearer {jwt}"}
+        if self.team_id:
+            header["x-team-id"] = self.team_id
+        return header
+
+    def _console_jwt(self) -> str:
+        """Sign in to the console and return a session JWT.
+
+        Only used when no API key is set. Cached for 50 minutes; Clerk tokens
+        last an hour.
+        """
+        with self._lock:
+            if self._jwt and time.time() - self._jwt_at < 3000:
+                return self._jwt
+            if not (self.email and self.password):
+                raise RunPodError(
+                    "RunPod is not configured. Set CX_RUNPOD_API_KEY, or "
+                    "CX_RUNPOD_EMAIL and CX_RUNPOD_PASSWORD for the console fallback."
+                )
+
+            with httpx.Client(timeout=self.timeout, follow_redirects=True, headers={
+                "User-Agent": "Mozilla/5.0", "Origin": CONSOLE_URL, "Referer": f"{CONSOLE_URL}/",
+            }) as client:
+                client.get(CONSOLE_URL)
+                signin = client.post(
+                    f"{CLERK_BASE}/v1/client/sign_ins?{CLERK_QS}",
+                    data={"identifier": self.email, "password": self.password, "strategy": "password"},
+                )
+                if signin.status_code >= 400:
+                    raise RunPodError(f"Console sign-in rejected ({signin.status_code}): {signin.text[:200]}")
+
+                body = signin.json().get("response") or {}
+                session_id = body.get("created_session_id")
+
+                if body.get("status") == "needs_second_factor":
+                    session_id = self._second_factor(client, body)
+
+                # The old script read the session out of this response and gave
+                # up when it was absent. Ask for the client record instead.
+                jwt, found_id = self._session_from_client(client, session_id)
+                if not jwt and found_id:
+                    jwt = self._mint_token(client, found_id)
+                if not jwt:
+                    raise RunPodError(
+                        "Console sign-in completed but no session token came back. RunPod's console auth has "
+                        "changed again - use CX_RUNPOD_API_KEY instead."
+                    )
+                self._jwt, self._jwt_at = jwt, time.time()
+                return jwt
+
+    def _second_factor(self, client: httpx.Client, body: dict[str, Any]) -> Optional[str]:
+        """Satisfy Clerk's second factor, if we have been given the means to.
+
+        The account currently uses emailed codes, which no unattended process
+        can read - that is the real reason the old login stopped working.
+        """
+        sign_in_id = body.get("id")
+        offered = [f.get("strategy") for f in (body.get("supported_second_factors") or [])]
+
+        code = ""
+        if "totp" in offered and self.totp_secret:
+            code, strategy = self._totp_code(), "totp"
+        elif self.otp_code:
+            code, strategy = self.otp_code, ("email_code" if "email_code" in offered else offered[0] if offered else "")
+            if "email_code" in offered:
+                client.post(f"{CLERK_BASE}/v1/client/sign_ins/{sign_in_id}/prepare_second_factor?{CLERK_QS}",
+                            data={"strategy": "email_code"})
+        if not code:
+            raise RunPodError(
+                "RunPod console login needs a second factor "
+                f"({', '.join(offered) or 'unknown strategy'}) and no code is available. This is why the old "
+                "script failed - the password is accepted, but the account has 2FA on. Use CX_RUNPOD_API_KEY for "
+                "unattended runs, or set CX_RUNPOD_TOTP_SECRET if you move the account to an authenticator app."
+            )
+
+        attempt = client.post(
+            f"{CLERK_BASE}/v1/client/sign_ins/{sign_in_id}/attempt_second_factor?{CLERK_QS}",
+            data={"strategy": strategy, "code": code},
+        )
+        if attempt.status_code >= 400:
+            raise RunPodError(f"Second factor rejected ({attempt.status_code}): {attempt.text[:200]}")
+        return ((attempt.json().get("response") or {}).get("created_session_id"))
+
+    def _totp_code(self) -> str:
+        """RFC 6238 code from a base32 secret - no third-party dependency."""
+        import base64
+        import hashlib
+        import hmac
+        import struct
+
+        secret = self.totp_secret.replace(" ", "").upper()
+        secret += "=" * (-len(secret) % 8)
+        key = base64.b32decode(secret, casefold=True)
+        counter = struct.pack(">Q", int(time.time()) // 30)
+        digest = hmac.new(key, counter, hashlib.sha1).digest()
+        offset = digest[-1] & 0x0F
+        value = struct.unpack(">I", digest[offset:offset + 4])[0] & 0x7FFFFFFF
+        return f"{value % 1_000_000:06d}"
+
+    def _session_from_client(self, client: httpx.Client, prefer_id: Optional[str]) -> tuple[str, str]:
+        resp = client.get(f"{CLERK_BASE}/v1/client?{CLERK_QS}")
+        if resp.status_code >= 400:
+            return "", ""
+        sessions = ((resp.json().get("response") or {}).get("sessions") or [])
+        if not sessions:
+            return "", ""
+        chosen = next((s for s in sessions if s.get("id") == prefer_id), sessions[0])
+        token = ((chosen.get("last_active_token") or {}).get("jwt")) or ""
+        return token, str(chosen.get("id") or "")
+
+    def _mint_token(self, client: httpx.Client, session_id: str) -> str:
+        resp = client.post(f"{CLERK_BASE}/v1/client/sessions/{session_id}/tokens?{CLERK_QS}")
+        if resp.status_code >= 400:
+            return ""
+        return resp.json().get("jwt", "")
+
+    # --- transport ----------------------------------------------------------
+
+    def execute(self, query: str, variables: Optional[dict[str, Any]] = None) -> dict[str, Any]:
+        headers = {"Content-Type": "application/json", **self._auth_header()}
+        payload: dict[str, Any] = {"query": query}
+        if variables is not None:
+            payload["variables"] = variables
+        with httpx.Client(timeout=self.timeout) as client:
+            resp = client.post(GRAPHQL_URL, json=payload, headers=headers)
+        if resp.status_code == 401:
+            raise RunPodError("RunPod rejected the credentials (401). Check CX_RUNPOD_API_KEY.")
+        if resp.status_code >= 400:
+            raise RunPodError(f"RunPod returned {resp.status_code}: {resp.text[:300]}")
+        body = resp.json()
+        if body.get("errors"):
+            raise RunPodError("; ".join(e.get("message", "?") for e in body["errors"])[:400])
+        data = body.get("data")
+        if data is None:
+            raise RunPodError(f"RunPod returned no data: {str(body)[:200]}")
+        return data
+
+    # --- operations ---------------------------------------------------------
+
+    def machines(self) -> list[dict[str, Any]]:
+        data = self.execute(MACHINES_QUERY)
+        return list(((data.get("myself") or {}).get("machines")) or [])
+
+    def summary(self) -> list[dict[str, Any]]:
+        data = self.execute(SUMMARY_QUERY)
+        return list(((data.get("myself") or {}).get("machinesSummary")) or [])
+
+    def list_machines(self, machine_ids: list[str]) -> dict[str, Any]:
+        return self.execute(LIST_MUTATION, {"input": {"machineIds": machine_ids}})
+
+    def unlist_machines(self, machine_ids: list[str]) -> dict[str, Any]:
+        return self.execute(UNLIST_MUTATION, {"input": {"machineIds": machine_ids}})
+
+    def schedule_maintenance(self, machine_ids: list[str], start_utc: str, minutes: int,
+                             reason: str = "EMERGENCY", destructive: bool = False) -> dict[str, Any]:
+        if reason not in ("UPGRADE", "ROUTINE", "EMERGENCY", "REMOVE"):
+            raise RunPodError(f"Unknown maintenance reason '{reason}'.")
+        return self.execute(MAINTENANCE_MUTATION, {"input": {
+            "machineIds": machine_ids, "maintenanceStartUtc": start_utc,
+            "maintenanceMinutes": minutes, "maintenanceReason": reason, "destructive": destructive,
+        }})
+
+    def check(self) -> dict[str, Any]:
+        """Cheap connectivity probe for the health endpoint."""
+        try:
+            machines = self.machines()
+            return {"ok": True, "mode": self.mode, "machines": len(machines)}
+        except RunPodError as exc:
+            return {"ok": False, "mode": self.mode, "error": str(exc)}

backend/app/runpod/delivery.py

@@ -0,0 +1,201 @@
+"""Outbound actions for RunPod hosts: Zendesk notification and Jira/RMA escalation.
+
+An unlisted machine earns nothing and may be stranding rented workloads, so the
+team wants a ticket the moment it happens. RunPod's own email already says what
+broke; this reproduces that content in Zendesk so it lands in the tracking
+platform even when the unlisting was done through the API and no email was sent.
+
+The same three gates as customer comms apply - configured, feature-flagged, and
+sending enabled globally - so a demo instance cannot raise tickets.
+"""
+from __future__ import annotations
+
+import datetime as dt
+from typing import Any, Optional
+
+import httpx
+from sqlalchemy.orm import Session
+
+from ..config import get_settings
+from ..delivery import DeliveryError, _check_cap
+from ..models import RunpodEventType, RunpodHost, User
+from .email_parse import classify
+from .service import record_event
+
+settings = get_settings()
+
+
+def unlisting_ticket(host: RunpodHost, error_text: str = "") -> dict[str, str]:
+    """Compose the ticket body, following the shape of RunPod's own email.
+
+    Keeping the same structure means whoever picks the ticket up reads the
+    familiar thing: what broke, how many GPUs it is costing, and where to look.
+    """
+    error = (error_text or host.last_error or "").strip()
+    hint = classify(error)
+    rented = f"{host.gpu_reserved} GPU(s) currently rented on this machine." if host.gpu_reserved \
+        else "No GPUs currently rented - the machine has drained."
+
+    body = "\n".join([
+        f"Machine {host.name} ({host.machine_id}) is unlisted and is not accepting new users.",
+        "",
+        "Error detected:",
+        error or "(no error text captured - check the RunPod dashboard or the notification email)",
+        "",
+        f"Impact: {rented}",
+        f"Likely cause: {hint['signature']}",
+        f"Suggested next step: {hint['suggested_action']}",
+        "",
+        "Common causes and where to look:",
+        "- GPU error or failure: check nvidia-smi for GPU health and dmesg for Xid errors.",
+        "- Unresponsive Docker daemon: check whether the docker service is running or hung.",
+        "- Pod sync errors: if df -h hangs, suspect a disk error or hung moosefs mount.",
+        "- Docker overlay storage: confirm the docker filesystem is XFS and /var/lib/docker is mounted.",
+        "- Portallocator port check: verify the publicIp ports in /etc/runpod/config.json are reachable.",
+        "",
+        "If the fix needs a reboot or hardware work, schedule maintenance from the Machines "
+        "Dashboard rather than pulling the machine abruptly - that drains workloads gracefully.",
+        "",
+        f"Unlisted {host.unlist_count} time(s) to date.",
+        "Raised automatically by CX Triage.",
+    ])
+    return {
+        "subject": f"{host.name} Unlisted - {hint['signature']}",
+        "body": body,
+        "signature": hint["signature"],
+    }
+
+
+async def raise_unlisting_ticket(db: Session, host: RunpodHost, actor: User,
+                                 *, error_text: str = "", requester: str = "",
+                                 subject: str = "", body: str = "") -> dict[str, Any]:
+    """Create (or comment on) the Zendesk ticket for an unlisted machine."""
+    if not settings.feature_send_enabled:
+        raise DeliveryError("Sending is disabled on this instance (CX_FEATURE_SEND_ENABLED is off).")
+    if not settings.zendesk_ready:
+        raise DeliveryError("Zendesk is not configured. Set CX_FEATURE_ZENDESK plus "
+                            "CX_ZENDESK_SUBDOMAIN, CX_ZENDESK_EMAIL and CX_ZENDESK_TOKEN.")
+    _check_cap(db)
+
+    composed = unlisting_ticket(host, error_text)
+    subject = subject or composed["subject"]
+    body = body or composed["body"]
+
+    base = f"https://{settings.zendesk_subdomain}.zendesk.com/api/v2"
+    auth = (f"{settings.zendesk_email}/token", settings.zendesk_token)
+    # Keyed on the machine, not the incident: one machine, one running thread.
+    external_id = f"cx-triage-runpod-{host.machine_id}"
+
+    async with httpx.AsyncClient(timeout=30) as client:
+        found = await client.get(f"{base}/search.json",
+                                 params={"query": f'type:ticket external_id:"{external_id}"'}, auth=auth)
+        existing = None
+        if found.status_code == 200:
+            results = found.json().get("results") or []
+            # Only reuse a ticket that is still open; a solved one starts a new thread.
+            existing = next((t for t in results if t.get("status") not in ("solved", "closed")), None)
+
+        # Internal note by default: this is an operations ticket, not a customer reply.
+        comment = {"body": body, "public": False}
+        if existing:
+            resp = await client.put(f"{base}/tickets/{existing['id']}.json",
+                                    json={"ticket": {"comment": comment}}, auth=auth)
+            action = "updated"
+        else:
+            ticket: dict[str, Any] = {
+                "subject": subject,
+                "comment": comment,
+                "priority": "high" if host.gpu_reserved else "normal",
+                "type": "incident",
+                "tags": ["cx-triage", "runpod", "unlisted", f"dc-{(host.data_center or 'unknown').lower()}"],
+                "external_id": external_id,
+            }
+            if requester:
+                ticket["requester"] = {"name": requester.split("@")[0], "email": requester}
+            resp = await client.post(f"{base}/tickets.json", json={"ticket": ticket}, auth=auth)
+            action = "created"
+
+    if resp.status_code not in (200, 201):
+        record_event(db, host, RunpodEventType.NOTE, actor=actor.email,
+                     detail=f"Zendesk ticket failed: HTTP {resp.status_code}")
+        db.commit()
+        raise DeliveryError(f"Zendesk returned {resp.status_code}: {resp.text[:300]}")
+
+    ticket_id = str((resp.json().get("ticket") or {}).get("id")
+                    or (existing or {}).get("id") or "")
+    url = f"https://{settings.zendesk_subdomain}.zendesk.com/agent/tickets/{ticket_id}"
+
+    host.zendesk_ticket = ticket_id or host.zendesk_ticket
+    record_event(db, host, RunpodEventType.ZENDESK_TICKET, actor=actor.email,
+                 detail=f"Zendesk ticket {ticket_id} {action} - {composed['signature']}",
+                 error_hint=error_text or host.last_error, zendesk_ticket=ticket_id)
+    db.commit()
+    return {"ok": True, "ticket_id": ticket_id, "url": url, "action": action,
+            "subject": subject, "signature": composed["signature"]}
+
+
+async def raise_rma_issue(db: Session, host: RunpodHost, actor: User, *,
+                          summary: str = "", description: str = "") -> dict[str, Any]:
+    """Open an issue on the RunPod/RMA Jira - a different instance to the OIE one."""
+    creds = settings.jira_for("runpod")
+    if not settings.feature_send_enabled:
+        raise DeliveryError("Sending is disabled on this instance (CX_FEATURE_SEND_ENABLED is off).")
+    if not (settings.feature_jira and creds["base"] and creds["email"] and creds["token"]):
+        raise DeliveryError(
+            "The RunPod Jira is not configured. Set CX_RUNPOD_JIRA_BASE, CX_RUNPOD_JIRA_EMAIL and "
+            "CX_RUNPOD_JIRA_TOKEN - or leave them blank to reuse the default instance."
+        )
+    _check_cap(db)
+
+    hint = classify(host.last_error or "")
+    summary = summary or f"RunPod host {host.name} ({host.machine_id}) - {hint['signature']}"
+    description = description or "\n".join([
+        f"Machine: {host.name} ({host.machine_id})",
+        f"Data centre: {host.data_center or 'unknown'}",
+        f"GPUs: {host.gpu_reserved}/{host.gpu_total} rented",
+        f"Unlisted {host.unlist_count} time(s) to date.",
+        "",
+        "Last error:",
+        host.last_error or "(none captured)",
+        "",
+        f"Likely cause: {hint['signature']}",
+        f"Suggested next step: {hint['suggested_action']}",
+        "",
+        f"Zendesk: {host.zendesk_ticket or '(none)'}",
+        "Raised from CX Triage.",
+    ])
+
+    base = creds["base"].rstrip("/")
+    auth = (creds["email"], creds["token"])
+    label = f"cx-triage-runpod-{host.machine_id}"
+
+    async with httpx.AsyncClient(timeout=30) as client:
+        found = await client.get(f"{base}/rest/api/3/search",
+                                 params={"jql": f'labels = "{label}"', "maxResults": 1}, auth=auth)
+        if found.status_code == 200 and (found.json().get("issues") or []):
+            issue = found.json()["issues"][0]
+            key = issue["key"]
+            host.jira_key = key
+            record_event(db, host, RunpodEventType.JIRA_LINKED, actor=actor.email,
+                         detail=f"Issue {key} already exists", jira_key=key)
+            db.commit()
+            return {"ok": True, "key": key, "url": f"{base}/browse/{key}", "action": "existing"}
+
+        resp = await client.post(f"{base}/rest/api/3/issue", json={"fields": {
+            "project": {"key": creds["project"]},
+            "summary": summary[:250],
+            "issuetype": {"name": creds["issue_type"]},
+            "labels": ["cx-triage", "runpod", label],
+            "description": {"type": "doc", "version": 1, "content": [
+                {"type": "paragraph", "content": [{"type": "text", "text": description[:30000]}]}]},
+        }}, auth=auth)
+
+    if resp.status_code not in (200, 201):
+        raise DeliveryError(f"Jira returned {resp.status_code}: {resp.text[:300]}")
+
+    key = resp.json().get("key", "")
+    host.jira_key = key
+    record_event(db, host, RunpodEventType.JIRA_LINKED, actor=actor.email,
+                 detail=f"Issue {key} created on the RunPod Jira", jira_key=key)
+    db.commit()
+    return {"ok": True, "key": key, "url": f"{base}/browse/{key}", "action": "created"}

backend/app/runpod/email_parse.py

@@ -0,0 +1,114 @@
+"""Parse RunPod's automated unlisting emails.
+
+Subject:  "<host> Unlisted - CRITICAL ERROR"
+Body:     the machine id in brackets, an "Error detected:" block, and an
+          "Impact: N GPU(s) currently rented" line.
+
+The error block is the useful part - it is the only place the actual reason
+appears, and it is what CX pastes into the Zendesk ticket and the handover.
+Written to cope with the HTML mail as well as a plain-text paste.
+"""
+from __future__ import annotations
+
+import html
+import quopri
+import re
+from typing import Any
+
+SUBJECT_RE = re.compile(r"^(?P<host>[\w.-]+)\s+Unlisted\b", re.I)
+MACHINE_RE = re.compile(r"machine\s+(?P<host>[\w.-]+)\s*\((?P<machine_id>[a-z0-9]{8,})\)", re.I)
+IMPACT_RE = re.compile(r"Impact:\s*(?P<gpus>\d+)\s*GPU", re.I)
+
+ERROR_START = re.compile(r"Error detected:\s*", re.I)
+ERROR_END = re.compile(r"(Impact:|Common causes|Need to take the machine offline)", re.I)
+
+# Recognisable failure signatures, mapped to what CX should do about them.
+SIGNATURES: list[tuple[str, str, str]] = [
+    (r"memory_remap|uncorrectable remapped memory",
+     "GPU memory remapping failure", "Hardware. Likely RMA - raise with the vendor."),
+    (r"xid", "XID error", "Check dmesg and nvidia-smi; usually a burn-in test before relisting."),
+    (r"gpu_cuda_ok.*expected 1, got 0|cuda initialization|fallen off the bus",
+     "GPU not visible to CUDA", "Check nvidia-smi and the PCIe link; reboot then burn-in."),
+    (r"nvidia-smi.*too many failures", "nvidia-smi failing repeatedly",
+     "Host-level GPU fault - stress test, then RMA if it repeats."),
+    (r"docker (service )?(unresponsive|hung)|container stuck",
+     "Docker daemon unresponsive", "Restart docker; check /var/lib/docker is XFS and mounted."),
+    (r"pod sync (failed|errors)", "Pod sync failing",
+     "Check df -h for a hung moosefs mount and network to the moosefs cluster."),
+    (r"portallocator|public port check fail", "Public port check failing",
+     "Verify the publicIp ports in /etc/runpod/config.json are reachable."),
+    (r"disk|filesystem|xfs", "Disk or filesystem error", "Check dmesg for disk failures."),
+]
+
+
+# Soft line breaks ("=\n") and "=3D" are the giveaways for a quoted-printable
+# body, which is how these mails arrive. Left undecoded, the error block comes
+# out full of "=3D" and split mid-word.
+_QP_HINT = re.compile(r"=\r?\n|=[0-9A-F]{2}")
+
+
+def _maybe_decode_qp(raw: str) -> str:
+    if not _QP_HINT.search(raw):
+        return raw
+    try:
+        return quopri.decodestring(raw.encode("utf-8", "replace")).decode("utf-8", "replace")
+    except Exception:
+        return raw
+
+
+def _text_from_html(raw: str) -> str:
+    text = re.sub(r"<(script|style)[^>]*>.*?</\1>", "", raw, flags=re.S | re.I)
+    text = re.sub(r"<br\s*/?>|</(p|div|tr|li|h[1-6]|table)>", "\n", text, flags=re.I)
+    text = re.sub(r"<[^>]+>", "", text)
+    return html.unescape(text)
+
+
+def classify(error_text: str) -> dict[str, str]:
+    """Name the failure and say what it usually means."""
+    blob = (error_text or "").lower()
+    for pattern, label, action in SIGNATURES:
+        if re.search(pattern, blob, re.I):
+            return {"signature": label, "suggested_action": action}
+    return {"signature": "Unrecognised error",
+            "suggested_action": "Read the error text and escalate if it is not obvious."}
+
+
+def parse(raw: str, subject: str = "") -> dict[str, Any]:
+    """Pull the machine, the error block and the rented-GPU impact out of an email."""
+    raw = _maybe_decode_qp(raw)
+    text = _text_from_html(raw) if "<" in raw and ">" in raw else raw
+    lines = [ln.strip() for ln in text.splitlines()]
+    body = "\n".join(ln for ln in lines if ln)
+
+    host = ""
+    machine_id = ""
+    match = MACHINE_RE.search(body)
+    if match:
+        host, machine_id = match.group("host"), match.group("machine_id")
+    if not host and subject:
+        subject_match = SUBJECT_RE.match(subject.strip())
+        if subject_match:
+            host = subject_match.group("host")
+
+    error_text = ""
+    start = ERROR_START.search(body)
+    if start:
+        rest = body[start.end():]
+        end = ERROR_END.search(rest)
+        error_text = (rest[:end.start()] if end else rest).strip()
+        # The mail repeats its own body; one copy is enough.
+        error_text = "\n".join(dict.fromkeys(ln for ln in error_text.splitlines() if ln.strip() != "---"))
+
+    gpus = None
+    impact = IMPACT_RE.search(body)
+    if impact:
+        gpus = int(impact.group("gpus"))
+
+    return {
+        "host": host,
+        "machine_id": machine_id,
+        "error_text": error_text.strip(),
+        "gpus_rented": gpus,
+        "is_critical": "critical" in (subject or body).lower(),
+        **classify(error_text),
+    }

backend/app/runpod/service.py

@@ -0,0 +1,102 @@
+"""Turning RunPod machine state into tracked history.
+
+The API only ever reports what is true now. Everything CX cares about - how
+often a machine has dropped out, when it drained, who relisted it - only exists
+if each sync writes down what changed.
+"""
+from __future__ import annotations
+
+import datetime as dt
+from typing import Any, Optional
+
+from sqlalchemy import select
+from sqlalchemy.orm import Session
+
+from ..models import RunpodColour, RunpodEvent, RunpodEventType, RunpodHost
+
+
+def record_event(db: Session, host: RunpodHost, event_type: RunpodEventType, *,
+                 actor: str = "system", detail: str = "", error_hint: str = "",
+                 zendesk_ticket: str = "", jira_key: str = "",
+                 gpu_reserved: Optional[int] = None,
+                 payload: Optional[dict[str, Any]] = None) -> RunpodEvent:
+    event = RunpodEvent(
+        event_type=event_type, actor=actor, detail=detail, error_hint=error_hint,
+        zendesk_ticket=zendesk_ticket, jira_key=jira_key,
+        gpu_reserved=host.gpu_reserved if gpu_reserved is None else gpu_reserved,
+        payload=payload,
+    )
+    # Through the relationship so an already-loaded history stays correct.
+    host.events.append(event)
+    db.add(event)
+    return event
+
+
+def sync_machines(db: Session, client: Any, actor: str = "system") -> dict[str, Any]:
+    """Pull current machines and write down every transition since last time."""
+    machines = client.machines()
+    existing = {h.machine_id: h for h in db.scalars(select(RunpodHost)).all()}
+    now = dt.datetime.now(dt.timezone.utc)
+
+    created = unlisted = relisted = drained = 0
+
+    for machine in machines:
+        machine_id = str(machine.get("id") or "")
+        if not machine_id:
+            continue
+        listed = bool(machine.get("listed"))
+        reserved = int(machine.get("gpuReserved") or 0)
+
+        host = existing.get(machine_id)
+        if host is None:
+            host = RunpodHost(machine_id=machine_id, name=str(machine.get("name") or ""))
+            db.add(host)
+            db.flush()
+            created += 1
+            record_event(db, host, RunpodEventType.LISTED if listed else RunpodEventType.UNLISTED,
+                         actor="runpod", detail="First seen by CX Triage")
+            if not listed:
+                host.unlisted_at = now
+
+        was_listed = host.listed
+        was_reserved = host.gpu_reserved
+
+        if was_listed and not listed:
+            unlisted += 1
+            host.unlisted_at = now
+            host.unlist_count += 1
+            # Nothing in the API says why; the email carries the reason and is
+            # attached separately through /ingest-email.
+            record_event(db, host, RunpodEventType.UNLISTED, actor="runpod",
+                         detail="Unlisted (detected on sync)", gpu_reserved=reserved)
+        elif not was_listed and listed:
+            relisted += 1
+            host.last_listed_at = now
+            host.unlisted_at = None
+            record_event(db, host, RunpodEventType.LISTED, actor=actor,
+                         detail="Relisted", gpu_reserved=reserved)
+
+        # Unlisted and the last renter has gone: the machine is now safe to work on.
+        if not listed and was_reserved > 0 and reserved == 0:
+            drained += 1
+            record_event(db, host, RunpodEventType.DRAINED, actor="runpod",
+                         detail="Unlisted machine has drained - no rented GPUs left",
+                         gpu_reserved=0)
+
+        gpu_type = machine.get("gpuType") or {}
+        host.name = str(machine.get("name") or host.name)
+        host.listed = listed
+        host.gpu_reserved = reserved
+        host.gpu_total = int(machine.get("gpuTotal") or 0)
+        host.gpu_type = str(gpu_type.get("displayName") or machine.get("gpuTypeId") or "")
+        host.data_center = str(machine.get("dataCenterId") or "")
+        host.uptime_one_week = machine.get("uptimePercentListedOneWeek")
+        host.maintenance_start = str(machine.get("maintenanceStart") or "")
+        host.maintenance_end = str(machine.get("maintenanceEnd") or "")
+
+    db.commit()
+    return {
+        "machines": len(machines), "created": created, "unlisted": unlisted,
+        "relisted": relisted, "drained": drained,
+        "synced_at": now.isoformat(),
+    }