Add shift handover and RunPod, and make CX-Tools work in a container

Handover
- The Confluence shift doc becomes the landing page: shift metadata, the
  top-of-page checks, key updates with their Zendesk/Jira refs and status, and
  the free-text comments. "Hand over shift" closes the shift, opens the next
  one and carries the live items across, dropping anything done or marked
  "remove at end of shift" - the retyping this replaces.
- The RunPod table on that page is read from live host state instead of being
  copied in by hand, with the six-colour key preserved.

RunPod
- GraphQL client keyed on CX_RUNPOD_API_KEY. The old console login is kept as a
  fallback but cannot run unattended: the account has 2FA, so Clerk verifies the
  password and then asks for an emailed code and never issues a session. That is
  the real cause of the "No active session found" failure, and the client now
  says so instead of failing opaquely. TOTP is supported if the account moves to
  an authenticator app.
- Hosts and their listing history are persisted, so "most problematic hosts" can
  be ranked and each machine has a timeline of who listed or unlisted it, with
  the Zendesk comment and the error hint.
- The unlisting emails are parsed for the error block (they arrive
  quoted-printable) and classified into a likely cause and a next step.

Zendesk and Jira
- Unlisting raises a Zendesk ticket that follows the format of RunPod's own
  email, keyed on the machine so one machine keeps one thread, posted as an
  internal note.
- Jira is split in two: the Infrahub/OIE instance and the RunPod/RMA one, which
  may be a different Atlassian site. Blank RunPod values fall back to the
  defaults rather than failing.

Running in a container
- CX-Tools reads its keys from 1Password, which needs a desktop app. Config is a
  dataclass whose lookups live in per-field default factories, so passing
  CX_INFRAHUB_TOKEN/CX_INFRAINSIGHT_TOKEN in means those factories never run and
  CX-Tools itself stays unmodified.
- CX-Tools reaches OpenStack with `docker exec <region>-osc`, so the image now
  carries the Docker client (the static binary, not the docker.io package) and
  compose mounts the host socket with group_add for it. Verified from inside the
  container: live OpenStack and Infrahub calls both succeed.

Also fixes Settings, which read the environment at class-definition time and so
ignored anything set afterwards; a fresh Settings() silently returned stale
values. Caught by the Jira scoping tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

.env.example

@@ -65,3 +65,43 @@ CX_JIRA_EMAIL=
 CX_JIRA_TOKEN=
 CX_JIRA_PROJECT=INFRA
 CX_JIRA_ISSUE_TYPE=Task
+
+# --- local compose paths ---------------------------------------------------
+# CX-Tools on the host. The app imports it and shells out to the *-osc
+# containers through the mounted Docker socket, so both must be present.
+CX_TOOLS_HOST_PATH=../CX-Tools
+CX_RUNPOD_EXPORT_HOST_PATH=../RunPod
+
+# Infrahub/InfraInsight keys. Set these and CX-Tools skips 1Password entirely,
+# which is the only way it can work inside a container.
+CX_INFRAHUB_TOKEN=
+CX_INFRAINSIGHT_TOKEN=
+
+# Fill an empty database with representative handover and RunPod data.
+CX_SEED_DEMO=true
+
+# --- RunPod ----------------------------------------------------------------
+# The API key is the supported path. Email/password only reaches a console
+# login that now requires a second factor, so it cannot run unattended.
+CX_FEATURE_RUNPOD=true
+CX_FEATURE_RUNPOD_WRITE=false
+CX_RUNPOD_API_KEY=
+CX_RUNPOD_EMAIL=
+CX_RUNPOD_PASSWORD=
+CX_RUNPOD_TEAM_ID=
+# Only if the account moves from emailed codes to an authenticator app.
+CX_RUNPOD_TOTP_SECRET=
+
+# --- Jira, second instance (RunPod / RMA) ----------------------------------
+# Leave blank to reuse the CX_JIRA_* values above.
+CX_RUNPOD_JIRA_BASE=
+CX_RUNPOD_JIRA_EMAIL=
+CX_RUNPOD_JIRA_TOKEN=
+CX_RUNPOD_JIRA_PROJECT=RMA
+
+CX_FEATURE_HANDOVER=true
+
+# --- Zendesk tickets for RunPod unlistings ---------------------------------
+# Uses the same Zendesk credentials as customer comms. Tickets are keyed on the
+# machine (external_id cx-triage-runpod-<machine_id>) and posted as an internal
+# note, since these are operations tickets rather than customer replies.

.gitignore

@@ -13,3 +13,5 @@ build/
 *.sqlite3
 data/
 .DS_Store
+tsconfig.tsbuildinfo
+*.tsbuildinfo

backend/Dockerfile

@@ -10,13 +10,25 @@ RUN npm run build
 FROM python:3.12-slim
 ENV PYTHONUNBUFFERED=1 PYTHONDONTWRITEBYTECODE=1
 
-# curl is what the triage engine shells out to for the Infrahub API; the docker
+# curl is what the triage engine shells out to for the Infrahub API.
-# CLI is only needed when Prometheus/OpenStack are reachable through the
-# CX-Tools containers rather than directly (see docs/DEPLOYMENT.md).
 RUN apt-get update \
- && apt-get install -y --no-install-recommends curl ca-certificates docker.io \
+ && apt-get install -y --no-install-recommends curl ca-certificates \
  && rm -rf /var/lib/apt/lists/*
 
+# Just the Docker *client*, from the official static build. CX-Tools reaches
+# OpenStack with `docker exec <region>-osc openstack ...`, so with the host's
+# socket mounted those become sibling containers. The `docker.io` apt package
+# would drag in the daemon and containerd for a binary we never run.
+ARG DOCKER_CLI_VERSION=27.3.1
+RUN set -eux; \
+    arch="$(dpkg --print-architecture)"; \
+    case "$arch" in amd64) dl=x86_64 ;; arm64) dl=aarch64 ;; *) echo "unsupported $arch" >&2; exit 1 ;; esac; \
+    curl -fsSL "https://download.docker.com/linux/static/stable/${dl}/docker-${DOCKER_CLI_VERSION}.tgz" \
+      | tar -xz -C /tmp docker/docker; \
+    mv /tmp/docker/docker /usr/local/bin/docker; \
+    rm -rf /tmp/docker; \
+    docker --version
+
 WORKDIR /app
 COPY backend/requirements.txt .
 RUN pip install --no-cache-dir -r requirements.txt

backend/app/config.py

@@ -21,56 +21,114 @@ def _int(name: str, default: int) -> int:
 
 
 class Settings:
-    # --- app ---------------------------------------------------------------
+    """Read once per instance, at construction time.
-    app_name = os.environ.get("CX_APP_NAME", "CX Triage")
-    base_url = os.environ.get("CX_BASE_URL", "http://localhost:8080")
-    secret_key = os.environ.get("CX_SECRET_KEY", "dev-only-change-me")
-    session_hours = _int("CX_SESSION_HOURS", 12)
-    static_dir = os.environ.get("CX_STATIC_DIR", "/app/static")
 
-    # --- database ----------------------------------------------------------
+    These were class attributes, which meant the environment was read when the
-    # sqlite for local/compose, postgres in the cluster.
+    module was first imported and never again - so a fresh Settings() silently
-    database_url = os.environ.get("CX_DATABASE_URL", "sqlite:////data/cx-triage.db")
+    returned stale values. Reading in __init__ makes construction mean what it
+    looks like it means.
+    """
 
-    # --- data sources ------------------------------------------------------
+    def __init__(self) -> None:
-    prometheus_base = os.environ.get("CX_PROMETHEUS_BASE", "http://10.11.254.250:9090")
+        # --- app ---------------------------------------------------------------
-    prometheus_relay = os.environ.get("CX_PROMETHEUS_RELAY", "")
+        self.app_name = os.environ.get("CX_APP_NAME", "CX Triage")
-    cx_tools_path = os.environ.get("CX_TOOLS_PATH", "")
+        self.base_url = os.environ.get("CX_BASE_URL", "http://localhost:8080")
+        self.secret_key = os.environ.get("CX_SECRET_KEY", "dev-only-change-me")
+        self.session_hours = _int("CX_SESSION_HOURS", 12)
+        self.static_dir = os.environ.get("CX_STATIC_DIR", "/app/static")
 
-    # --- auth --------------------------------------------------------------
+        # --- database ----------------------------------------------------------
-    # Local accounts are for development and for a cluster without SSO yet.
+        # sqlite for local/compose, postgres in the cluster.
-    # When CX_OIDC_ENABLED is on, Authentik becomes the source of truth.
+        self.database_url = os.environ.get("CX_DATABASE_URL", "sqlite:////data/cx-triage.db")
-    auth_local_enabled = _bool("CX_AUTH_LOCAL_ENABLED", True)
-    bootstrap_admin_email = os.environ.get("CX_BOOTSTRAP_ADMIN_EMAIL", "admin@localhost")
-    bootstrap_admin_password = os.environ.get("CX_BOOTSTRAP_ADMIN_PASSWORD", "")
 
-    oidc_enabled = _bool("CX_OIDC_ENABLED", False)
+        # --- data sources ------------------------------------------------------
-    oidc_issuer = os.environ.get("CX_OIDC_ISSUER", "")            # e.g. https://sso/application/o/cx-triage/
+        self.prometheus_base = os.environ.get("CX_PROMETHEUS_BASE", "http://10.11.254.250:9090")
-    oidc_client_id = os.environ.get("CX_OIDC_CLIENT_ID", "")
+        self.prometheus_relay = os.environ.get("CX_PROMETHEUS_RELAY", "")
-    oidc_client_secret = os.environ.get("CX_OIDC_CLIENT_SECRET", "")
+        self.cx_tools_path = os.environ.get("CX_TOOLS_PATH", "")
-    oidc_scopes = os.environ.get("CX_OIDC_SCOPES", "openid email profile")
-    oidc_admin_group = os.environ.get("CX_OIDC_ADMIN_GROUP", "cx-triage-admins")
-    oidc_groups_claim = os.environ.get("CX_OIDC_GROUPS_CLAIM", "groups")
 
-    # --- feature flags -----------------------------------------------------
+        # --- auth --------------------------------------------------------------
-    # Sending must be switched on deliberately; a demo instance cannot email.
+        # Local accounts are for development and for a cluster without SSO yet.
-    feature_send_enabled = _bool("CX_FEATURE_SEND_ENABLED", False)
+        # When CX_OIDC_ENABLED is on, Authentik becomes the source of truth.
-    feature_zendesk = _bool("CX_FEATURE_ZENDESK", False)
+        self.auth_local_enabled = _bool("CX_AUTH_LOCAL_ENABLED", True)
-    feature_jira = _bool("CX_FEATURE_JIRA", False)
+        self.bootstrap_admin_email = os.environ.get("CX_BOOTSTRAP_ADMIN_EMAIL", "admin@localhost")
-    feature_linkage_scan = _bool("CX_FEATURE_LINKAGE_SCAN", True)
+        self.bootstrap_admin_password = os.environ.get("CX_BOOTSTRAP_ADMIN_PASSWORD", "")
-    send_daily_cap = _int("CX_SEND_DAILY_CAP", 25)
 
-    # --- integrations ------------------------------------------------------
+        self.oidc_enabled = _bool("CX_OIDC_ENABLED", False)
-    zendesk_subdomain = os.environ.get("CX_ZENDESK_SUBDOMAIN", "")
+        self.oidc_issuer = os.environ.get("CX_OIDC_ISSUER", "")            # e.g. https://sso/application/o/cx-triage/
-    zendesk_email = os.environ.get("CX_ZENDESK_EMAIL", "")
+        self.oidc_client_id = os.environ.get("CX_OIDC_CLIENT_ID", "")
-    zendesk_token = os.environ.get("CX_ZENDESK_TOKEN", "")
+        self.oidc_client_secret = os.environ.get("CX_OIDC_CLIENT_SECRET", "")
-    zendesk_default_public = _bool("CX_ZENDESK_PUBLIC_REPLY", True)
+        self.oidc_scopes = os.environ.get("CX_OIDC_SCOPES", "openid email profile")
+        self.oidc_admin_group = os.environ.get("CX_OIDC_ADMIN_GROUP", "cx-triage-admins")
+        self.oidc_groups_claim = os.environ.get("CX_OIDC_GROUPS_CLAIM", "groups")
 
-    jira_base = os.environ.get("CX_JIRA_BASE", "")
+        # --- feature flags -----------------------------------------------------
-    jira_email = os.environ.get("CX_JIRA_EMAIL", "")
+        # Sending must be switched on deliberately; a demo instance cannot email.
-    jira_token = os.environ.get("CX_JIRA_TOKEN", "")
+        self.feature_send_enabled = _bool("CX_FEATURE_SEND_ENABLED", False)
-    jira_project = os.environ.get("CX_JIRA_PROJECT", "INFRA")
+        self.feature_zendesk = _bool("CX_FEATURE_ZENDESK", False)
-    jira_issue_type = os.environ.get("CX_JIRA_ISSUE_TYPE", "Task")
+        self.feature_jira = _bool("CX_FEATURE_JIRA", False)
+        self.feature_linkage_scan = _bool("CX_FEATURE_LINKAGE_SCAN", True)
+        self.send_daily_cap = _int("CX_SEND_DAILY_CAP", 25)
+
+        # --- integrations ------------------------------------------------------
+        self.zendesk_subdomain = os.environ.get("CX_ZENDESK_SUBDOMAIN", "")
+        self.zendesk_email = os.environ.get("CX_ZENDESK_EMAIL", "")
+        self.zendesk_token = os.environ.get("CX_ZENDESK_TOKEN", "")
+        self.zendesk_default_public = _bool("CX_ZENDESK_PUBLIC_REPLY", True)
+
+        # Jira, instance 1: the Infrahub/OpenStack side (OIE).
+        self.jira_base = os.environ.get("CX_JIRA_BASE", "")
+        self.jira_email = os.environ.get("CX_JIRA_EMAIL", "")
+        self.jira_token = os.environ.get("CX_JIRA_TOKEN", "")
+        self.jira_project = os.environ.get("CX_JIRA_PROJECT", "OIE")
+        self.jira_issue_type = os.environ.get("CX_JIRA_ISSUE_TYPE", "Task")
+
+        # Jira, instance 2: RunPod hosts and RMAs. These may be a different
+        # Atlassian site entirely, so they get their own credentials; anything left
+        # blank falls back to the values above rather than failing.
+        self.runpod_jira_base = os.environ.get("CX_RUNPOD_JIRA_BASE", "")
+        self.runpod_jira_email = os.environ.get("CX_RUNPOD_JIRA_EMAIL", "")
+        self.runpod_jira_token = os.environ.get("CX_RUNPOD_JIRA_TOKEN", "")
+        self.runpod_jira_project = os.environ.get("CX_RUNPOD_JIRA_PROJECT", "RMA")
+        self.runpod_jira_issue_type = os.environ.get("CX_RUNPOD_JIRA_ISSUE_TYPE", "Task")
+
+        # --- RunPod ------------------------------------------------------------
+        # The API key is the supported path. Email/password only reaches a console
+        # login that now requires a second factor, so it cannot run unattended -
+        # see app/runpod/client.py.
+        self.runpod_api_key = os.environ.get("CX_RUNPOD_API_KEY", "")
+        self.runpod_email = os.environ.get("CX_RUNPOD_EMAIL", "")
+        self.runpod_password = os.environ.get("CX_RUNPOD_PASSWORD", "")
+        self.runpod_team_id = os.environ.get("CX_RUNPOD_TEAM_ID", "")
+        self.runpod_totp_secret = os.environ.get("CX_RUNPOD_TOTP_SECRET", "")
+        self.feature_runpod = _bool("CX_FEATURE_RUNPOD", True)
+        self.feature_handover = _bool("CX_FEATURE_HANDOVER", True)
+        # Unlisting a machine is destructive to earnings; gated separately.
+        self.feature_runpod_write = _bool("CX_FEATURE_RUNPOD_WRITE", False)
+
+    def jira_for(self, scope: str) -> dict:
+        """Credentials for a Jira scope: 'runpod' or anything else (default)."""
+        if scope == "runpod":
+            return {
+                "base": self.runpod_jira_base or self.jira_base,
+                "email": self.runpod_jira_email or self.jira_email,
+                "token": self.runpod_jira_token or self.jira_token,
+                "project": self.runpod_jira_project,
+                "issue_type": self.runpod_jira_issue_type,
+            }
+        return {
+            "base": self.jira_base, "email": self.jira_email, "token": self.jira_token,
+            "project": self.jira_project, "issue_type": self.jira_issue_type,
+        }
+
+    @property
+    def runpod_ready(self) -> bool:
+        return bool(self.feature_runpod and (self.runpod_api_key or
+                                             (self.runpod_email and self.runpod_password)))
+
+    @property
+    def runpod_jira_ready(self) -> bool:
+        creds = self.jira_for("runpod")
+        return bool(self.feature_jira and creds["base"] and creds["email"] and creds["token"])
 
     @property
     def zendesk_ready(self) -> bool:
@@ -93,6 +151,11 @@ class Settings:
             "send_enabled": self.feature_send_enabled,
             "linkage_scan": self.feature_linkage_scan,
             "jira_project": self.jira_project if self.jira_ready else "",
+            "runpod_ready": self.runpod_ready,
+            "runpod_write": self.feature_runpod_write,
+            "runpod_jira_ready": self.runpod_jira_ready,
+            "runpod_jira_project": self.runpod_jira_project if self.runpod_jira_ready else "",
+            "handover": self.feature_handover,
         }
 
 

backend/app/delivery.py

@@ -29,7 +29,7 @@ class DeliveryError(RuntimeError):
     pass
 
 
-def _guard(kind: str) -> None:
+def _guard(kind: str, scope: str = "default") -> None:
     if not settings.feature_send_enabled:
         raise DeliveryError(
             "Sending is disabled on this instance (CX_FEATURE_SEND_ENABLED is off). "
@@ -38,9 +38,15 @@ def _guard(kind: str) -> None:
     if kind == "zendesk" and not settings.zendesk_ready:
         raise DeliveryError("Zendesk is not configured. Set CX_FEATURE_ZENDESK plus "
                             "CX_ZENDESK_SUBDOMAIN, CX_ZENDESK_EMAIL and CX_ZENDESK_TOKEN.")
-    if kind == "jira" and not settings.jira_ready:
+    if kind == "jira":
-        raise DeliveryError("Jira is not configured. Set CX_FEATURE_JIRA plus "
+        creds = settings.jira_for(scope)
-                            "CX_JIRA_BASE, CX_JIRA_EMAIL, CX_JIRA_TOKEN and CX_JIRA_PROJECT.")
+        if not (settings.feature_jira and creds["base"] and creds["email"] and creds["token"]):
+            prefix = "CX_RUNPOD_JIRA_" if scope == "runpod" else "CX_JIRA_"
+            raise DeliveryError(
+                f"Jira ({scope}) is not configured. Set CX_FEATURE_JIRA plus {prefix}BASE, "
+                f"{prefix}EMAIL and {prefix}TOKEN - or leave the {prefix}* values blank to reuse "
+                "the default instance."
+            )
 
 
 def sends_today(db: Session) -> int:
@@ -125,12 +131,20 @@ async def send_zendesk(db: Session, case: Case, actor: User, *, to: str, subject
 
 async def create_jira(db: Session, case: Case, actor: User, *, summary: str, description: str,
                       project: str = "", issue_type: str = "",
-                      labels: Optional[list[str]] = None) -> dict[str, Any]:
+                      labels: Optional[list[str]] = None,
-    _guard("jira")
+                      scope: str = "default") -> dict[str, Any]:
+    """Raise a Jira issue on the instance that owns this kind of work.
+
+    `scope="runpod"` targets the RunPod/RMA project, which may live on an
+    entirely different Atlassian site - hence separate credentials rather than
+    just a different project key.
+    """
+    _guard("jira", scope)
     _check_cap(db)
 
-    base = settings.jira_base.rstrip("/")
+    creds = settings.jira_for(scope)
-    auth = (settings.jira_email, settings.jira_token)
+    base = creds["base"].rstrip("/")
+    auth = (creds["email"], creds["token"])
     label = f"cx-triage-{case.fingerprint}"
     all_labels = sorted(set((labels or []) + ["cx-triage", label]))
 
@@ -147,9 +161,9 @@ async def create_jira(db: Session, case: Case, actor: User, *, summary: str, des
             return {"ok": True, "key": key, "url": url, "action": "existing"}
 
         payload = {"fields": {
-            "project": {"key": project or settings.jira_project},
+            "project": {"key": project or creds["project"]},
             "summary": summary[:250],
-            "issuetype": {"name": issue_type or settings.jira_issue_type},
+            "issuetype": {"name": issue_type or creds["issue_type"]},
             "labels": all_labels,
             "description": {
                 "type": "doc", "version": 1,

backend/app/main.py

@@ -13,11 +13,15 @@ from .auth import ensure_bootstrap_admin
 from .config import get_settings
 from .db import SessionLocal, init_db
 from .routers import (actions_router, alerts_router, auth_router, cases_router,
-                      linkage_router, settings_router)
+                      handover_router, linkage_router, runpod_router, settings_router)
 
 settings = get_settings()
 
 
+def _bool_env(name: str) -> bool:
+    return str(os.environ.get(name, "")).strip().lower() in {"1", "true", "yes", "on"}
+
+
 @asynccontextmanager
 async def lifespan(app: FastAPI):
     init_db()
@@ -28,6 +32,17 @@ async def lifespan(app: FastAPI):
             print(f"[auth] {message}")
     finally:
         db.close()
+    if _bool_env("CX_SEED_DEMO"):
+        db = SessionLocal()
+        try:
+            from .seed.loader import run as seed_run
+            for line in seed_run(db):
+                print(f"[seed] {line}")
+        except Exception as exc:
+            print(f"[seed] skipped: {type(exc).__name__}: {exc}")
+        finally:
+            db.close()
+
     print(f"[startup] {settings.app_name} {VERSION}")
     print(f"[startup] prometheus: {settings.prometheus_base}")
     print(f"[startup] sso: {'on' if settings.oidc_enabled else 'off'} | "
@@ -41,7 +56,7 @@ async def lifespan(app: FastAPI):
 app = FastAPI(title=settings.app_name, version=VERSION, lifespan=lifespan)
 
 for module in (auth_router, alerts_router, cases_router, settings_router,
-               linkage_router, actions_router):
+               linkage_router, actions_router, handover_router, runpod_router):
     app.include_router(module.router)
 
 

backend/app/models.py

@@ -12,7 +12,7 @@ import datetime as dt
 import enum
 from typing import Any
 
-from sqlalchemy import (JSON, Boolean, DateTime, Enum, ForeignKey, Index, Integer,
+from sqlalchemy import (JSON, Boolean, Date, DateTime, Enum, ForeignKey, Index, Integer,
                         String, Text, UniqueConstraint)
 from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, relationship
 
@@ -25,6 +25,18 @@ def _now() -> dt.datetime:
     return dt.datetime.now(dt.timezone.utc)
 
 
+def _as_utc(value: dt.datetime | None) -> dt.datetime | None:
+    """Re-attach UTC to a value read back from the database.
+
+    SQLite has no timezone type, so a datetime stored as aware comes back naive
+    and cannot be compared with `now()`. Postgres round-trips correctly; this
+    keeps both behaving the same.
+    """
+    if value is None:
+        return None
+    return value if value.tzinfo else value.replace(tzinfo=dt.timezone.utc)
+
+
 class CaseStatus(str, enum.Enum):
     """Where a piece of work has got to."""
 
@@ -206,3 +218,249 @@ class AppSetting(Base):
     key: Mapped[str] = mapped_column(String(80), primary_key=True)
     value: Mapped[dict[str, Any] | None] = mapped_column(JSON, nullable=True)
     updated_at: Mapped[dt.datetime] = mapped_column(DateTime(timezone=True), default=_now, onupdate=_now)
+
+
+# --- shift handover ---------------------------------------------------------
+
+class ShiftName(str, enum.Enum):
+    APAC = "APAC"
+    EMEA = "EMEA"
+    AMER = "AMER"
+
+
+class HandoverStatus(str, enum.Enum):
+    DRAFT = "draft"
+    HANDED_OVER = "handed_over"
+
+
+class ItemState(str, enum.Enum):
+    """The status line CX writes at the bottom of each key update."""
+
+    PENDING_INFRA = "pending_infra"
+    PENDING_CUSTOMER = "pending_customer"
+    PENDING_RUNPOD = "pending_runpod"
+    PENDING_RMA = "pending_rma"
+    PENDING_CX = "pending_cx"
+    IN_PROGRESS = "in_progress"
+    MONITORING = "monitoring"
+    NO_FURTHER_ENGAGEMENT = "no_further_engagement"
+    DONE = "done"
+
+
+class Handover(Base):
+    """One shift handover document."""
+
+    __tablename__ = "handovers"
+    __table_args__ = (UniqueConstraint("shift_date", "shift", name="uq_handover_date_shift"),)
+
+    id: Mapped[int] = mapped_column(primary_key=True)
+    shift_date: Mapped[dt.date] = mapped_column(Date, index=True)
+    shift: Mapped[ShiftName] = mapped_column(Enum(ShiftName), default=ShiftName.APAC)
+    handing_to: Mapped[str] = mapped_column(String(40), default="")   # "EMEA" in "APAC -> EMEA"
+    status: Mapped[HandoverStatus] = mapped_column(Enum(HandoverStatus), default=HandoverStatus.DRAFT)
+
+    team_members: Mapped[str] = mapped_column(String(400), default="")
+    # The tick-boxes at the top of the Confluence page.
+    significant_issues_checked: Mapped[bool] = mapped_column(Boolean, default=False)
+    hubspot_checked: Mapped[bool] = mapped_column(Boolean, default=False)
+    total_open_tickets: Mapped[int | None] = mapped_column(Integer, nullable=True)
+    # [{"name": "Parham", "hs_checked": true, "jira_checked": true}, ...]
+    member_checks: Mapped[list[dict[str, Any]] | None] = mapped_column(JSON, default=list)
+
+    other_comments: Mapped[str] = mapped_column(Text, default="")
+    reviewed_by: Mapped[str] = mapped_column(String(200), default="")
+    reviewed_at_utc: Mapped[str] = mapped_column(String(40), default="")
+    following_shift_checked: Mapped[bool] = mapped_column(Boolean, default=False)
+
+    created_by_id: Mapped[int | None] = mapped_column(ForeignKey("users.id"), nullable=True)
+    created_by: Mapped[User | None] = relationship(lazy="joined")
+    created_at: Mapped[dt.datetime] = mapped_column(DateTime(timezone=True), default=_now)
+    updated_at: Mapped[dt.datetime] = mapped_column(DateTime(timezone=True), default=_now, onupdate=_now)
+
+    items: Mapped[list["HandoverItem"]] = relationship(
+        back_populates="handover", cascade="all, delete-orphan", order_by="HandoverItem.position")
+
+    def to_json(self, with_items: bool = False) -> dict[str, Any]:
+        data = {
+            "id": self.id,
+            "shift_date": self.shift_date.isoformat() if self.shift_date else None,
+            "shift": self.shift.value,
+            "handing_to": self.handing_to,
+            "status": self.status.value,
+            "title": f"{self.shift.value} {self.shift_date.strftime('%d %B %Y')}" if self.shift_date else "",
+            "team_members": self.team_members,
+            "significant_issues_checked": self.significant_issues_checked,
+            "hubspot_checked": self.hubspot_checked,
+            "total_open_tickets": self.total_open_tickets,
+            "member_checks": self.member_checks or [],
+            "other_comments": self.other_comments,
+            "reviewed_by": self.reviewed_by,
+            "reviewed_at_utc": self.reviewed_at_utc,
+            "following_shift_checked": self.following_shift_checked,
+            "created_by": self.created_by.to_json() if self.created_by else None,
+            "updated_at": self.updated_at.isoformat() if self.updated_at else None,
+            "item_count": len(self.items),
+        }
+        if with_items:
+            data["items"] = [i.to_json() for i in self.items]
+        return data
+
+
+class HandoverItem(Base):
+    """One key update: the Zendesk/Jira refs, the narrative, and where it stands.
+
+    Carried forward between shifts rather than retyped - `carried_from_id`
+    records where it came from, so an issue keeps one thread across days.
+    """
+
+    __tablename__ = "handover_items"
+
+    id: Mapped[int] = mapped_column(primary_key=True)
+    handover_id: Mapped[int] = mapped_column(ForeignKey("handovers.id"), index=True)
+    handover: Mapped[Handover] = relationship(back_populates="items")
+
+    position: Mapped[int] = mapped_column(Integer, default=0)
+    title: Mapped[str] = mapped_column(String(400), default="")
+    zendesk_tickets: Mapped[str] = mapped_column(String(300), default="")  # "#8495, #8453"
+    jira_key: Mapped[str] = mapped_column(String(60), default="")
+    jira_status: Mapped[str] = mapped_column(String(60), default="")
+    body: Mapped[str] = mapped_column(Text, default="")
+    links: Mapped[list[str] | None] = mapped_column(JSON, default=list)
+    state: Mapped[ItemState] = mapped_column(Enum(ItemState), default=ItemState.IN_PROGRESS)
+    # "Remove at end of shift" in the doc.
+    remove_at_end_of_shift: Mapped[bool] = mapped_column(Boolean, default=False)
+
+    carried_from_id: Mapped[int | None] = mapped_column(ForeignKey("handover_items.id"), nullable=True)
+    first_raised_on: Mapped[dt.date | None] = mapped_column(Date, nullable=True)
+    created_at: Mapped[dt.datetime] = mapped_column(DateTime(timezone=True), default=_now)
+
+    def to_json(self) -> dict[str, Any]:
+        return {
+            "id": self.id, "position": self.position, "title": self.title,
+            "zendesk_tickets": self.zendesk_tickets, "jira_key": self.jira_key,
+            "jira_status": self.jira_status, "body": self.body, "links": self.links or [],
+            "state": self.state.value,
+            "remove_at_end_of_shift": self.remove_at_end_of_shift,
+            "carried_from_id": self.carried_from_id,
+            "first_raised_on": self.first_raised_on.isoformat() if self.first_raised_on else None,
+        }
+
+
+# --- RunPod -----------------------------------------------------------------
+
+class RunpodColour(str, enum.Enum):
+    """The colour key CX uses on the handover doc, kept verbatim."""
+
+    RED = "red"        # blocked from relisting, recurring issue
+    PURPLE = "purple"  # waiting on RunPod
+    YELLOW = "yellow"  # waiting on Infrastructure or the DC team
+    BLUE = "blue"      # pending removal from the RunPod platform
+    GREEN = "green"    # stress testing >24h, GPUs may look reserved
+    WHITE = "white"    # actionable by CX
+
+
+class RunpodEventType(str, enum.Enum):
+    UNLISTED = "unlisted"
+    LISTED = "listed"
+    MAINTENANCE_SCHEDULED = "maintenance_scheduled"
+    MAINTENANCE_ENDED = "maintenance_ended"
+    DRAINED = "drained"            # unlisted and rented GPUs reached zero
+    NOTE = "note"
+    ZENDESK_TICKET = "zendesk_ticket"
+    JIRA_LINKED = "jira_linked"
+
+
+class RunpodHost(Base):
+    """A RunPod machine, and where CX has got to with it."""
+
+    __tablename__ = "runpod_hosts"
+
+    id: Mapped[int] = mapped_column(primary_key=True)
+    machine_id: Mapped[str] = mapped_column(String(40), unique=True, index=True)
+    name: Mapped[str] = mapped_column(String(200), default="", index=True)
+
+    listed: Mapped[bool] = mapped_column(Boolean, default=True, index=True)
+    gpu_reserved: Mapped[int] = mapped_column(Integer, default=0)
+    gpu_total: Mapped[int] = mapped_column(Integer, default=0)
+    gpu_type: Mapped[str] = mapped_column(String(80), default="")
+    data_center: Mapped[str] = mapped_column(String(40), default="")
+    uptime_one_week: Mapped[float | None] = mapped_column(nullable=True)
+    maintenance_start: Mapped[str] = mapped_column(String(40), default="")
+    maintenance_end: Mapped[str] = mapped_column(String(40), default="")
+
+    # CX-side state, mirrored onto the handover doc.
+    colour: Mapped[RunpodColour] = mapped_column(Enum(RunpodColour), default=RunpodColour.WHITE)
+    zendesk_ticket: Mapped[str] = mapped_column(String(40), default="")
+    jira_key: Mapped[str] = mapped_column(String(60), default="")
+    jira_status: Mapped[str] = mapped_column(String(60), default="")
+    last_error: Mapped[str] = mapped_column(Text, default="")     # hint from the RunPod email
+    next_steps: Mapped[str] = mapped_column(Text, default="")
+
+    # How often this machine has been unlisted; the "problem host" ranking.
+    unlist_count: Mapped[int] = mapped_column(Integer, default=0, index=True)
+    historic_count: Mapped[int] = mapped_column(Integer, default=0)
+    unlisted_at: Mapped[dt.datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
+    last_listed_at: Mapped[dt.datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
+    first_seen_at: Mapped[dt.datetime] = mapped_column(DateTime(timezone=True), default=_now)
+    updated_at: Mapped[dt.datetime] = mapped_column(DateTime(timezone=True), default=_now, onupdate=_now)
+
+    events: Mapped[list["RunpodEvent"]] = relationship(
+        back_populates="host", cascade="all, delete-orphan",
+        order_by="RunpodEvent.occurred_at.desc()")
+
+    @property
+    def hours_unlisted(self) -> float | None:
+        started = _as_utc(self.unlisted_at)
+        if self.listed or not started:
+            return None
+        return round((_now() - started).total_seconds() / 3600, 1)
+
+    def to_json(self, with_events: bool = False) -> dict[str, Any]:
+        data = {
+            "id": self.id, "machine_id": self.machine_id, "name": self.name,
+            "listed": self.listed, "gpu_reserved": self.gpu_reserved, "gpu_total": self.gpu_total,
+            "gpu_type": self.gpu_type, "data_center": self.data_center,
+            "uptime_one_week": self.uptime_one_week,
+            "maintenance_start": self.maintenance_start, "maintenance_end": self.maintenance_end,
+            "colour": self.colour.value, "zendesk_ticket": self.zendesk_ticket,
+            "jira_key": self.jira_key, "jira_status": self.jira_status,
+            "last_error": self.last_error, "next_steps": self.next_steps,
+            "unlist_count": self.unlist_count, "historic_count": self.historic_count,
+            "unlisted_at": self.unlisted_at.isoformat() if self.unlisted_at else None,
+            "hours_unlisted": self.hours_unlisted,
+            "event_count": len(self.events),
+        }
+        if with_events:
+            data["events"] = [e.to_json() for e in self.events]
+        return data
+
+
+class RunpodEvent(Base):
+    """Append-only history for a machine: listings, unlistings and who did what."""
+
+    __tablename__ = "runpod_events"
+
+    id: Mapped[int] = mapped_column(primary_key=True)
+    host_id: Mapped[int] = mapped_column(ForeignKey("runpod_hosts.id"), index=True)
+    host: Mapped[RunpodHost] = relationship(back_populates="events")
+
+    event_type: Mapped[RunpodEventType] = mapped_column(Enum(RunpodEventType), index=True)
+    # "runpod" when RunPod unlisted it automatically, otherwise the operator.
+    actor: Mapped[str] = mapped_column(String(200), default="system")
+    detail: Mapped[str] = mapped_column(Text, default="")
+    error_hint: Mapped[str] = mapped_column(Text, default="")   # parsed from the unlisting email
+    zendesk_ticket: Mapped[str] = mapped_column(String(40), default="")
+    jira_key: Mapped[str] = mapped_column(String(60), default="")
+    gpu_reserved: Mapped[int | None] = mapped_column(Integer, nullable=True)
+    payload: Mapped[dict[str, Any] | None] = mapped_column(JSON, nullable=True)
+    occurred_at: Mapped[dt.datetime] = mapped_column(DateTime(timezone=True), default=_now, index=True)
+
+    def to_json(self) -> dict[str, Any]:
+        return {
+            "id": self.id, "event_type": self.event_type.value, "actor": self.actor,
+            "detail": self.detail, "error_hint": self.error_hint,
+            "zendesk_ticket": self.zendesk_ticket, "jira_key": self.jira_key,
+            "gpu_reserved": self.gpu_reserved,
+            "occurred_at": self.occurred_at.isoformat() if self.occurred_at else None,
+            "payload": self.payload,
+        }

backend/app/routers/handover_router.py

@@ -0,0 +1,222 @@
+"""Shift handover: the document CX fills in at the end of every shift."""
+from __future__ import annotations
+
+import datetime as dt
+
+from fastapi import APIRouter, Depends, HTTPException, status
+from pydantic import BaseModel
+from sqlalchemy import select
+from sqlalchemy.orm import Session
+
+from ..auth import current_user
+from ..db import get_db
+from ..models import (Handover, HandoverItem, HandoverStatus, ItemState, RunpodColour,
+                      RunpodHost, ShiftName, User)
+
+router = APIRouter(prefix="/api/handover", tags=["handover"])
+
+
+class HandoverBody(BaseModel):
+    shift_date: str | None = None
+    shift: str = "APAC"
+    handing_to: str = ""
+    team_members: str = ""
+    significant_issues_checked: bool = False
+    hubspot_checked: bool = False
+    total_open_tickets: int | None = None
+    member_checks: list[dict] = []
+    other_comments: str = ""
+    reviewed_by: str = ""
+    reviewed_at_utc: str = ""
+    following_shift_checked: bool = False
+
+
+class ItemBody(BaseModel):
+    title: str = ""
+    zendesk_tickets: str = ""
+    jira_key: str = ""
+    jira_status: str = ""
+    body: str = ""
+    links: list[str] = []
+    state: str = "in_progress"
+    remove_at_end_of_shift: bool = False
+    position: int | None = None
+
+
+def _get(db: Session, handover_id: int) -> Handover:
+    found = db.get(Handover, handover_id)
+    if not found:
+        raise HTTPException(status.HTTP_404_NOT_FOUND, "No such handover")
+    return found
+
+
+@router.get("")
+def list_handovers(limit: int = 30, db: Session = Depends(get_db),
+                   user: User = Depends(current_user)):
+    rows = db.scalars(select(Handover).order_by(Handover.shift_date.desc(), Handover.id.desc())
+                      .limit(max(1, min(limit, 200)))).all()
+    return {
+        "handovers": [h.to_json() for h in rows],
+        "shifts": [s.value for s in ShiftName],
+        "states": [s.value for s in ItemState],
+    }
+
+
+@router.get("/current")
+def current(db: Session = Depends(get_db), user: User = Depends(current_user)):
+    """The newest draft, or the newest handover of any kind."""
+    row = db.scalars(select(Handover).where(Handover.status == HandoverStatus.DRAFT)
+                     .order_by(Handover.shift_date.desc(), Handover.id.desc()).limit(1)).first()
+    if row is None:
+        row = db.scalars(select(Handover).order_by(Handover.shift_date.desc(),
+                                                   Handover.id.desc()).limit(1)).first()
+    if row is None:
+        return {"handover": None}
+    return {"handover": row.to_json(with_items=True), "runpod": _runpod_section(db)}
+
+
+@router.get("/{handover_id}")
+def get_one(handover_id: int, db: Session = Depends(get_db), user: User = Depends(current_user)):
+    return {"handover": _get(db, handover_id).to_json(with_items=True), "runpod": _runpod_section(db)}
+
+
+def _runpod_section(db: Session) -> list[dict]:
+    """The RunPod table on the handover, straight from live host state.
+
+    This is the part that used to be retyped by hand every shift.
+    """
+    rows = db.scalars(select(RunpodHost).where(RunpodHost.listed.is_(False))
+                      .order_by(RunpodHost.unlist_count.desc(), RunpodHost.name)).all()
+    return [{
+        "machine_id": h.machine_id, "name": h.name, "colour": h.colour.value,
+        "zendesk_ticket": h.zendesk_ticket, "jira_key": h.jira_key, "jira_status": h.jira_status,
+        "gpu_reserved": h.gpu_reserved, "gpu_total": h.gpu_total,
+        "last_error": h.last_error, "next_steps": h.next_steps,
+        "hours_unlisted": h.hours_unlisted, "unlist_count": h.unlist_count,
+    } for h in rows]
+
+
+@router.post("")
+def create(body: HandoverBody, db: Session = Depends(get_db), user: User = Depends(current_user)):
+    shift_date = dt.date.fromisoformat(body.shift_date) if body.shift_date else dt.date.today()
+    try:
+        shift = ShiftName(body.shift)
+    except ValueError as exc:
+        raise HTTPException(status.HTTP_400_BAD_REQUEST, f"Unknown shift '{body.shift}'") from exc
+
+    existing = db.scalars(select(Handover).where(Handover.shift_date == shift_date,
+                                                 Handover.shift == shift)).first()
+    if existing:
+        raise HTTPException(status.HTTP_409_CONFLICT,
+                            f"A {shift.value} handover already exists for {shift_date}")
+
+    row = Handover(shift_date=shift_date, shift=shift, handing_to=body.handing_to,
+                   team_members=body.team_members, created_by_id=user.id)
+    db.add(row)
+    db.commit()
+    return row.to_json(with_items=True)
+
+
+@router.post("/{handover_id}")
+def update(handover_id: int, body: HandoverBody, db: Session = Depends(get_db),
+           user: User = Depends(current_user)):
+    row = _get(db, handover_id)
+    for field in ("handing_to", "team_members", "significant_issues_checked", "hubspot_checked",
+                  "total_open_tickets", "member_checks", "other_comments", "reviewed_by",
+                  "reviewed_at_utc", "following_shift_checked"):
+        setattr(row, field, getattr(body, field))
+    if body.shift_date:
+        row.shift_date = dt.date.fromisoformat(body.shift_date)
+    db.commit()
+    return row.to_json(with_items=True)
+
+
+@router.post("/{handover_id}/hand-over")
+def hand_over(handover_id: int, db: Session = Depends(get_db), user: User = Depends(current_user)):
+    """Close the shift and start the next one, carrying the live items forward.
+
+    Items flagged "remove at end of shift", and anything already done, are left
+    behind - which is the manual step this replaces.
+    """
+    row = _get(db, handover_id)
+    row.status = HandoverStatus.HANDED_OVER
+
+    order = [ShiftName.APAC, ShiftName.EMEA, ShiftName.AMER]
+    next_shift = order[(order.index(row.shift) + 1) % len(order)]
+    next_date = row.shift_date + dt.timedelta(days=1) if next_shift == ShiftName.APAC else row.shift_date
+
+    following = db.scalars(select(Handover).where(Handover.shift_date == next_date,
+                                                  Handover.shift == next_shift)).first()
+    if following is None:
+        following = Handover(shift_date=next_date, shift=next_shift,
+                             handing_to=order[(order.index(next_shift) + 1) % len(order)].value,
+                             created_by_id=user.id)
+        db.add(following)
+        db.flush()
+
+    carried = 0
+    closed = {ItemState.DONE, ItemState.NO_FURTHER_ENGAGEMENT}
+    for item in row.items:
+        if item.remove_at_end_of_shift or item.state in closed:
+            continue
+        following.items.append(HandoverItem(
+            position=item.position, title=item.title, zendesk_tickets=item.zendesk_tickets,
+            jira_key=item.jira_key, jira_status=item.jira_status, body=item.body,
+            links=list(item.links or []), state=item.state,
+            carried_from_id=item.id,
+            first_raised_on=item.first_raised_on or row.shift_date,
+        ))
+        carried += 1
+
+    db.commit()
+    return {"closed": row.to_json(), "next": following.to_json(with_items=True), "carried": carried}
+
+
+@router.post("/{handover_id}/items")
+def add_item(handover_id: int, body: ItemBody, db: Session = Depends(get_db),
+             user: User = Depends(current_user)):
+    row = _get(db, handover_id)
+    try:
+        state = ItemState(body.state)
+    except ValueError as exc:
+        raise HTTPException(status.HTTP_400_BAD_REQUEST, f"Unknown state '{body.state}'") from exc
+    item = HandoverItem(
+        position=body.position if body.position is not None else len(row.items),
+        title=body.title, zendesk_tickets=body.zendesk_tickets, jira_key=body.jira_key,
+        jira_status=body.jira_status, body=body.body, links=body.links, state=state,
+        remove_at_end_of_shift=body.remove_at_end_of_shift, first_raised_on=row.shift_date,
+    )
+    row.items.append(item)
+    db.commit()
+    return row.to_json(with_items=True)
+
+
+@router.post("/{handover_id}/items/{item_id}")
+def update_item(handover_id: int, item_id: int, body: ItemBody, db: Session = Depends(get_db),
+                user: User = Depends(current_user)):
+    row = _get(db, handover_id)
+    item = next((i for i in row.items if i.id == item_id), None)
+    if item is None:
+        raise HTTPException(status.HTTP_404_NOT_FOUND, "No such item on this handover")
+    try:
+        item.state = ItemState(body.state)
+    except ValueError as exc:
+        raise HTTPException(status.HTTP_400_BAD_REQUEST, f"Unknown state '{body.state}'") from exc
+    for field in ("title", "zendesk_tickets", "jira_key", "jira_status", "body", "links",
+                  "remove_at_end_of_shift"):
+        setattr(item, field, getattr(body, field))
+    if body.position is not None:
+        item.position = body.position
+    db.commit()
+    return row.to_json(with_items=True)
+
+
+@router.delete("/{handover_id}/items/{item_id}")
+def delete_item(handover_id: int, item_id: int, db: Session = Depends(get_db),
+                user: User = Depends(current_user)):
+    row = _get(db, handover_id)
+    item = next((i for i in row.items if i.id == item_id), None)
+    if item is not None:
+        db.delete(item)
+        db.commit()
+    return _get(db, handover_id).to_json(with_items=True)

backend/app/routers/runpod_router.py

@@ -0,0 +1,262 @@
+"""RunPod machines: current state, per-host history, and problem ranking."""
+from __future__ import annotations
+
+import datetime as dt
+
+from fastapi import APIRouter, Depends, HTTPException, status
+from pydantic import BaseModel
+from sqlalchemy import func, select
+from sqlalchemy.orm import Session
+
+from ..auth import current_user
+from ..config import get_settings
+from ..db import get_db
+from ..models import RunpodColour, RunpodEvent, RunpodEventType, RunpodHost, User
+from ..runpod.client import RunPodClient, RunPodError
+from ..runpod.email_parse import parse as parse_email
+from ..runpod.delivery import raise_rma_issue, raise_unlisting_ticket, unlisting_ticket
+from ..runpod.service import record_event, sync_machines
+from ..delivery import DeliveryError
+
+router = APIRouter(prefix="/api/runpod", tags=["runpod"])
+settings = get_settings()
+
+
+def _client() -> RunPodClient:
+    return RunPodClient(
+        api_key=settings.runpod_api_key, email=settings.runpod_email,
+        password=settings.runpod_password, team_id=settings.runpod_team_id,
+        totp_secret=settings.runpod_totp_secret,
+    )
+
+
+class HostPatch(BaseModel):
+    colour: str | None = None
+    zendesk_ticket: str | None = None
+    jira_key: str | None = None
+    jira_status: str | None = None
+    next_steps: str | None = None
+    last_error: str | None = None
+
+
+class NoteBody(BaseModel):
+    detail: str
+    event_type: str = "note"
+    zendesk_ticket: str = ""
+    jira_key: str = ""
+
+
+class EmailBody(BaseModel):
+    raw: str
+    subject: str = ""
+
+
+class TicketBody(BaseModel):
+    error_text: str = ""
+    requester: str = ""
+    subject: str = ""
+    body: str = ""
+
+
+class RmaBody(BaseModel):
+    summary: str = ""
+    description: str = ""
+
+
+@router.get("/status")
+def runpod_status(db: Session = Depends(get_db), user: User = Depends(current_user)):
+    hosts = db.scalars(select(RunpodHost)).all()
+    unlisted = [h for h in hosts if not h.listed]
+    return {
+        "configured": settings.runpod_ready,
+        "mode": _client().mode,
+        "write_enabled": settings.feature_runpod_write,
+        "totals": {
+            "machines": len(hosts),
+            "listed": len(hosts) - len(unlisted),
+            "unlisted": len(unlisted),
+            "gpus_rented": sum(h.gpu_reserved for h in hosts),
+            "gpus_total": sum(h.gpu_total for h in hosts),
+        },
+        "colours": [c.value for c in RunpodColour],
+        "event_types": [e.value for e in RunpodEventType],
+    }
+
+
+@router.get("/hosts")
+def hosts(unlisted_only: bool = False, db: Session = Depends(get_db),
+          user: User = Depends(current_user)):
+    query = select(RunpodHost)
+    if unlisted_only:
+        query = query.where(RunpodHost.listed.is_(False))
+    rows = db.scalars(query.order_by(RunpodHost.listed, RunpodHost.name)).all()
+    return {"hosts": [h.to_json() for h in rows]}
+
+
+@router.get("/hosts/{machine_id}")
+def host_detail(machine_id: str, db: Session = Depends(get_db), user: User = Depends(current_user)):
+    row = db.scalars(select(RunpodHost).where(RunpodHost.machine_id == machine_id)).first()
+    if row is None:
+        raise HTTPException(status.HTTP_404_NOT_FOUND, "No such machine")
+    return row.to_json(with_events=True)
+
+
+@router.get("/problem-hosts")
+def problem_hosts(limit: int = 40, db: Session = Depends(get_db),
+                  user: User = Depends(current_user)):
+    """Machines ranked by how often they have been unlisted.
+
+    The repeat offenders are the ones worth an RMA conversation rather than
+    another burn-in, which is what the old script's historical table showed.
+    """
+    unlist_counts = (
+        select(RunpodEvent.host_id, func.count(RunpodEvent.id).label("n"),
+               func.max(RunpodEvent.occurred_at).label("last"))
+        .where(RunpodEvent.event_type == RunpodEventType.UNLISTED)
+        .group_by(RunpodEvent.host_id).subquery()
+    )
+    rows = db.execute(
+        select(RunpodHost, unlist_counts.c.n, unlist_counts.c.last)
+        .join(unlist_counts, unlist_counts.c.host_id == RunpodHost.id, isouter=True)
+        .order_by(func.coalesce(unlist_counts.c.n, 0).desc(), RunpodHost.historic_count.desc())
+        .limit(max(1, min(limit, 200)))
+    ).all()
+
+    out = []
+    for host, count, last in rows:
+        total = int(count or 0) or host.historic_count
+        if not total:
+            continue
+        payload = host.to_json()
+        payload["unlist_events"] = int(count or 0)
+        payload["effective_count"] = total
+        payload["last_unlisted"] = last.isoformat() if last else (
+            host.unlisted_at.isoformat() if host.unlisted_at else None)
+        out.append(payload)
+    return {"hosts": out}
+
+
+@router.post("/sync")
+def sync(db: Session = Depends(get_db), user: User = Depends(current_user)):
+    if not settings.runpod_ready:
+        raise HTTPException(status.HTTP_400_BAD_REQUEST,
+                            "RunPod is not configured. Set CX_RUNPOD_API_KEY.")
+    try:
+        result = sync_machines(db, _client(), actor=user.email)
+    except RunPodError as exc:
+        raise HTTPException(status.HTTP_502_BAD_GATEWAY, str(exc)) from exc
+    return result
+
+
+@router.post("/hosts/{machine_id}")
+def patch_host(machine_id: str, body: HostPatch, db: Session = Depends(get_db),
+               user: User = Depends(current_user)):
+    row = db.scalars(select(RunpodHost).where(RunpodHost.machine_id == machine_id)).first()
+    if row is None:
+        raise HTTPException(status.HTTP_404_NOT_FOUND, "No such machine")
+    changes = []
+    if body.colour is not None:
+        try:
+            row.colour = RunpodColour(body.colour)
+        except ValueError as exc:
+            raise HTTPException(status.HTTP_400_BAD_REQUEST, f"Unknown colour '{body.colour}'") from exc
+        changes.append(f"colour={body.colour}")
+    for field in ("zendesk_ticket", "jira_key", "jira_status", "next_steps", "last_error"):
+        value = getattr(body, field)
+        if value is not None:
+            setattr(row, field, value)
+            changes.append(field)
+    if changes:
+        record_event(db, row, RunpodEventType.NOTE, actor=user.email,
+                     detail="Updated " + ", ".join(changes))
+    db.commit()
+    return row.to_json(with_events=True)
+
+
+@router.post("/hosts/{machine_id}/events")
+def add_event(machine_id: str, body: NoteBody, db: Session = Depends(get_db),
+              user: User = Depends(current_user)):
+    row = db.scalars(select(RunpodHost).where(RunpodHost.machine_id == machine_id)).first()
+    if row is None:
+        raise HTTPException(status.HTTP_404_NOT_FOUND, "No such machine")
+    try:
+        event_type = RunpodEventType(body.event_type)
+    except ValueError as exc:
+        raise HTTPException(status.HTTP_400_BAD_REQUEST, f"Unknown event type") from exc
+    record_event(db, row, event_type, actor=user.email, detail=body.detail,
+                 zendesk_ticket=body.zendesk_ticket, jira_key=body.jira_key)
+    db.commit()
+    return row.to_json(with_events=True)
+
+
+@router.get("/hosts/{machine_id}/ticket-preview")
+def ticket_preview(machine_id: str, db: Session = Depends(get_db),
+                   user: User = Depends(current_user)):
+    """The Zendesk ticket that would be raised, without raising it."""
+    row = db.scalars(select(RunpodHost).where(RunpodHost.machine_id == machine_id)).first()
+    if row is None:
+        raise HTTPException(status.HTTP_404_NOT_FOUND, "No such machine")
+    return {
+        **unlisting_ticket(row),
+        "zendesk_ready": settings.zendesk_ready,
+        "send_enabled": settings.feature_send_enabled,
+        "runpod_jira_ready": settings.runpod_jira_ready,
+        "existing_ticket": row.zendesk_ticket,
+        "existing_jira": row.jira_key,
+    }
+
+
+@router.post("/hosts/{machine_id}/zendesk")
+async def zendesk_ticket(machine_id: str, body: TicketBody, db: Session = Depends(get_db),
+                         user: User = Depends(current_user)):
+    row = db.scalars(select(RunpodHost).where(RunpodHost.machine_id == machine_id)).first()
+    if row is None:
+        raise HTTPException(status.HTTP_404_NOT_FOUND, "No such machine")
+    try:
+        return await raise_unlisting_ticket(db, row, user, error_text=body.error_text,
+                                            requester=body.requester, subject=body.subject,
+                                            body=body.body)
+    except DeliveryError as exc:
+        raise HTTPException(status.HTTP_400_BAD_REQUEST, str(exc)) from exc
+
+
+@router.post("/hosts/{machine_id}/rma")
+async def rma_issue(machine_id: str, body: RmaBody, db: Session = Depends(get_db),
+                    user: User = Depends(current_user)):
+    row = db.scalars(select(RunpodHost).where(RunpodHost.machine_id == machine_id)).first()
+    if row is None:
+        raise HTTPException(status.HTTP_404_NOT_FOUND, "No such machine")
+    try:
+        return await raise_rma_issue(db, row, user, summary=body.summary,
+                                     description=body.description)
+    except DeliveryError as exc:
+        raise HTTPException(status.HTTP_400_BAD_REQUEST, str(exc)) from exc
+
+
+@router.post("/ingest-email")
+def ingest_email(body: EmailBody, db: Session = Depends(get_db), user: User = Depends(current_user)):
+    """Take a RunPod unlisting email and attach its error hint to the machine.
+
+    The email is the only place the actual failure reason appears, so it is
+    worth capturing even when the unlisting itself came through the API.
+    """
+    parsed = parse_email(body.raw, body.subject)
+    if not parsed["machine_id"] and not parsed["host"]:
+        raise HTTPException(status.HTTP_400_BAD_REQUEST,
+                            "Could not find a machine in that email.")
+    query = select(RunpodHost)
+    row = db.scalars(query.where(RunpodHost.machine_id == parsed["machine_id"])).first() \
+        if parsed["machine_id"] else None
+    if row is None and parsed["host"]:
+        row = db.scalars(query.where(RunpodHost.name == parsed["host"])).first()
+    if row is None:
+        return {"parsed": parsed, "matched": False,
+                "detail": "Parsed the email, but no machine in the database matches."}
+
+    row.last_error = parsed["error_text"] or row.last_error
+    record_event(db, row, RunpodEventType.UNLISTED, actor="runpod",
+                 detail=f"{parsed['signature']} — {parsed['suggested_action']}",
+                 error_hint=parsed["error_text"], gpu_reserved=parsed["gpus_rented"],
+                 payload={"source": "email", "critical": parsed["is_critical"]})
+    db.commit()
+    return {"parsed": parsed, "matched": True, "host": row.to_json(with_events=True)}

backend/app/runpod/client.py

@@ -0,0 +1,285 @@
+"""RunPod access.
+
+Two transports, deliberately unequal:
+
+* **API key (default).** One header against the documented GraphQL endpoint.
+  Nothing to expire, nothing to refresh, works from a container.
+* **Console login (fallback).** The old flow: sign in to Clerk with an email and
+  password, then reuse the session JWT. Kept because it is the only path that
+  works if the API key is revoked, but it cannot run unattended.
+
+The "No active session found" failure in the old script was not a UI change:
+the account now has two-factor authentication on. Clerk verifies the password
+and then answers ``status: needs_second_factor`` with an emailed code, so no
+session is ever created. That cannot be automated without reading the mailbox.
+If the account is switched from email codes to an authenticator app, set
+CX_RUNPOD_TOTP_SECRET and this fallback can complete on its own; otherwise
+CX_RUNPOD_2FA_CODE accepts a code for a single manual run.
+
+Set CX_RUNPOD_API_KEY and this module never touches the console at all.
+"""
+from __future__ import annotations
+
+import os
+import threading
+import time
+from typing import Any, Optional
+
+import httpx
+
+GRAPHQL_URL = os.environ.get("CX_RUNPOD_GRAPHQL_URL", "https://api.runpod.io/graphql")
+CLERK_BASE = os.environ.get("CX_RUNPOD_CLERK_BASE", "https://clerk.runpod.io")
+CONSOLE_URL = "https://console.runpod.io"
+CLERK_QS = "__clerk_api_version=2025-11-10&_clerk_js_version=5.125.12"
+
+
+class RunPodError(RuntimeError):
+    pass
+
+
+# --- queries ----------------------------------------------------------------
+
+MACHINES_QUERY = """
+query getMachinesForHostDashboard {
+  myself {
+    machineQuota
+    machines {
+      id name listed registered verified
+      gpuTypeId gpuReserved gpuTotal
+      dataCenterId machineType
+      hostPricePerGpu margin
+      uptimePercentListedOneWeek uptimePercentListedFourWeek
+      maintenanceStart maintenanceEnd
+      gpuType { displayName manufacturer }
+      machineSystem { os cudaVersion kernelVersion }
+    }
+  }
+}
+"""
+
+SUMMARY_QUERY = """
+query getMyMachines {
+  myself {
+    machinesSummary {
+      id displayName listed machineType
+      gpuTypeId gpuRented gpuTotal
+      podProfitPerHr diskProfitPerHr
+      onDemandPods spotPods
+    }
+  }
+}
+"""
+
+LIST_MUTATION = """
+mutation listMachineBulk($input: MachineListBulkInput) { machineListBulk(input: $input) }
+"""
+
+UNLIST_MUTATION = """
+mutation unlistMachineBulk($input: MachineUnlistBulkInput) { machineUnlistBulk(input: $input) }
+"""
+
+MAINTENANCE_MUTATION = """
+mutation machineScheduleMaintenance($input: MachineScheduleMaintenanceInput) {
+  machineScheduleMaintenance(input: $input)
+}
+"""
+
+
+class RunPodClient:
+    def __init__(self, api_key: str = "", email: str = "", password: str = "",
+                 team_id: str = "", totp_secret: str = "", otp_code: str = "",
+                 timeout: int = 30):
+        self.api_key = (api_key or "").strip()
+        self.email = (email or "").strip()
+        self.password = password or ""
+        self.team_id = (team_id or "").strip()
+        self.totp_secret = (totp_secret or "").strip()
+        self.otp_code = (otp_code or "").strip()
+        self.timeout = timeout
+        self._jwt: Optional[str] = None
+        self._jwt_at = 0.0
+        self._lock = threading.Lock()
+
+    # --- auth ---------------------------------------------------------------
+
+    @property
+    def mode(self) -> str:
+        if self.api_key:
+            return "api_key"
+        if self.email and self.password:
+            return "console_login"
+        return "unconfigured"
+
+    def _auth_header(self) -> dict[str, str]:
+        if self.api_key:
+            return {"Authorization": f"Bearer {self.api_key}"}
+        jwt = self._console_jwt()
+        header = {"Authorization": f"Bearer {jwt}"}
+        if self.team_id:
+            header["x-team-id"] = self.team_id
+        return header
+
+    def _console_jwt(self) -> str:
+        """Sign in to the console and return a session JWT.
+
+        Only used when no API key is set. Cached for 50 minutes; Clerk tokens
+        last an hour.
+        """
+        with self._lock:
+            if self._jwt and time.time() - self._jwt_at < 3000:
+                return self._jwt
+            if not (self.email and self.password):
+                raise RunPodError(
+                    "RunPod is not configured. Set CX_RUNPOD_API_KEY, or "
+                    "CX_RUNPOD_EMAIL and CX_RUNPOD_PASSWORD for the console fallback."
+                )
+
+            with httpx.Client(timeout=self.timeout, follow_redirects=True, headers={
+                "User-Agent": "Mozilla/5.0", "Origin": CONSOLE_URL, "Referer": f"{CONSOLE_URL}/",
+            }) as client:
+                client.get(CONSOLE_URL)
+                signin = client.post(
+                    f"{CLERK_BASE}/v1/client/sign_ins?{CLERK_QS}",
+                    data={"identifier": self.email, "password": self.password, "strategy": "password"},
+                )
+                if signin.status_code >= 400:
+                    raise RunPodError(f"Console sign-in rejected ({signin.status_code}): {signin.text[:200]}")
+
+                body = signin.json().get("response") or {}
+                session_id = body.get("created_session_id")
+
+                if body.get("status") == "needs_second_factor":
+                    session_id = self._second_factor(client, body)
+
+                # The old script read the session out of this response and gave
+                # up when it was absent. Ask for the client record instead.
+                jwt, found_id = self._session_from_client(client, session_id)
+                if not jwt and found_id:
+                    jwt = self._mint_token(client, found_id)
+                if not jwt:
+                    raise RunPodError(
+                        "Console sign-in completed but no session token came back. RunPod's console auth has "
+                        "changed again - use CX_RUNPOD_API_KEY instead."
+                    )
+                self._jwt, self._jwt_at = jwt, time.time()
+                return jwt
+
+    def _second_factor(self, client: httpx.Client, body: dict[str, Any]) -> Optional[str]:
+        """Satisfy Clerk's second factor, if we have been given the means to.
+
+        The account currently uses emailed codes, which no unattended process
+        can read - that is the real reason the old login stopped working.
+        """
+        sign_in_id = body.get("id")
+        offered = [f.get("strategy") for f in (body.get("supported_second_factors") or [])]
+
+        code = ""
+        if "totp" in offered and self.totp_secret:
+            code, strategy = self._totp_code(), "totp"
+        elif self.otp_code:
+            code, strategy = self.otp_code, ("email_code" if "email_code" in offered else offered[0] if offered else "")
+            if "email_code" in offered:
+                client.post(f"{CLERK_BASE}/v1/client/sign_ins/{sign_in_id}/prepare_second_factor?{CLERK_QS}",
+                            data={"strategy": "email_code"})
+        if not code:
+            raise RunPodError(
+                "RunPod console login needs a second factor "
+                f"({', '.join(offered) or 'unknown strategy'}) and no code is available. This is why the old "
+                "script failed - the password is accepted, but the account has 2FA on. Use CX_RUNPOD_API_KEY for "
+                "unattended runs, or set CX_RUNPOD_TOTP_SECRET if you move the account to an authenticator app."
+            )
+
+        attempt = client.post(
+            f"{CLERK_BASE}/v1/client/sign_ins/{sign_in_id}/attempt_second_factor?{CLERK_QS}",
+            data={"strategy": strategy, "code": code},
+        )
+        if attempt.status_code >= 400:
+            raise RunPodError(f"Second factor rejected ({attempt.status_code}): {attempt.text[:200]}")
+        return ((attempt.json().get("response") or {}).get("created_session_id"))
+
+    def _totp_code(self) -> str:
+        """RFC 6238 code from a base32 secret - no third-party dependency."""
+        import base64
+        import hashlib
+        import hmac
+        import struct
+
+        secret = self.totp_secret.replace(" ", "").upper()
+        secret += "=" * (-len(secret) % 8)
+        key = base64.b32decode(secret, casefold=True)
+        counter = struct.pack(">Q", int(time.time()) // 30)
+        digest = hmac.new(key, counter, hashlib.sha1).digest()
+        offset = digest[-1] & 0x0F
+        value = struct.unpack(">I", digest[offset:offset + 4])[0] & 0x7FFFFFFF
+        return f"{value % 1_000_000:06d}"
+
+    def _session_from_client(self, client: httpx.Client, prefer_id: Optional[str]) -> tuple[str, str]:
+        resp = client.get(f"{CLERK_BASE}/v1/client?{CLERK_QS}")
+        if resp.status_code >= 400:
+            return "", ""
+        sessions = ((resp.json().get("response") or {}).get("sessions") or [])
+        if not sessions:
+            return "", ""
+        chosen = next((s for s in sessions if s.get("id") == prefer_id), sessions[0])
+        token = ((chosen.get("last_active_token") or {}).get("jwt")) or ""
+        return token, str(chosen.get("id") or "")
+
+    def _mint_token(self, client: httpx.Client, session_id: str) -> str:
+        resp = client.post(f"{CLERK_BASE}/v1/client/sessions/{session_id}/tokens?{CLERK_QS}")
+        if resp.status_code >= 400:
+            return ""
+        return resp.json().get("jwt", "")
+
+    # --- transport ----------------------------------------------------------
+
+    def execute(self, query: str, variables: Optional[dict[str, Any]] = None) -> dict[str, Any]:
+        headers = {"Content-Type": "application/json", **self._auth_header()}
+        payload: dict[str, Any] = {"query": query}
+        if variables is not None:
+            payload["variables"] = variables
+        with httpx.Client(timeout=self.timeout) as client:
+            resp = client.post(GRAPHQL_URL, json=payload, headers=headers)
+        if resp.status_code == 401:
+            raise RunPodError("RunPod rejected the credentials (401). Check CX_RUNPOD_API_KEY.")
+        if resp.status_code >= 400:
+            raise RunPodError(f"RunPod returned {resp.status_code}: {resp.text[:300]}")
+        body = resp.json()
+        if body.get("errors"):
+            raise RunPodError("; ".join(e.get("message", "?") for e in body["errors"])[:400])
+        data = body.get("data")
+        if data is None:
+            raise RunPodError(f"RunPod returned no data: {str(body)[:200]}")
+        return data
+
+    # --- operations ---------------------------------------------------------
+
+    def machines(self) -> list[dict[str, Any]]:
+        data = self.execute(MACHINES_QUERY)
+        return list(((data.get("myself") or {}).get("machines")) or [])
+
+    def summary(self) -> list[dict[str, Any]]:
+        data = self.execute(SUMMARY_QUERY)
+        return list(((data.get("myself") or {}).get("machinesSummary")) or [])
+
+    def list_machines(self, machine_ids: list[str]) -> dict[str, Any]:
+        return self.execute(LIST_MUTATION, {"input": {"machineIds": machine_ids}})
+
+    def unlist_machines(self, machine_ids: list[str]) -> dict[str, Any]:
+        return self.execute(UNLIST_MUTATION, {"input": {"machineIds": machine_ids}})
+
+    def schedule_maintenance(self, machine_ids: list[str], start_utc: str, minutes: int,
+                             reason: str = "EMERGENCY", destructive: bool = False) -> dict[str, Any]:
+        if reason not in ("UPGRADE", "ROUTINE", "EMERGENCY", "REMOVE"):
+            raise RunPodError(f"Unknown maintenance reason '{reason}'.")
+        return self.execute(MAINTENANCE_MUTATION, {"input": {
+            "machineIds": machine_ids, "maintenanceStartUtc": start_utc,
+            "maintenanceMinutes": minutes, "maintenanceReason": reason, "destructive": destructive,
+        }})
+
+    def check(self) -> dict[str, Any]:
+        """Cheap connectivity probe for the health endpoint."""
+        try:
+            machines = self.machines()
+            return {"ok": True, "mode": self.mode, "machines": len(machines)}
+        except RunPodError as exc:
+            return {"ok": False, "mode": self.mode, "error": str(exc)}

backend/app/runpod/delivery.py

@@ -0,0 +1,201 @@
+"""Outbound actions for RunPod hosts: Zendesk notification and Jira/RMA escalation.
+
+An unlisted machine earns nothing and may be stranding rented workloads, so the
+team wants a ticket the moment it happens. RunPod's own email already says what
+broke; this reproduces that content in Zendesk so it lands in the tracking
+platform even when the unlisting was done through the API and no email was sent.
+
+The same three gates as customer comms apply - configured, feature-flagged, and
+sending enabled globally - so a demo instance cannot raise tickets.
+"""
+from __future__ import annotations
+
+import datetime as dt
+from typing import Any, Optional
+
+import httpx
+from sqlalchemy.orm import Session
+
+from ..config import get_settings
+from ..delivery import DeliveryError, _check_cap
+from ..models import RunpodEventType, RunpodHost, User
+from .email_parse import classify
+from .service import record_event
+
+settings = get_settings()
+
+
+def unlisting_ticket(host: RunpodHost, error_text: str = "") -> dict[str, str]:
+    """Compose the ticket body, following the shape of RunPod's own email.
+
+    Keeping the same structure means whoever picks the ticket up reads the
+    familiar thing: what broke, how many GPUs it is costing, and where to look.
+    """
+    error = (error_text or host.last_error or "").strip()
+    hint = classify(error)
+    rented = f"{host.gpu_reserved} GPU(s) currently rented on this machine." if host.gpu_reserved \
+        else "No GPUs currently rented - the machine has drained."
+
+    body = "\n".join([
+        f"Machine {host.name} ({host.machine_id}) is unlisted and is not accepting new users.",
+        "",
+        "Error detected:",
+        error or "(no error text captured - check the RunPod dashboard or the notification email)",
+        "",
+        f"Impact: {rented}",
+        f"Likely cause: {hint['signature']}",
+        f"Suggested next step: {hint['suggested_action']}",
+        "",
+        "Common causes and where to look:",
+        "- GPU error or failure: check nvidia-smi for GPU health and dmesg for Xid errors.",
+        "- Unresponsive Docker daemon: check whether the docker service is running or hung.",
+        "- Pod sync errors: if df -h hangs, suspect a disk error or hung moosefs mount.",
+        "- Docker overlay storage: confirm the docker filesystem is XFS and /var/lib/docker is mounted.",
+        "- Portallocator port check: verify the publicIp ports in /etc/runpod/config.json are reachable.",
+        "",
+        "If the fix needs a reboot or hardware work, schedule maintenance from the Machines "
+        "Dashboard rather than pulling the machine abruptly - that drains workloads gracefully.",
+        "",
+        f"Unlisted {host.unlist_count} time(s) to date.",
+        "Raised automatically by CX Triage.",
+    ])
+    return {
+        "subject": f"{host.name} Unlisted - {hint['signature']}",
+        "body": body,
+        "signature": hint["signature"],
+    }
+
+
+async def raise_unlisting_ticket(db: Session, host: RunpodHost, actor: User,
+                                 *, error_text: str = "", requester: str = "",
+                                 subject: str = "", body: str = "") -> dict[str, Any]:
+    """Create (or comment on) the Zendesk ticket for an unlisted machine."""
+    if not settings.feature_send_enabled:
+        raise DeliveryError("Sending is disabled on this instance (CX_FEATURE_SEND_ENABLED is off).")
+    if not settings.zendesk_ready:
+        raise DeliveryError("Zendesk is not configured. Set CX_FEATURE_ZENDESK plus "
+                            "CX_ZENDESK_SUBDOMAIN, CX_ZENDESK_EMAIL and CX_ZENDESK_TOKEN.")
+    _check_cap(db)
+
+    composed = unlisting_ticket(host, error_text)
+    subject = subject or composed["subject"]
+    body = body or composed["body"]
+
+    base = f"https://{settings.zendesk_subdomain}.zendesk.com/api/v2"
+    auth = (f"{settings.zendesk_email}/token", settings.zendesk_token)
+    # Keyed on the machine, not the incident: one machine, one running thread.
+    external_id = f"cx-triage-runpod-{host.machine_id}"
+
+    async with httpx.AsyncClient(timeout=30) as client:
+        found = await client.get(f"{base}/search.json",
+                                 params={"query": f'type:ticket external_id:"{external_id}"'}, auth=auth)
+        existing = None
+        if found.status_code == 200:
+            results = found.json().get("results") or []
+            # Only reuse a ticket that is still open; a solved one starts a new thread.
+            existing = next((t for t in results if t.get("status") not in ("solved", "closed")), None)
+
+        # Internal note by default: this is an operations ticket, not a customer reply.
+        comment = {"body": body, "public": False}
+        if existing:
+            resp = await client.put(f"{base}/tickets/{existing['id']}.json",
+                                    json={"ticket": {"comment": comment}}, auth=auth)
+            action = "updated"
+        else:
+            ticket: dict[str, Any] = {
+                "subject": subject,
+                "comment": comment,
+                "priority": "high" if host.gpu_reserved else "normal",
+                "type": "incident",
+                "tags": ["cx-triage", "runpod", "unlisted", f"dc-{(host.data_center or 'unknown').lower()}"],
+                "external_id": external_id,
+            }
+            if requester:
+                ticket["requester"] = {"name": requester.split("@")[0], "email": requester}
+            resp = await client.post(f"{base}/tickets.json", json={"ticket": ticket}, auth=auth)
+            action = "created"
+
+    if resp.status_code not in (200, 201):
+        record_event(db, host, RunpodEventType.NOTE, actor=actor.email,
+                     detail=f"Zendesk ticket failed: HTTP {resp.status_code}")
+        db.commit()
+        raise DeliveryError(f"Zendesk returned {resp.status_code}: {resp.text[:300]}")
+
+    ticket_id = str((resp.json().get("ticket") or {}).get("id")
+                    or (existing or {}).get("id") or "")
+    url = f"https://{settings.zendesk_subdomain}.zendesk.com/agent/tickets/{ticket_id}"
+
+    host.zendesk_ticket = ticket_id or host.zendesk_ticket
+    record_event(db, host, RunpodEventType.ZENDESK_TICKET, actor=actor.email,
+                 detail=f"Zendesk ticket {ticket_id} {action} - {composed['signature']}",
+                 error_hint=error_text or host.last_error, zendesk_ticket=ticket_id)
+    db.commit()
+    return {"ok": True, "ticket_id": ticket_id, "url": url, "action": action,
+            "subject": subject, "signature": composed["signature"]}
+
+
+async def raise_rma_issue(db: Session, host: RunpodHost, actor: User, *,
+                          summary: str = "", description: str = "") -> dict[str, Any]:
+    """Open an issue on the RunPod/RMA Jira - a different instance to the OIE one."""
+    creds = settings.jira_for("runpod")
+    if not settings.feature_send_enabled:
+        raise DeliveryError("Sending is disabled on this instance (CX_FEATURE_SEND_ENABLED is off).")
+    if not (settings.feature_jira and creds["base"] and creds["email"] and creds["token"]):
+        raise DeliveryError(
+            "The RunPod Jira is not configured. Set CX_RUNPOD_JIRA_BASE, CX_RUNPOD_JIRA_EMAIL and "
+            "CX_RUNPOD_JIRA_TOKEN - or leave them blank to reuse the default instance."
+        )
+    _check_cap(db)
+
+    hint = classify(host.last_error or "")
+    summary = summary or f"RunPod host {host.name} ({host.machine_id}) - {hint['signature']}"
+    description = description or "\n".join([
+        f"Machine: {host.name} ({host.machine_id})",
+        f"Data centre: {host.data_center or 'unknown'}",
+        f"GPUs: {host.gpu_reserved}/{host.gpu_total} rented",
+        f"Unlisted {host.unlist_count} time(s) to date.",
+        "",
+        "Last error:",
+        host.last_error or "(none captured)",
+        "",
+        f"Likely cause: {hint['signature']}",
+        f"Suggested next step: {hint['suggested_action']}",
+        "",
+        f"Zendesk: {host.zendesk_ticket or '(none)'}",
+        "Raised from CX Triage.",
+    ])
+
+    base = creds["base"].rstrip("/")
+    auth = (creds["email"], creds["token"])
+    label = f"cx-triage-runpod-{host.machine_id}"
+
+    async with httpx.AsyncClient(timeout=30) as client:
+        found = await client.get(f"{base}/rest/api/3/search",
+                                 params={"jql": f'labels = "{label}"', "maxResults": 1}, auth=auth)
+        if found.status_code == 200 and (found.json().get("issues") or []):
+            issue = found.json()["issues"][0]
+            key = issue["key"]
+            host.jira_key = key
+            record_event(db, host, RunpodEventType.JIRA_LINKED, actor=actor.email,
+                         detail=f"Issue {key} already exists", jira_key=key)
+            db.commit()
+            return {"ok": True, "key": key, "url": f"{base}/browse/{key}", "action": "existing"}
+
+        resp = await client.post(f"{base}/rest/api/3/issue", json={"fields": {
+            "project": {"key": creds["project"]},
+            "summary": summary[:250],
+            "issuetype": {"name": creds["issue_type"]},
+            "labels": ["cx-triage", "runpod", label],
+            "description": {"type": "doc", "version": 1, "content": [
+                {"type": "paragraph", "content": [{"type": "text", "text": description[:30000]}]}]},
+        }}, auth=auth)
+
+    if resp.status_code not in (200, 201):
+        raise DeliveryError(f"Jira returned {resp.status_code}: {resp.text[:300]}")
+
+    key = resp.json().get("key", "")
+    host.jira_key = key
+    record_event(db, host, RunpodEventType.JIRA_LINKED, actor=actor.email,
+                 detail=f"Issue {key} created on the RunPod Jira", jira_key=key)
+    db.commit()
+    return {"ok": True, "key": key, "url": f"{base}/browse/{key}", "action": "created"}

backend/app/runpod/email_parse.py

@@ -0,0 +1,114 @@
+"""Parse RunPod's automated unlisting emails.
+
+Subject:  "<host> Unlisted - CRITICAL ERROR"
+Body:     the machine id in brackets, an "Error detected:" block, and an
+          "Impact: N GPU(s) currently rented" line.
+
+The error block is the useful part - it is the only place the actual reason
+appears, and it is what CX pastes into the Zendesk ticket and the handover.
+Written to cope with the HTML mail as well as a plain-text paste.
+"""
+from __future__ import annotations
+
+import html
+import quopri
+import re
+from typing import Any
+
+SUBJECT_RE = re.compile(r"^(?P<host>[\w.-]+)\s+Unlisted\b", re.I)
+MACHINE_RE = re.compile(r"machine\s+(?P<host>[\w.-]+)\s*\((?P<machine_id>[a-z0-9]{8,})\)", re.I)
+IMPACT_RE = re.compile(r"Impact:\s*(?P<gpus>\d+)\s*GPU", re.I)
+
+ERROR_START = re.compile(r"Error detected:\s*", re.I)
+ERROR_END = re.compile(r"(Impact:|Common causes|Need to take the machine offline)", re.I)
+
+# Recognisable failure signatures, mapped to what CX should do about them.
+SIGNATURES: list[tuple[str, str, str]] = [
+    (r"memory_remap|uncorrectable remapped memory",
+     "GPU memory remapping failure", "Hardware. Likely RMA - raise with the vendor."),
+    (r"xid", "XID error", "Check dmesg and nvidia-smi; usually a burn-in test before relisting."),
+    (r"gpu_cuda_ok.*expected 1, got 0|cuda initialization|fallen off the bus",
+     "GPU not visible to CUDA", "Check nvidia-smi and the PCIe link; reboot then burn-in."),
+    (r"nvidia-smi.*too many failures", "nvidia-smi failing repeatedly",
+     "Host-level GPU fault - stress test, then RMA if it repeats."),
+    (r"docker (service )?(unresponsive|hung)|container stuck",
+     "Docker daemon unresponsive", "Restart docker; check /var/lib/docker is XFS and mounted."),
+    (r"pod sync (failed|errors)", "Pod sync failing",
+     "Check df -h for a hung moosefs mount and network to the moosefs cluster."),
+    (r"portallocator|public port check fail", "Public port check failing",
+     "Verify the publicIp ports in /etc/runpod/config.json are reachable."),
+    (r"disk|filesystem|xfs", "Disk or filesystem error", "Check dmesg for disk failures."),
+]
+
+
+# Soft line breaks ("=\n") and "=3D" are the giveaways for a quoted-printable
+# body, which is how these mails arrive. Left undecoded, the error block comes
+# out full of "=3D" and split mid-word.
+_QP_HINT = re.compile(r"=\r?\n|=[0-9A-F]{2}")
+
+
+def _maybe_decode_qp(raw: str) -> str:
+    if not _QP_HINT.search(raw):
+        return raw
+    try:
+        return quopri.decodestring(raw.encode("utf-8", "replace")).decode("utf-8", "replace")
+    except Exception:
+        return raw
+
+
+def _text_from_html(raw: str) -> str:
+    text = re.sub(r"<(script|style)[^>]*>.*?</\1>", "", raw, flags=re.S | re.I)
+    text = re.sub(r"<br\s*/?>|</(p|div|tr|li|h[1-6]|table)>", "\n", text, flags=re.I)
+    text = re.sub(r"<[^>]+>", "", text)
+    return html.unescape(text)
+
+
+def classify(error_text: str) -> dict[str, str]:
+    """Name the failure and say what it usually means."""
+    blob = (error_text or "").lower()
+    for pattern, label, action in SIGNATURES:
+        if re.search(pattern, blob, re.I):
+            return {"signature": label, "suggested_action": action}
+    return {"signature": "Unrecognised error",
+            "suggested_action": "Read the error text and escalate if it is not obvious."}
+
+
+def parse(raw: str, subject: str = "") -> dict[str, Any]:
+    """Pull the machine, the error block and the rented-GPU impact out of an email."""
+    raw = _maybe_decode_qp(raw)
+    text = _text_from_html(raw) if "<" in raw and ">" in raw else raw
+    lines = [ln.strip() for ln in text.splitlines()]
+    body = "\n".join(ln for ln in lines if ln)
+
+    host = ""
+    machine_id = ""
+    match = MACHINE_RE.search(body)
+    if match:
+        host, machine_id = match.group("host"), match.group("machine_id")
+    if not host and subject:
+        subject_match = SUBJECT_RE.match(subject.strip())
+        if subject_match:
+            host = subject_match.group("host")
+
+    error_text = ""
+    start = ERROR_START.search(body)
+    if start:
+        rest = body[start.end():]
+        end = ERROR_END.search(rest)
+        error_text = (rest[:end.start()] if end else rest).strip()
+        # The mail repeats its own body; one copy is enough.
+        error_text = "\n".join(dict.fromkeys(ln for ln in error_text.splitlines() if ln.strip() != "---"))
+
+    gpus = None
+    impact = IMPACT_RE.search(body)
+    if impact:
+        gpus = int(impact.group("gpus"))
+
+    return {
+        "host": host,
+        "machine_id": machine_id,
+        "error_text": error_text.strip(),
+        "gpus_rented": gpus,
+        "is_critical": "critical" in (subject or body).lower(),
+        **classify(error_text),
+    }

backend/app/runpod/service.py

@@ -0,0 +1,102 @@
+"""Turning RunPod machine state into tracked history.
+
+The API only ever reports what is true now. Everything CX cares about - how
+often a machine has dropped out, when it drained, who relisted it - only exists
+if each sync writes down what changed.
+"""
+from __future__ import annotations
+
+import datetime as dt
+from typing import Any, Optional
+
+from sqlalchemy import select
+from sqlalchemy.orm import Session
+
+from ..models import RunpodColour, RunpodEvent, RunpodEventType, RunpodHost
+
+
+def record_event(db: Session, host: RunpodHost, event_type: RunpodEventType, *,
+                 actor: str = "system", detail: str = "", error_hint: str = "",
+                 zendesk_ticket: str = "", jira_key: str = "",
+                 gpu_reserved: Optional[int] = None,
+                 payload: Optional[dict[str, Any]] = None) -> RunpodEvent:
+    event = RunpodEvent(
+        event_type=event_type, actor=actor, detail=detail, error_hint=error_hint,
+        zendesk_ticket=zendesk_ticket, jira_key=jira_key,
+        gpu_reserved=host.gpu_reserved if gpu_reserved is None else gpu_reserved,
+        payload=payload,
+    )
+    # Through the relationship so an already-loaded history stays correct.
+    host.events.append(event)
+    db.add(event)
+    return event
+
+
+def sync_machines(db: Session, client: Any, actor: str = "system") -> dict[str, Any]:
+    """Pull current machines and write down every transition since last time."""
+    machines = client.machines()
+    existing = {h.machine_id: h for h in db.scalars(select(RunpodHost)).all()}
+    now = dt.datetime.now(dt.timezone.utc)
+
+    created = unlisted = relisted = drained = 0
+
+    for machine in machines:
+        machine_id = str(machine.get("id") or "")
+        if not machine_id:
+            continue
+        listed = bool(machine.get("listed"))
+        reserved = int(machine.get("gpuReserved") or 0)
+
+        host = existing.get(machine_id)
+        if host is None:
+            host = RunpodHost(machine_id=machine_id, name=str(machine.get("name") or ""))
+            db.add(host)
+            db.flush()
+            created += 1
+            record_event(db, host, RunpodEventType.LISTED if listed else RunpodEventType.UNLISTED,
+                         actor="runpod", detail="First seen by CX Triage")
+            if not listed:
+                host.unlisted_at = now
+
+        was_listed = host.listed
+        was_reserved = host.gpu_reserved
+
+        if was_listed and not listed:
+            unlisted += 1
+            host.unlisted_at = now
+            host.unlist_count += 1
+            # Nothing in the API says why; the email carries the reason and is
+            # attached separately through /ingest-email.
+            record_event(db, host, RunpodEventType.UNLISTED, actor="runpod",
+                         detail="Unlisted (detected on sync)", gpu_reserved=reserved)
+        elif not was_listed and listed:
+            relisted += 1
+            host.last_listed_at = now
+            host.unlisted_at = None
+            record_event(db, host, RunpodEventType.LISTED, actor=actor,
+                         detail="Relisted", gpu_reserved=reserved)
+
+        # Unlisted and the last renter has gone: the machine is now safe to work on.
+        if not listed and was_reserved > 0 and reserved == 0:
+            drained += 1
+            record_event(db, host, RunpodEventType.DRAINED, actor="runpod",
+                         detail="Unlisted machine has drained - no rented GPUs left",
+                         gpu_reserved=0)
+
+        gpu_type = machine.get("gpuType") or {}
+        host.name = str(machine.get("name") or host.name)
+        host.listed = listed
+        host.gpu_reserved = reserved
+        host.gpu_total = int(machine.get("gpuTotal") or 0)
+        host.gpu_type = str(gpu_type.get("displayName") or machine.get("gpuTypeId") or "")
+        host.data_center = str(machine.get("dataCenterId") or "")
+        host.uptime_one_week = machine.get("uptimePercentListedOneWeek")
+        host.maintenance_start = str(machine.get("maintenanceStart") or "")
+        host.maintenance_end = str(machine.get("maintenanceEnd") or "")
+
+    db.commit()
+    return {
+        "machines": len(machines), "created": created, "unlisted": unlisted,
+        "relisted": relisted, "drained": drained,
+        "synced_at": now.isoformat(),
+    }

backend/app/seed/data.py

@@ -0,0 +1,174 @@
+"""Seed data drawn from the real exports, so a fresh stack is worth looking at.
+
+The handover comes from APAC 06 August 2026; the RunPod hosts come from the
+monitoring script's own JSON. Only enough is transcribed to make the pages
+representative - it is sample data, not a migration.
+"""
+from __future__ import annotations
+
+HANDOVER = {
+    "shift_date": "2026-08-06",
+    "shift": "APAC",
+    "handing_to": "EMEA",
+    "team_members": "Parham",
+    "significant_issues_checked": True,
+    "hubspot_checked": True,
+    "total_open_tickets": 214,
+    "member_checks": [
+        {"name": "Prasad", "hs_checked": False, "jira_checked": False},
+        {"name": "Parham", "hs_checked": True, "jira_checked": True},
+    ],
+    "other_comments": (
+        "Weekend Coverage Annual Leave Sheet is on SharePoint.\n\n"
+        "Confluence Sanity Check: going through current docs to see what needs updating "
+        "and removing.\n\n"
+        "Kodekloud Training: more team time needed on Kubernetes CKNA. Team leads are "
+        "working on a schedule.\n\n"
+        "New runbook WIP to protect customers and infrastructure from suspicious activity - "
+        "locking a VM and, if required, detaching a public IP allocation.\n\n"
+        "For any Windmill flows that require a ticket reference, use the numeric number and "
+        "not the URL for now.\n\n"
+        "Check the Zendesk sync tool - some Hyperstack orgs are not showing in Zendesk even "
+        "though they exist in InfraInsight (e.g. org 29084, 29548)."
+    ),
+    "items": [
+        {
+            "title": "Request to detach volume from hibernated VM",
+            "zendesk_tickets": "#8495",
+            "state": "no_further_engagement",
+            "remove_at_end_of_shift": True,
+            "body": ("Details confirmed. Ran the Windmill flow but hit an error on API keys. "
+                     "Volume has since been detached successfully and the customer has been informed."),
+            "links": ["https://windmill.ngbackend.cloud/run/019fd28c-662b-3b56-199a-6af73bb1993c"],
+        },
+        {
+            "title": "Baseten node-level failure on b200-worker-frank-quetzal",
+            "zendesk_tickets": "#8453",
+            "state": "pending_infra",
+            "body": ("Aranya reached out on behalf of Baseten about a node-level failure "
+                     "(internal IP 10.32.10.2). Triage provided three times; the first two uploads "
+                     "showed no issues. Customer still reports the node is wedged. Waiting on the "
+                     "third upload for review. Asked Infra whether the nodes just need a reboot."),
+            "links": ["https://nexgen-cloud.slack.com/archives/C0AP4KV1XT5/p1785827125227079"],
+        },
+        {
+            "title": "[NO1] Investigate inbound traffic spike",
+            "zendesk_tickets": "8238",
+            "jira_key": "OIE-3213", "jira_status": "Complete",
+            "state": "pending_customer",
+            "body": ("CARMA Media Insight (org 3429) reported a transatlantic throughput collapse - "
+                     "EU fine, North America down to ~45-141 KB/s. Kheano traced it to edge-level "
+                     "congestion in NO1 affecting all tenants, top talker FIP 149.36.0.199 showing "
+                     "signs of a flood attack. Infra have confirmed WAN utilisation has normalised."),
+            "links": ["https://nexgen-cloud.slack.com/archives/C049Q9JRGM7/p1785526451626789"],
+        },
+        {
+            "title": "[EU1-BM2] Reported network latency investigation",
+            "zendesk_tickets": "#8409",
+            "jira_key": "OIE-3185", "jira_status": "In Progress",
+            "state": "pending_infra",
+            "body": "Baseten experienced a latency spike on 31 July and have requested investigation.",
+            "links": ["https://nexgen-cloud.slack.com/archives/C0AP4KV1XT5/p1784855398156009"],
+        },
+        {
+            "title": "Network storage speed",
+            "zendesk_tickets": "#8207",
+            "jira_key": "OIE-3210", "jira_status": "In Progress",
+            "state": "pending_infra",
+            "body": ("RunPod advised one of their machines is reporting network slowness to their "
+                     "storage. Confirmed against another machine in the same region using the same "
+                     "storage by creating files of random data."),
+            "links": ["https://nexgen-cloud.slack.com/archives/C049Q9JRGM7/p1785451129222409"],
+        },
+        {
+            "title": "[EU1-BM2] GPU issues on eu1-bm2-lv1-b200sxm-0(16|55)",
+            "zendesk_tickets": "#7772",
+            "jira_key": "OIE-3177", "jira_status": "Pending",
+            "state": "pending_infra",
+            "body": ("Baseten requested support on 2 nodes with XID errors. One looks transient; "
+                     "eu1-bm2-lv1-b200sxm-055 is more concerning with UECC. Raised with Lenovo for "
+                     "next steps."),
+            "links": ["https://nexgen-cloud.slack.com/archives/C0AP4KV1XT5/p1784749263687239"],
+        },
+        {
+            "title": "Shadeform locked VM - billing adjustment",
+            "zendesk_tickets": "#8215",
+            "state": "no_further_engagement",
+            "remove_at_end_of_shift": True,
+            "body": ("DevOps raised a ticket to notify the user their VM was locked (945280 / "
+                     "O6QTQyQgqDRN, Shadeform) due to unusual activity. Customer asked for it to be "
+                     "deleted; we unlocked and they deleted it. Locked for 5 days 3 hours 21 minutes "
+                     "= 7401 minutes = 123.35 hours at $1.6756/hr. Total credit $206.69 to come off "
+                     "the next invoice, since the state was never changed to shutoff/locked."),
+        },
+        {
+            "title": "185.216.20.188 network slowness - CA1-SRV-CPU6",
+            "zendesk_tickets": "#7793",
+            "jira_key": "OIE-3189", "jira_status": "Done",
+            "state": "pending_customer",
+            "body": ("Customer reported intermittent network instability on VM 298953 "
+                     "(CA1-SRV-CPU6, 185.216.20.188). The problematic VM 924706 / JqXcTaGOt has been "
+                     "siloed by Infra and the routers investigated. Network has been stable for the "
+                     "past couple of days."),
+        },
+        {
+            "title": "VM stuck - CA1-ESC8-068 host not reachable",
+            "zendesk_tickets": "#8078, #8076, #7787, #8077",
+            "jira_key": "OIE-3183", "jira_status": "In Progress",
+            "state": "pending_customer",
+            "body": ("Host maintenance scheduled 06 August 2026 at 00:00 UTC. Communication sent to "
+                     "the customer. Four VMs still on the host and reachable: vm-regA (786651), "
+                     "dubbix_instance_1 (737376), gianpaolo-dev-2 (602721), triton-backup-2 (213340). "
+                     "Follow-up emails sent. Pending customer reply on the maintenance window."),
+        },
+    ],
+}
+
+# The colour key from the handover doc, so the UI can explain itself.
+RUNPOD_COLOURS = {
+    "red": "Blocked from being relisted due to recurring issues. Needs in-depth investigation or is tied to a system issue.",
+    "purple": "Pending RunPod. We are waiting on RunPod for something.",
+    "yellow": "Unactionable by CX. Pending investigation or remediation from Infrastructure or the DC team.",
+    "blue": "Unactionable by CX. Pending removal from the RunPod platform.",
+    "green": "Stress testing for >24 hours. GPUs may appear reserved.",
+    "white": "Actionable by CX.",
+}
+
+# The unlisted table from the same handover, with the states CX had recorded.
+RUNPOD_BOARD = [
+    {"name": "no1-os1-5090-016-contract-001", "machine_id": "hedh664udd3g", "zendesk_ticket": "8405",
+     "colour": "green", "last_error": "std burn in failed on 03 Aug",
+     "next_steps": "24 Hr default burn-in test in progress. 05 Aug 2026 15:06 UTC (16:06 BST)"},
+    {"name": "no1-os1-5090-099-contract-001", "machine_id": "9bpfxa1o9our", "zendesk_ticket": "6279",
+     "colour": "red", "jira_key": "RMA-103", "jira_status": "Waiting",
+     "last_error": "nvidia-smi: too many failures. Raised for RMA",
+     "next_steps": "24 HR burn-in started 05 Aug 2026 18:38 UTC. Luis asked for multiple 24hr stress tests with full logs in Jira."},
+    {"name": "no1-os1-5090-027-contract-001", "machine_id": "seed-5090-027", "zendesk_ticket": "#8385",
+     "colour": "green", "last_error": "nvidia-smi many failures. First burn-in failed so running again",
+     "next_steps": "24HR burn-in running, started 05 Aug 2026 20:15 UTC (21:15 BST)"},
+    {"name": "no1-os1-5090-050", "machine_id": "gpc1q3vboxfo", "zendesk_ticket": "#8497",
+     "colour": "green",
+     "last_error": "dcgm-xid-check: potential XID issue detected\ngpu health check failed: metric gpu_cuda_ok: expected 1, got 0",
+     "next_steps": "24HR burn-in running, started 05 Aug 2026 17:28 UTC (18:28 BST)"},
+    {"name": "no1-os1-4090-009", "machine_id": "f2vwayvyxorv", "zendesk_ticket": "#7342",
+     "colour": "yellow", "last_error": "nvidia-smi: too many failures. 2/8 GPU in use",
+     "next_steps": "Maintenance scheduled 4 Aug 2026 19:00 UTC. Host has connectivity issues - may need escalating."},
+    {"name": "no1-os1-4090-020", "machine_id": "s7kl180cusjr", "zendesk_ticket": "#7359",
+     "colour": "yellow", "last_error": "pod sync failed 16 times. 2/8 GPU in use",
+     "next_steps": "Maintenance scheduled 04 Aug 2026 13:02 UTC for 1 day"},
+    {"name": "ca1-esc8-106", "machine_id": "x0gn8v2rthk4", "zendesk_ticket": "419195749569",
+     "colour": "red", "jira_status": "Under Test",
+     "last_error": ("dcgm-xid-check: potential XID issue detected\n"
+                    "gpu health check failed: error indicator present: "
+                    "gpu_failed{reason=memory_remap,uuid=GPU-d29f591c-63e9-5f1d-add1-ed0e2dad3660}"),
+     "next_steps": "Repeated offender - does not stay listed for more than 48 hours. Booked maintenance 17 Jun 2026 14:52 BST."},
+    {"name": "ca1-esc8-121", "machine_id": "seed-esc8-121", "zendesk_ticket": "419270908096",
+     "colour": "yellow", "jira_status": "Maintenance Scheduled",
+     "last_error": ("dcgm-xid-check: potential XID issue detected\n"
+                    "gpu health check failed: metric gpu_cuda_ok: expected 1, got 0"),
+     "next_steps": "3/8 in use. Maintenance 18 Jun 2026 19:00 UTC (20:00 BST)"},
+    {"name": "no1-os1-4090-013", "machine_id": "seed-4090-013", "zendesk_ticket": "418292796638",
+     "colour": "blue", "jira_key": "RMA-60", "jira_status": "Pending RMA",
+     "last_error": "container stuck: docker service unresponsive",
+     "next_steps": "Reboot failed, received timeout error. Pending RMA."},
+]

backend/app/seed/loader.py

@@ -0,0 +1,150 @@
+"""Populate an empty database with representative data.
+
+Runs on startup when CX_SEED_DEMO is on, and only when the relevant table is
+empty, so it never overwrites real work.
+"""
+from __future__ import annotations
+
+import datetime as dt
+import json
+import os
+import random
+from pathlib import Path
+
+from sqlalchemy import select
+from sqlalchemy.orm import Session
+
+from ..models import (AppSetting, Handover, HandoverItem, HandoverStatus, ItemState,
+                      RunpodColour, RunpodEventType, RunpodHost, ShiftName)
+from ..runpod.service import record_event
+from .data import HANDOVER, RUNPOD_BOARD, RUNPOD_COLOURS
+
+# Optional: the monitoring script's own exports, if they have been mounted in.
+RUNPOD_EXPORT_DIR = os.environ.get("CX_RUNPOD_EXPORT_DIR", "/seed/runpod")
+
+
+def seed_handover(db: Session) -> str:
+    if db.scalar(select(Handover).limit(1)):
+        return "handover: already present, left alone"
+
+    row = Handover(
+        shift_date=dt.date.fromisoformat(HANDOVER["shift_date"]),
+        shift=ShiftName(HANDOVER["shift"]),
+        handing_to=HANDOVER["handing_to"],
+        team_members=HANDOVER["team_members"],
+        significant_issues_checked=HANDOVER["significant_issues_checked"],
+        hubspot_checked=HANDOVER["hubspot_checked"],
+        total_open_tickets=HANDOVER["total_open_tickets"],
+        member_checks=HANDOVER["member_checks"],
+        other_comments=HANDOVER["other_comments"],
+        status=HandoverStatus.DRAFT,
+    )
+    for position, item in enumerate(HANDOVER["items"]):
+        row.items.append(HandoverItem(
+            position=position, title=item["title"],
+            zendesk_tickets=item.get("zendesk_tickets", ""),
+            jira_key=item.get("jira_key", ""), jira_status=item.get("jira_status", ""),
+            body=item.get("body", ""), links=item.get("links", []),
+            state=ItemState(item.get("state", "in_progress")),
+            remove_at_end_of_shift=item.get("remove_at_end_of_shift", False),
+            first_raised_on=row.shift_date,
+        ))
+    db.add(row)
+    db.commit()
+    return f"handover: seeded {row.shift.value} {row.shift_date} with {len(row.items)} items"
+
+
+def _load_export(name: str):
+    path = Path(RUNPOD_EXPORT_DIR) / name
+    if not path.is_file():
+        return None
+    try:
+        return json.loads(path.read_text())
+    except (OSError, json.JSONDecodeError):
+        return None
+
+
+def seed_runpod(db: Session) -> str:
+    if db.scalar(select(RunpodHost).limit(1)):
+        return "runpod: already present, left alone"
+
+    now = dt.datetime.now(dt.timezone.utc)
+    rng = random.Random(20260806)   # deterministic, so the demo looks the same each time
+    hosts: dict[str, RunpodHost] = {}
+
+    # 1. The board rows from the handover, with their CX state.
+    for entry in RUNPOD_BOARD:
+        host = RunpodHost(
+            machine_id=entry["machine_id"], name=entry["name"], listed=False,
+            gpu_reserved=0, gpu_total=8, gpu_type="RTX 5090" if "5090" in entry["name"] else "RTX 4090",
+            data_center="NO1" if entry["name"].startswith("no1") else "CA1",
+            colour=RunpodColour(entry.get("colour", "white")),
+            zendesk_ticket=entry.get("zendesk_ticket", ""),
+            jira_key=entry.get("jira_key", ""), jira_status=entry.get("jira_status", ""),
+            last_error=entry.get("last_error", ""), next_steps=entry.get("next_steps", ""),
+            unlisted_at=now - dt.timedelta(hours=rng.randint(6, 96)),
+        )
+        db.add(host)
+        db.flush()
+        hosts[host.machine_id] = host
+
+    # 2. Anything else the monitoring script had recorded.
+    hosts_db = _load_export("hosts_db.json") or {}
+    jira_db = _load_export("jira_db.json") or {}
+    for machine_id, record in list(hosts_db.items())[:200]:
+        if machine_id in hosts:
+            continue
+        jira = jira_db.get(machine_id, {})
+        host = RunpodHost(
+            machine_id=machine_id, name=str(record.get("name") or ""),
+            listed=bool(record.get("listed", True)),
+            gpu_reserved=int(record.get("gpuReserved") or 0), gpu_total=8,
+            historic_count=int(jira.get("historic_count") or 0),
+            jira_key=jira.get("jira_issue_key") or "", jira_status=jira.get("jira_status") or "",
+        )
+        if not host.listed:
+            host.unlisted_at = now - dt.timedelta(hours=rng.randint(2, 240))
+        db.add(host)
+        db.flush()
+        hosts[machine_id] = host
+
+    # 3. A plausible history, so the timeline and ranking have something to show.
+    for host in hosts.values():
+        rounds = max(host.historic_count, 1 if not host.listed else 0)
+        cursor = now - dt.timedelta(days=min(45, 3 + rounds * 4))
+        for _ in range(min(rounds, 12)):
+            cursor += dt.timedelta(hours=rng.randint(8, 72))
+            if cursor >= now:
+                break
+            record_event(db, host, RunpodEventType.UNLISTED, actor="runpod",
+                         detail="Unlisted automatically after a critical error",
+                         error_hint=host.last_error or "gpu health check failed",
+                         gpu_reserved=rng.choice([0, 1, 2, 8]))
+            host.events[-1].occurred_at = cursor
+            host.unlist_count += 1
+
+            cursor += dt.timedelta(hours=rng.randint(4, 48))
+            if cursor >= now or not host.listed:
+                continue
+            operator = rng.choice(["parham.monfared@nexgencloud.com", "luis.sarabando@nexgencloud.com",
+                                   "kheano.martinez@nexgencloud.com"])
+            record_event(db, host, RunpodEventType.LISTED, actor=operator,
+                         detail="Relisted after burn-in passed",
+                         zendesk_ticket=host.zendesk_ticket)
+            host.events[-1].occurred_at = cursor
+
+        if not host.listed and host.last_error:
+            record_event(db, host, RunpodEventType.UNLISTED, actor="runpod",
+                         detail="Current unlisting", error_hint=host.last_error,
+                         zendesk_ticket=host.zendesk_ticket, jira_key=host.jira_key)
+            host.events[-1].occurred_at = host.unlisted_at or now
+            host.unlist_count += 1
+
+    db.merge(AppSetting(key="runpod_colours", value=RUNPOD_COLOURS))
+    db.commit()
+    unlisted = sum(1 for h in hosts.values() if not h.listed)
+    return f"runpod: seeded {len(hosts)} machines ({unlisted} unlisted) with history"
+
+
+def run(db: Session) -> list[str]:
+    return [seed_handover(db), seed_runpod(db)]

backend/tests/test_runpod.py

@@ -0,0 +1,104 @@
+"""RunPod client modes, email parsing, ticket composition and Jira scoping."""
+import os
+import sys
+import tempfile
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
+
+os.environ.setdefault("CX_DATABASE_URL", f"sqlite:///{tempfile.mkdtemp()}/rp.db")
+
+FAILS = []
+
+
+def expect(label, cond, got=""):
+    print(("  PASS  " if cond else "  FAIL  ") + label + ("" if cond else f"   <- {got}"))
+    if not cond:
+        FAILS.append(label)
+
+
+print("\nCLIENT MODES")
+from app.runpod.client import RunPodClient, RunPodError
+
+expect("api key wins over console credentials",
+       RunPodClient(api_key="k", email="e", password="p").mode == "api_key")
+expect("falls back to console login", RunPodClient(email="e", password="p").mode == "console_login")
+expect("reports unconfigured", RunPodClient().mode == "unconfigured")
+try:
+    RunPodClient()._console_jwt()
+    expect("unconfigured client refuses to call out", False)
+except RunPodError as exc:
+    expect("unconfigured client refuses to call out", "not configured" in str(exc).lower())
+
+code = RunPodClient(totp_secret="JBSWY3DPEHPK3PXP")._totp_code()
+expect("TOTP generator returns a 6-digit code", code.isdigit() and len(code) == 6, code)
+
+print("\nEMAIL PARSING")
+from app.runpod.email_parse import classify, parse
+
+SAMPLE = """Your machine ca1-esc8-106 (x0gn8v2rthk4) hit a critical error and was automatically unlisted.
+Error detected:
+dcgm-xid-check: potential XID issue detected
+gpu health check failed: metric gpu_cuda_ok: expected 1, got 0
+error indicator present: gpu_failed{reason=memory_remap,uuid=GPU-d29f}
+Impact: 8 GPU(s) currently rented on this machine.
+Common causes and where to look:
+"""
+r = parse(SAMPLE, subject="ca1-esc8-106 Unlisted - CRITICAL ERROR")
+expect("host extracted", r["host"] == "ca1-esc8-106", r["host"])
+expect("machine id extracted", r["machine_id"] == "x0gn8v2rthk4", r["machine_id"])
+expect("rented GPUs extracted", r["gpus_rented"] == 8, r["gpus_rented"])
+expect("marked critical", r["is_critical"])
+expect("error block captured, boilerplate excluded",
+       "dcgm-xid-check" in r["error_text"] and "Common causes" not in r["error_text"])
+expect("memory remap recognised as hardware",
+       r["signature"] == "GPU memory remapping failure", r["signature"])
+
+expect("quoted-printable is decoded",
+       "=3D" not in parse("Error detected:\ngpu_failed{a=3Db}\nImpact: 1 GPU(s)")["error_text"])
+expect("subject-only still yields a host",
+       parse("no machine line here", subject="no1-os1-5090-050 Unlisted")["host"] == "no1-os1-5090-050")
+for text, want in [("nvidia-smi: too many failures", "nvidia-smi failing repeatedly"),
+                   ("pod sync failed 16 times", "Pod sync failing"),
+                   ("container stuck: docker service unresponsive", "Docker daemon unresponsive"),
+                   ("total gibberish", "Unrecognised error")]:
+    expect(f"classify: {text[:34]}", classify(text)["signature"] == want, classify(text)["signature"])
+
+print("\nTICKET COMPOSITION")
+from app.models import RunpodHost
+from app.runpod.delivery import unlisting_ticket
+
+host = RunpodHost(machine_id="x0gn8v2rthk4", name="ca1-esc8-106", gpu_reserved=8, gpu_total=8,
+                  unlist_count=7, data_center="CA1",
+                  last_error="dcgm-xid-check: potential XID issue detected\nreason=memory_remap")
+t = unlisting_ticket(host)
+expect("subject names the host and the fault",
+       "ca1-esc8-106" in t["subject"] and "memory" in t["subject"].lower(), t["subject"])
+expect("body carries the error text", "dcgm-xid-check" in t["body"])
+expect("body states the rented impact", "8 GPU(s) currently rented" in t["body"])
+expect("body carries the repeat count", "7 time(s)" in t["body"])
+
+drained = RunpodHost(machine_id="m", name="h", gpu_reserved=0, gpu_total=8, unlist_count=1)
+expect("a drained machine says so", "drained" in unlisting_ticket(drained)["body"])
+
+print("\nJIRA SCOPING")
+from app.config import Settings
+
+os.environ.update({"CX_JIRA_BASE": "https://oie.atlassian.net", "CX_JIRA_EMAIL": "oie@x",
+                   "CX_JIRA_TOKEN": "t1", "CX_JIRA_PROJECT": "OIE"})
+s = Settings()
+expect("default scope uses the OIE instance", s.jira_for("default")["base"] == "https://oie.atlassian.net")
+expect("runpod inherits when unset", s.jira_for("runpod")["base"] == "https://oie.atlassian.net")
+expect("runpod keeps its own project", s.jira_for("runpod")["project"] == "RMA")
+
+os.environ.update({"CX_RUNPOD_JIRA_BASE": "https://rma.atlassian.net",
+                   "CX_RUNPOD_JIRA_EMAIL": "rma@x", "CX_RUNPOD_JIRA_TOKEN": "t2"})
+s = Settings()
+expect("runpod uses its own instance when given one",
+       s.jira_for("runpod")["base"] == "https://rma.atlassian.net")
+expect("the two scopes stay separate",
+       s.jira_for("runpod")["token"] != s.jira_for("default")["token"])
+expect("default is untouched by the runpod values",
+       s.jira_for("default")["base"] == "https://oie.atlassian.net")
+
+print("\n" + ("ALL CHECKS PASSED" if not FAILS else f"{len(FAILS)} FAILED: {FAILS}"))
+sys.exit(1 if FAILS else 0)

backend/triagelib/cxbridge.py

@@ -55,22 +55,60 @@ def _import_cxlib(path: str):
     return cxlib
 
 
+def _tokens_from_env() -> dict[str, str]:
+    """API keys supplied directly, bypassing 1Password.
+
+    CX-Tools reads its keys from 1Password, which needs an interactive session
+    and a desktop app - neither exists in a container. `Config` is a dataclass
+    whose 1Password lookups live in per-field default factories, so passing the
+    values in means those factories never run. CX-Tools itself is unmodified.
+    """
+    return {
+        "api_key": os.environ.get("CX_INFRAHUB_TOKEN", "").strip(),
+        "insight_api_key": os.environ.get("CX_INFRAINSIGHT_TOKEN", "").strip(),
+    }
+
+
+def _silence_1password(path: str) -> None:
+    """Stop the CX-Tools secret loader from reaching for `op`.
+
+    `Config.os_cmd` re-reads the merged config on every OpenStack call, which
+    would otherwise retry a sign-in that cannot succeed here. Marking the module
+    as already loaded makes those calls no-ops.
+    """
+    if path not in sys.path:
+        sys.path.insert(0, path)
+    try:
+        import secrets_1password  # noqa: PLC0415
+    except Exception:
+        return
+    secrets_1password._loaded = True
+
+
 def bootstrap() -> tuple[Any, Any]:
     """Import cxlib and build the shared Config, loading secrets exactly once.
 
-    Call this from the foreground at startup: constructing Config triggers the
+    With CX_INFRAHUB_TOKEN / CX_INFRAINSIGHT_TOKEN set, credentials come from
-    CX-Tools 1Password loader, which may need an interactive sign-in.
+    the environment. Without them, CX-Tools falls back to 1Password, which is
+    what a laptop run does.
     """
     with _lock:
         if _state["config"] is not None:
             return _state["cx"], _state["config"]
         path = locate_cx_tools()
         cx = _import_cxlib(path)
-        config = cx.Config(no_color=True, debug=bool(os.environ.get("CX_DEBUG")))
+
+        tokens = _tokens_from_env()
+        if tokens["api_key"]:
+            _silence_1password(path)
+            config = cx.Config(no_color=True, debug=bool(os.environ.get("CX_DEBUG")),
+                               **{k: v for k, v in tokens.items() if v})
+        else:
+            config = cx.Config(no_color=True, debug=bool(os.environ.get("CX_DEBUG")))
         if not config.api_key or config.api_key in {"REDACT", "REPLACE_WITH_API_KEY"}:
             raise BridgeError(
-                "CX-Tools could not load the Infrahub API key from 1Password. "
+                "No Infrahub API key. Set CX_INFRAHUB_TOKEN (and CX_INFRAINSIGHT_TOKEN), "
-                "Run `op signin` in this shell, then restart cx-triage."
+                "or run `op signin` in this shell for the 1Password path."
             )
         _state.update({"path": path, "cx": cx, "config": config})
         return cx, config

docker-compose.yml

@@ -17,12 +17,25 @@ services:
     environment:
       CX_DATABASE_URL: ${CX_DATABASE_URL:-postgresql+psycopg://cx:cx@db:5432/cxtriage}
       CX_STATIC_DIR: /app/static
+      CX_TOOLS_PATH: /opt/cx-tools
+      CX_RUNPOD_EXPORT_DIR: /seed/runpod
     volumes:
-      # The engine shells out to `docker exec <region>-osc ...`, so it needs the
+      # The engine shells out to `docker exec <region>-osc ...`. Mounting the
-      # host's Docker socket. Mount read-only and drop it if you point the app
+      # host's socket makes those *sibling* containers - the ones CX-Tools
-      # at Prometheus/OpenStack directly instead.
+      # already relies on - reachable from inside this one. Not read-only:
-      - /var/run/docker.sock:/var/run/docker.sock:ro
+      # `docker exec` needs to create an exec instance.
+      - /var/run/docker.sock:/var/run/docker.sock
+      # CX-Tools itself, straight off the host. Nothing is written to it.
+      - ${CX_TOOLS_HOST_PATH:-../CX-Tools}:/opt/cx-tools:ro
+      # The RunPod monitor's exports, used only to seed demo data.
+      - ${CX_RUNPOD_EXPORT_HOST_PATH:-../RunPod}:/seed/runpod:ro
       - cx-data:/data
+    # The mounted socket is owned by root, and the image runs as an unprivileged
+    # user, so the container needs to be in the socket's group to use it. On
+    # Docker Desktop that group is 0; on a Linux host it is the host's `docker`
+    # group - find it with `stat -c %g /var/run/docker.sock`.
+    group_add:
+      - "${CX_DOCKER_GID:-0}"
     depends_on:
       db: { condition: service_healthy }
     restart: unless-stopped

frontend/src/App.tsx

@@ -6,6 +6,8 @@ import Login from "./pages/Login";
 import Queue from "./pages/Queue";
 import Linkage from "./pages/Linkage";
 import SettingsPage from "./pages/Settings";
+import HandoverPage from "./pages/Handover";
+import RunpodPage from "./pages/Runpod";
 
 export default function App() {
   const [user, setUser] = useState<User | null>(null);
@@ -38,7 +40,9 @@ export default function App() {
         <div className="brand">
           {config?.app_name ?? "CX Triage"} <em>&mdash; alert triage</em>
         </div>
-        <NavLink to="/" end className={({ isActive }) => `navlink${isActive ? " on" : ""}`}>Queue</NavLink>
+        <NavLink to="/" end className={({ isActive }) => `navlink${isActive ? " on" : ""}`}>Handover</NavLink>
+        <NavLink to="/alerts" className={({ isActive }) => `navlink${isActive ? " on" : ""}`}>Infrahub alerts</NavLink>
+        <NavLink to="/runpod" className={({ isActive }) => `navlink${isActive ? " on" : ""}`}>RunPod</NavLink>
         {config?.linkage_scan && (
           <NavLink to="/linkage" className={({ isActive }) => `navlink${isActive ? " on" : ""}`}>Linkage</NavLink>
         )}
@@ -51,7 +55,9 @@ export default function App() {
         <button className="sm" onClick={logout}>Sign out</button>
       </header>
       <Routes>
-        <Route path="/" element={<Queue config={config!} />} />
+        <Route path="/" element={<HandoverPage />} />
+        <Route path="/alerts" element={<Queue config={config!} />} />
+        <Route path="/runpod" element={<RunpodPage />} />
         <Route path="/linkage" element={<Linkage />} />
         <Route path="/settings" element={<SettingsPage user={user} config={config!} />} />
         <Route path="*" element={<Navigate to="/" replace />} />

frontend/src/pages/Handover.tsx

@@ -0,0 +1,241 @@
+import { useCallback, useEffect, useState } from "react";
+import { api } from "../lib/api";
+
+interface Item {
+  id: number; position: number; title: string; zendesk_tickets: string;
+  jira_key: string; jira_status: string; body: string; links: string[];
+  state: string; remove_at_end_of_shift: boolean; first_raised_on: string | null;
+  carried_from_id: number | null;
+}
+interface RunpodRow {
+  machine_id: string; name: string; colour: string; zendesk_ticket: string;
+  jira_key: string; jira_status: string; gpu_reserved: number; gpu_total: number;
+  last_error: string; next_steps: string; hours_unlisted: number | null; unlist_count: number;
+}
+interface Doc {
+  id: number; title: string; shift: string; shift_date: string; handing_to: string;
+  status: string; team_members: string; significant_issues_checked: boolean;
+  hubspot_checked: boolean; total_open_tickets: number | null;
+  member_checks: { name: string; hs_checked: boolean; jira_checked: boolean }[];
+  other_comments: string; item_count: number; items: Item[];
+}
+
+const STATES: Record<string, string> = {
+  pending_infra: "Pending Infra", pending_customer: "Pending customer",
+  pending_runpod: "Pending RunPod", pending_rma: "Pending RMA", pending_cx: "Pending CX",
+  in_progress: "In progress", monitoring: "Monitoring",
+  no_further_engagement: "No further engagement", done: "Done",
+};
+
+const COLOURS: Record<string, { dot: string; text: string }> = {
+  red: { dot: "#f2545b", text: "Blocked from relisting — recurring issue" },
+  purple: { dot: "#a97bf0", text: "Pending RunPod" },
+  yellow: { dot: "#e0a336", text: "Pending Infrastructure / DC team" },
+  blue: { dot: "#59a0f5", text: "Pending removal from the RunPod platform" },
+  green: { dot: "#35c46a", text: "Stress testing >24h — GPUs may look reserved" },
+  white: { dot: "#8a93a5", text: "Actionable by CX" },
+};
+
+const blankItem = (): Partial<Item> => ({
+  title: "", zendesk_tickets: "", jira_key: "", jira_status: "", body: "",
+  links: [], state: "in_progress", remove_at_end_of_shift: false,
+});
+
+export default function HandoverPage() {
+  const [doc, setDoc] = useState<Doc | null>(null);
+  const [runpod, setRunpod] = useState<RunpodRow[]>([]);
+  const [editing, setEditing] = useState<Partial<Item> | null>(null);
+  const [saved, setSaved] = useState("");
+  const [busy, setBusy] = useState(false);
+
+  const load = useCallback(async () => {
+    const r = await api.get<{ handover: Doc | null; runpod: RunpodRow[] }>("/api/handover/current");
+    setDoc(r.handover); setRunpod(r.runpod ?? []);
+  }, []);
+  useEffect(() => { void load(); }, [load]);
+
+  const flash = (m: string) => { setSaved(m); setTimeout(() => setSaved(""), 2200); };
+
+  const saveDoc = async (patch: Partial<Doc>) => {
+    if (!doc) return;
+    const next = await api.post<Doc>(`/api/handover/${doc.id}`, { ...doc, ...patch });
+    setDoc(next); flash("Saved");
+  };
+
+  const saveItem = async () => {
+    if (!doc || !editing) return;
+    setBusy(true);
+    try {
+      const path = editing.id
+        ? `/api/handover/${doc.id}/items/${editing.id}`
+        : `/api/handover/${doc.id}/items`;
+      setDoc(await api.post<Doc>(path, editing));
+      setEditing(null); flash("Item saved");
+    } finally { setBusy(false); }
+  };
+
+  const handOver = async () => {
+    if (!doc) return;
+    if (!confirm("Close this shift and start the next one? Live items carry forward; " +
+                 "anything done or flagged 'remove at end of shift' is left behind.")) return;
+    const r = await api.post<{ next: Doc; carried: number }>(`/api/handover/${doc.id}/hand-over`);
+    setDoc(r.next); flash(`Handed over — ${r.carried} items carried forward`);
+  };
+
+  if (!doc) {
+    return (
+      <main style={{ padding: 26 }}>
+        <div className="card">
+          <h3 className="sec">No handover yet</h3>
+          <button className="pri" onClick={async () => {
+            await api.post("/api/handover", { shift: "APAC", handing_to: "EMEA" });
+            void load();
+          }}>Start today's handover</button>
+        </div>
+      </main>
+    );
+  }
+
+  return (
+    <main style={{ padding: "22px 26px 80px", maxWidth: 1180 }}>
+      <div className="row">
+        <h2 style={{ margin: 0, fontSize: 20 }}>{doc.title}</h2>
+        <span className="badge">{doc.shift} → {doc.handing_to}</span>
+        <span className={`badge ${doc.status === "draft" ? "" : "ok"}`}>{doc.status.replace("_", " ")}</span>
+        <span className="spacer" style={{ marginLeft: "auto" }} />
+        {saved && <span className="t-ok" style={{ fontSize: 12.5 }}>{saved}</span>}
+        <button className="pri" onClick={handOver} disabled={doc.status !== "draft"}>Hand over shift</button>
+      </div>
+
+      <div className="card mt">
+        <h3 className="sec">Shift</h3>
+        <div style={{ display: "grid", gridTemplateColumns: "190px 1fr", gap: "10px 12px", alignItems: "center" }}>
+          <label className="hint" style={{ margin: 0 }}>CX team members</label>
+          <input defaultValue={doc.team_members} onBlur={(e) => saveDoc({ team_members: e.target.value })} />
+          <label className="hint" style={{ margin: 0 }}>Total open tickets</label>
+          <input type="number" defaultValue={doc.total_open_tickets ?? undefined} style={{ maxWidth: 140 }}
+                 onBlur={(e) => saveDoc({ total_open_tickets: Number(e.target.value) })} />
+          <label className="hint" style={{ margin: 0 }}>Checks</label>
+          <div className="row">
+            <label className="row" style={{ gap: 6 }}>
+              <input type="checkbox" style={{ width: "auto" }} checked={doc.significant_issues_checked}
+                     onChange={(e) => saveDoc({ significant_issues_checked: e.target.checked })} />
+              Active issues of significance
+            </label>
+            <label className="row" style={{ gap: 6 }}>
+              <input type="checkbox" style={{ width: "auto" }} checked={doc.hubspot_checked}
+                     onChange={(e) => saveDoc({ hubspot_checked: e.target.checked })} />
+              HubSpot “My tickets on hold”
+            </label>
+          </div>
+        </div>
+      </div>
+
+      <div className="row mt">
+        <h3 className="sec" style={{ flex: 1, margin: 0 }}>Key updates ({doc.items.length})</h3>
+        <button onClick={() => setEditing(blankItem())}>Add an update</button>
+      </div>
+
+      {doc.items.map((it) => (
+        <div className="card" key={it.id} style={{ marginBottom: 10 }}>
+          <div className="row">
+            <b style={{ flex: 1 }}>{it.title || "(untitled)"}</b>
+            {it.carried_from_id && <span className="badge" title={`First raised ${it.first_raised_on}`}>carried over</span>}
+            {it.remove_at_end_of_shift && <span className="badge">remove at end of shift</span>}
+            <span className={`badge ${it.state === "done" ? "ok" : it.state.startsWith("pending") ? "real" : ""}`}>
+              {STATES[it.state] ?? it.state}</span>
+            <button className="sm" onClick={() => setEditing(it)}>Edit</button>
+            <button className="sm del" onClick={async () => {
+              if (!confirm("Remove this update?")) return;
+              setDoc(await api.del<Doc>(`/api/handover/${doc.id}/items/${it.id}`));
+            }}>Remove</button>
+          </div>
+          <div className="row" style={{ gap: 14, marginTop: 2 }}>
+            {it.zendesk_tickets && <span className="hint" style={{ margin: 0 }}>Zendesk {it.zendesk_tickets}</span>}
+            {it.jira_key && <span className="hint" style={{ margin: 0 }}>{it.jira_key} {it.jira_status}</span>}
+          </div>
+          {it.body && <div style={{ whiteSpace: "pre-wrap", marginTop: 8, fontSize: 13.5 }}>{it.body}</div>}
+          {(it.links ?? []).map((l) => (
+            <div key={l}><a href={l} target="_blank" rel="noreferrer" style={{ fontSize: 12 }}>{l}</a></div>
+          ))}
+        </div>
+      ))}
+
+      {editing && (
+        <div className="card" style={{ borderColor: "var(--accent)" }}>
+          <div className="fld"><label>Title</label>
+            <input value={editing.title ?? ""} onChange={(e) => setEditing({ ...editing, title: e.target.value })} /></div>
+          <div className="row">
+            <div className="fld" style={{ flex: 1 }}><label>Zendesk ticket(s)</label>
+              <input value={editing.zendesk_tickets ?? ""} placeholder="#8495, #8453"
+                     onChange={(e) => setEditing({ ...editing, zendesk_tickets: e.target.value })} /></div>
+            <div className="fld" style={{ flex: 1 }}><label>Jira key</label>
+              <input value={editing.jira_key ?? ""} placeholder="OIE-3213"
+                     onChange={(e) => setEditing({ ...editing, jira_key: e.target.value })} /></div>
+            <div className="fld" style={{ flex: 1 }}><label>Jira status</label>
+              <input value={editing.jira_status ?? ""} placeholder="In Progress"
+                     onChange={(e) => setEditing({ ...editing, jira_status: e.target.value })} /></div>
+          </div>
+          <div className="fld"><label>Detail</label>
+            <textarea value={editing.body ?? ""} onChange={(e) => setEditing({ ...editing, body: e.target.value })} /></div>
+          <div className="row">
+            <div className="fld" style={{ flex: 1 }}><label>Status</label>
+              <select value={editing.state ?? "in_progress"}
+                      onChange={(e) => setEditing({ ...editing, state: e.target.value })}>
+                {Object.entries(STATES).map(([v, l]) => <option key={v} value={v}>{l}</option>)}
+              </select></div>
+            <label className="row" style={{ gap: 6, marginTop: 18 }}>
+              <input type="checkbox" style={{ width: "auto" }} checked={!!editing.remove_at_end_of_shift}
+                     onChange={(e) => setEditing({ ...editing, remove_at_end_of_shift: e.target.checked })} />
+              Remove at end of shift
+            </label>
+          </div>
+          <div className="row">
+            <button className="pri" onClick={saveItem} disabled={busy}>Save</button>
+            <button onClick={() => setEditing(null)}>Cancel</button>
+          </div>
+        </div>
+      )}
+
+      <h3 className="sec" style={{ marginTop: 28 }}>RunPod — unlisted machines ({runpod.length})</h3>
+      <div className="hint" style={{ marginTop: 0 }}>
+        Live from RunPod, not retyped. Colours follow the handover key.
+      </div>
+      <div className="row" style={{ gap: 14, margin: "8px 0 12px" }}>
+        {Object.entries(COLOURS).map(([k, v]) => (
+          <span key={k} className="hint" style={{ margin: 0 }} title={v.text}>
+            <i style={{ display: "inline-block", width: 9, height: 9, borderRadius: 2,
+                        background: v.dot, marginRight: 5 }} />{k}
+          </span>
+        ))}
+      </div>
+      <div className="card" style={{ padding: 0 }}>
+        {runpod.length === 0 ? <div className="empty">Nothing unlisted.</div> : runpod.map((r, i) => (
+          <div key={r.machine_id} style={{ padding: "10px 14px", borderTop: i ? "1px solid var(--line)" : undefined }}>
+            <div className="row">
+              <i style={{ width: 9, height: 9, borderRadius: 2, background: COLOURS[r.colour]?.dot ?? "#888" }}
+                 title={COLOURS[r.colour]?.text} />
+              <b style={{ flex: 1 }}>{r.name}</b>
+              <span className="hint" style={{ margin: 0, fontFamily: "var(--mono)" }}>{r.machine_id}</span>
+              {r.zendesk_ticket && <span className="badge">ZD {r.zendesk_ticket}</span>}
+              {r.jira_key && <span className="badge">{r.jira_key} {r.jira_status}</span>}
+              <span className="badge">{r.gpu_reserved}/{r.gpu_total} GPU</span>
+              {r.hours_unlisted != null && <span className="badge">{r.hours_unlisted}h</span>}
+              {r.unlist_count > 3 && <span className="badge overdue">{r.unlist_count}× unlisted</span>}
+            </div>
+            {r.last_error && (
+              <pre style={{ marginTop: 6, fontSize: 11.5, background: "var(--bg)" }}>{r.last_error}</pre>
+            )}
+            {r.next_steps && <div className="hint" style={{ marginTop: 4 }}>{r.next_steps}</div>}
+          </div>
+        ))}
+      </div>
+
+      <h3 className="sec" style={{ marginTop: 28 }}>Other comments</h3>
+      <div className="card">
+        <textarea defaultValue={doc.other_comments} style={{ minHeight: 160 }}
+                  onBlur={(e) => saveDoc({ other_comments: e.target.value })} />
+      </div>
+    </main>
+  );
+}

frontend/src/pages/Runpod.tsx

@@ -0,0 +1,238 @@
+import { useCallback, useEffect, useState } from "react";
+import { api } from "../lib/api";
+
+interface Event {
+  id: number; event_type: string; actor: string; detail: string;
+  error_hint: string; zendesk_ticket: string; jira_key: string;
+  gpu_reserved: number | null; occurred_at: string;
+}
+interface Host {
+  machine_id: string; name: string; listed: boolean; gpu_reserved: number; gpu_total: number;
+  gpu_type: string; data_center: string; colour: string; zendesk_ticket: string;
+  jira_key: string; jira_status: string; last_error: string; next_steps: string;
+  unlist_count: number; historic_count: number; hours_unlisted: number | null;
+  event_count: number; effective_count?: number; last_unlisted?: string | null;
+  events?: Event[];
+}
+
+const DOT: Record<string, string> = {
+  red: "#f2545b", purple: "#a97bf0", yellow: "#e0a336",
+  blue: "#59a0f5", green: "#35c46a", white: "#8a93a5",
+};
+
+const EVENT_STYLE: Record<string, string> = {
+  unlisted: "t-bad", listed: "t-ok", drained: "t-warn",
+  maintenance_scheduled: "t-warn", zendesk_ticket: "", jira_linked: "", note: "",
+};
+
+export default function RunpodPage() {
+  const [tab, setTab] = useState<"current" | "history">("current");
+  const [status, setStatus] = useState<any>(null);
+  const [hosts, setHosts] = useState<Host[]>([]);
+  const [problem, setProblem] = useState<Host[]>([]);
+  const [open, setOpen] = useState<Host | null>(null);
+  const [busy, setBusy] = useState(false);
+  const [ticket, setTicket] = useState<any>(null);
+  const [sending, setSending] = useState("");
+
+  const load = useCallback(async () => {
+    const [s, h, p] = await Promise.all([
+      api.get<any>("/api/runpod/status"),
+      api.get<{ hosts: Host[] }>("/api/runpod/hosts?unlisted_only=true"),
+      api.get<{ hosts: Host[] }>("/api/runpod/problem-hosts?limit=50"),
+    ]);
+    setStatus(s); setHosts(h.hosts); setProblem(p.hosts);
+  }, []);
+  useEffect(() => { void load(); }, [load]);
+
+  const openHost = async (machineId: string) => {
+    setTicket(null); setSending("");
+    const [host, preview] = await Promise.all([
+      api.get<Host>(`/api/runpod/hosts/${machineId}`),
+      api.get<any>(`/api/runpod/hosts/${machineId}/ticket-preview`).catch(() => null),
+    ]);
+    setOpen(host); setTicket(preview);
+  };
+
+  const act = async (path: string, label: string) => {
+    if (!open) return;
+    setSending(label);
+    try {
+      const r = await api.post<any>(`/api/runpod/hosts/${open.machine_id}/${path}`, {});
+      alert(`${label}: ${r.action} — ${r.ticket_id ?? r.key}`);
+      await openHost(open.machine_id); await load();
+    } catch (e) {
+      alert(e instanceof Error ? e.message : `${label} failed`);
+    } finally { setSending(""); }
+  };
+
+  if (!status) return <div className="empty"><span className="spin" /></div>;
+
+  return (
+    <main style={{ padding: "22px 26px 80px", maxWidth: 1400 }}>
+      <div className="row">
+        <h2 style={{ margin: 0, fontSize: 20 }}>RunPod</h2>
+        <span className="badge">{status.totals.machines} machines</span>
+        <span className="badge overdue">{status.totals.unlisted} unlisted</span>
+        <span className="badge">{status.totals.gpus_rented}/{status.totals.gpus_total} GPUs rented</span>
+        <span className="badge" title="api_key is the supported mode; console_login cannot run unattended">
+          auth: {status.mode}</span>
+        <span className="spacer" style={{ marginLeft: "auto" }} />
+        <button disabled={!status.configured || busy} onClick={async () => {
+          setBusy(true);
+          try { const r = await api.post<any>("/api/runpod/sync"); await load();
+                alert(`Synced ${r.machines} machines — ${r.unlisted} newly unlisted, ${r.relisted} relisted, ${r.drained} drained`); }
+          catch (e) { alert(e instanceof Error ? e.message : "Sync failed"); }
+          finally { setBusy(false); }
+        }}>{busy ? "Syncing…" : "Sync from RunPod"}</button>
+      </div>
+
+      {!status.configured && (
+        <div className="warnbox mt">
+          RunPod is not configured. Set <code>CX_RUNPOD_API_KEY</code>. The email/password fallback
+          cannot run unattended — the account has 2FA, so no session is ever created.
+        </div>
+      )}
+
+      <div className="row mt">
+        <span className={`chip${tab === "current" ? " on" : ""}`} onClick={() => setTab("current")}>
+          Unlisted now <b>{hosts.length}</b></span>
+        <span className={`chip${tab === "history" ? " on" : ""}`} onClick={() => setTab("history")}>
+          Problem hosts <b>{problem.length}</b></span>
+      </div>
+
+      {tab === "current" && (
+        <div className="card mt" style={{ padding: 0 }}>
+          {hosts.length === 0 ? <div className="empty">Nothing unlisted.</div> : hosts.map((h, i) => (
+            <div key={h.machine_id} style={{ padding: "10px 14px", borderTop: i ? "1px solid var(--line)" : undefined, cursor: "pointer" }}
+                 onClick={() => void openHost(h.machine_id)}>
+              <div className="row">
+                <i style={{ width: 9, height: 9, borderRadius: 2, background: DOT[h.colour] ?? "#888" }} />
+                <b style={{ flex: 1 }}>{h.name}</b>
+                <span className="hint" style={{ margin: 0, fontFamily: "var(--mono)" }}>{h.machine_id}</span>
+                {h.zendesk_ticket && <span className="badge">ZD {h.zendesk_ticket}</span>}
+                {h.jira_key && <span className="badge">{h.jira_key}</span>}
+                <span className="badge">{h.gpu_reserved}/{h.gpu_total} GPU</span>
+                {h.hours_unlisted != null && <span className="badge">{h.hours_unlisted}h</span>}
+              </div>
+              {h.last_error && <div className="hint" style={{ marginTop: 4, whiteSpace: "pre-wrap" }}>
+                {h.last_error.split("\n")[0]}</div>}
+            </div>
+          ))}
+        </div>
+      )}
+
+      {tab === "history" && (
+        <>
+          <div className="hint">
+            Ranked by how often each machine has been unlisted. Repeat offenders are the RMA
+            conversation, not another burn-in.
+          </div>
+          <div className="card mt" style={{ padding: 0 }}>
+            {problem.map((h, i) => (
+              <div key={h.machine_id} className="row"
+                   style={{ padding: "9px 14px", borderTop: i ? "1px solid var(--line)" : undefined, cursor: "pointer" }}
+                   onClick={() => void openHost(h.machine_id)}>
+                <span style={{ width: 26, color: "var(--faint)", fontFamily: "var(--mono)" }}>{i + 1}</span>
+                <i style={{ width: 9, height: 9, borderRadius: 2, background: DOT[h.colour] ?? "#888" }} />
+                <b style={{ flex: 1 }}>{h.name}</b>
+                <span className={h.listed ? "t-ok" : "t-bad"} style={{ fontSize: 12 }}>
+                  {h.listed ? "listed" : "unlisted"}</span>
+                <span className="badge overdue">{h.effective_count ?? h.unlist_count}× unlisted</span>
+                <span className="badge">{h.event_count} events</span>
+                {h.last_unlisted && <span className="hint" style={{ margin: 0 }}>
+                  last {h.last_unlisted.slice(0, 10)}</span>}
+              </div>
+            ))}
+          </div>
+        </>
+      )}
+
+      {open && (
+        <>
+          <div className="scrim" onClick={() => setOpen(null)} />
+          <aside className="drawer">
+            <div className="dh">
+              <h2>{open.name}</h2>
+              <button className="sm" onClick={() => setOpen(null)}>Close</button>
+            </div>
+            <div className="db">
+              <div className="row">
+                <span className="badge" style={{ fontFamily: "var(--mono)" }}>{open.machine_id}</span>
+                <span className={open.listed ? "badge ok" : "badge overdue"}>
+                  {open.listed ? "listed" : "unlisted"}</span>
+                <span className="badge">{open.gpu_reserved}/{open.gpu_total} GPU</span>
+                {open.data_center && <span className="badge">{open.data_center}</span>}
+                <span className="badge">{open.unlist_count}× unlisted</span>
+              </div>
+
+              {open.last_error && (
+                <>
+                  <h3 className="sec" style={{ marginTop: 16 }}>Last error</h3>
+                  <pre>{open.last_error}</pre>
+                </>
+              )}
+              {open.next_steps && (
+                <>
+                  <h3 className="sec" style={{ marginTop: 16 }}>Next steps</h3>
+                  <div style={{ fontSize: 13 }}>{open.next_steps}</div>
+                </>
+              )}
+
+              {ticket && (
+                <>
+                  <h3 className="sec" style={{ marginTop: 18 }}>Actions</h3>
+                  {(!ticket.send_enabled || !ticket.zendesk_ready) && (
+                    <div className="warnbox">
+                      <b>Preview only.</b>{" "}
+                      {!ticket.send_enabled
+                        ? "Sending is off on this instance (CX_FEATURE_SEND_ENABLED)."
+                        : "Zendesk is not configured."}{" "}Nothing will be sent.
+                    </div>
+                  )}
+                  <div className="row">
+                    <button className="pri"
+                            disabled={!ticket.send_enabled || !ticket.zendesk_ready || !!sending}
+                            onClick={() => act("zendesk", "Zendesk ticket")}>
+                      {sending === "Zendesk ticket" ? "Raising…" : "Raise Zendesk ticket"}
+                    </button>
+                    <button disabled={!ticket.send_enabled || !ticket.runpod_jira_ready || !!sending}
+                            onClick={() => act("rma", "RMA issue")}>
+                      {sending === "RMA issue" ? "Raising…" : "Raise RMA issue"}
+                    </button>
+                    {ticket.existing_ticket && <span className="badge">ZD {ticket.existing_ticket}</span>}
+                    {ticket.existing_jira && <span className="badge">{ticket.existing_jira}</span>}
+                  </div>
+                  <details className="fold" style={{ marginTop: 10 }}>
+                    <summary>Ticket that would be raised</summary>
+                    <div className="foldb">
+                      <div className="hint" style={{ marginTop: 0 }}>{ticket.subject}</div>
+                      <pre style={{ marginTop: 6 }}>{ticket.body}</pre>
+                    </div>
+                  </details>
+                </>
+              )}
+
+              <h3 className="sec" style={{ marginTop: 18 }}>History ({open.events?.length ?? 0})</h3>
+              <ul className="timeline">
+                {(open.events ?? []).map((e) => (
+                  <li key={e.id}>
+                    <b className={EVENT_STYLE[e.event_type] ?? ""}>{e.event_type.replace(/_/g, " ")}</b>
+                    {" — "}{e.detail}
+                    {e.zendesk_ticket && <span className="badge" style={{ marginLeft: 6 }}>ZD {e.zendesk_ticket}</span>}
+                    {e.jira_key && <span className="badge" style={{ marginLeft: 6 }}>{e.jira_key}</span>}
+                    {e.error_hint && <pre style={{ marginTop: 5 }}>{e.error_hint}</pre>}
+                    <div className="ts">
+                      {new Date(e.occurred_at).toLocaleString()} · {e.actor}
+                      {e.gpu_reserved != null && ` · ${e.gpu_reserved} GPU rented`}
+                    </div>
+                  </li>
+                ))}
+              </ul>
+            </div>
+          </aside>
+        </>
+      )}
+    </main>
+  );
+}