Add shift handover and RunPod, and make CX-Tools work in a container

Handover
- The Confluence shift doc becomes the landing page: shift metadata, the
  top-of-page checks, key updates with their Zendesk/Jira refs and status, and
  the free-text comments. "Hand over shift" closes the shift, opens the next
  one and carries the live items across, dropping anything done or marked
  "remove at end of shift" - the retyping this replaces.
- The RunPod table on that page is read from live host state instead of being
  copied in by hand, with the six-colour key preserved.

RunPod
- GraphQL client keyed on CX_RUNPOD_API_KEY. The old console login is kept as a
  fallback but cannot run unattended: the account has 2FA, so Clerk verifies the
  password and then asks for an emailed code and never issues a session. That is
  the real cause of the "No active session found" failure, and the client now
  says so instead of failing opaquely. TOTP is supported if the account moves to
  an authenticator app.
- Hosts and their listing history are persisted, so "most problematic hosts" can
  be ranked and each machine has a timeline of who listed or unlisted it, with
  the Zendesk comment and the error hint.
- The unlisting emails are parsed for the error block (they arrive
  quoted-printable) and classified into a likely cause and a next step.

Zendesk and Jira
- Unlisting raises a Zendesk ticket that follows the format of RunPod's own
  email, keyed on the machine so one machine keeps one thread, posted as an
  internal note.
- Jira is split in two: the Infrahub/OIE instance and the RunPod/RMA one, which
  may be a different Atlassian site. Blank RunPod values fall back to the
  defaults rather than failing.

Running in a container
- CX-Tools reads its keys from 1Password, which needs a desktop app. Config is a
  dataclass whose lookups live in per-field default factories, so passing
  CX_INFRAHUB_TOKEN/CX_INFRAINSIGHT_TOKEN in means those factories never run and
  CX-Tools itself stays unmodified.
- CX-Tools reaches OpenStack with `docker exec <region>-osc`, so the image now
  carries the Docker client (the static binary, not the docker.io package) and
  compose mounts the host socket with group_add for it. Verified from inside the
  container: live OpenStack and Infrahub calls both succeed.

Also fixes Settings, which read the environment at class-definition time and so
ignored anything set afterwards; a fresh Settings() silently returned stale
values. Caught by the Jira scoping tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

backend/app/config.py

@@ -21,56 +21,114 @@ def _int(name: str, default: int) -> int:
 
 
 class Settings:
-    # --- app ---------------------------------------------------------------
-    app_name = os.environ.get("CX_APP_NAME", "CX Triage")
-    base_url = os.environ.get("CX_BASE_URL", "http://localhost:8080")
-    secret_key = os.environ.get("CX_SECRET_KEY", "dev-only-change-me")
-    session_hours = _int("CX_SESSION_HOURS", 12)
-    static_dir = os.environ.get("CX_STATIC_DIR", "/app/static")
+    """Read once per instance, at construction time.
 
-    # --- database ----------------------------------------------------------
-    # sqlite for local/compose, postgres in the cluster.
-    database_url = os.environ.get("CX_DATABASE_URL", "sqlite:////data/cx-triage.db")
+    These were class attributes, which meant the environment was read when the
+    module was first imported and never again - so a fresh Settings() silently
+    returned stale values. Reading in __init__ makes construction mean what it
+    looks like it means.
+    """
 
-    # --- data sources ------------------------------------------------------
-    prometheus_base = os.environ.get("CX_PROMETHEUS_BASE", "http://10.11.254.250:9090")
-    prometheus_relay = os.environ.get("CX_PROMETHEUS_RELAY", "")
-    cx_tools_path = os.environ.get("CX_TOOLS_PATH", "")
+    def __init__(self) -> None:
+        # --- app ---------------------------------------------------------------
+        self.app_name = os.environ.get("CX_APP_NAME", "CX Triage")
+        self.base_url = os.environ.get("CX_BASE_URL", "http://localhost:8080")
+        self.secret_key = os.environ.get("CX_SECRET_KEY", "dev-only-change-me")
+        self.session_hours = _int("CX_SESSION_HOURS", 12)
+        self.static_dir = os.environ.get("CX_STATIC_DIR", "/app/static")
 
-    # --- auth --------------------------------------------------------------
-    # Local accounts are for development and for a cluster without SSO yet.
-    # When CX_OIDC_ENABLED is on, Authentik becomes the source of truth.
-    auth_local_enabled = _bool("CX_AUTH_LOCAL_ENABLED", True)
-    bootstrap_admin_email = os.environ.get("CX_BOOTSTRAP_ADMIN_EMAIL", "admin@localhost")
-    bootstrap_admin_password = os.environ.get("CX_BOOTSTRAP_ADMIN_PASSWORD", "")
+        # --- database ----------------------------------------------------------
+        # sqlite for local/compose, postgres in the cluster.
+        self.database_url = os.environ.get("CX_DATABASE_URL", "sqlite:////data/cx-triage.db")
 
-    oidc_enabled = _bool("CX_OIDC_ENABLED", False)
-    oidc_issuer = os.environ.get("CX_OIDC_ISSUER", "")            # e.g. https://sso/application/o/cx-triage/
-    oidc_client_id = os.environ.get("CX_OIDC_CLIENT_ID", "")
-    oidc_client_secret = os.environ.get("CX_OIDC_CLIENT_SECRET", "")
-    oidc_scopes = os.environ.get("CX_OIDC_SCOPES", "openid email profile")
-    oidc_admin_group = os.environ.get("CX_OIDC_ADMIN_GROUP", "cx-triage-admins")
-    oidc_groups_claim = os.environ.get("CX_OIDC_GROUPS_CLAIM", "groups")
+        # --- data sources ------------------------------------------------------
+        self.prometheus_base = os.environ.get("CX_PROMETHEUS_BASE", "http://10.11.254.250:9090")
+        self.prometheus_relay = os.environ.get("CX_PROMETHEUS_RELAY", "")
+        self.cx_tools_path = os.environ.get("CX_TOOLS_PATH", "")
 
-    # --- feature flags -----------------------------------------------------
-    # Sending must be switched on deliberately; a demo instance cannot email.
-    feature_send_enabled = _bool("CX_FEATURE_SEND_ENABLED", False)
-    feature_zendesk = _bool("CX_FEATURE_ZENDESK", False)
-    feature_jira = _bool("CX_FEATURE_JIRA", False)
-    feature_linkage_scan = _bool("CX_FEATURE_LINKAGE_SCAN", True)
-    send_daily_cap = _int("CX_SEND_DAILY_CAP", 25)
+        # --- auth --------------------------------------------------------------
+        # Local accounts are for development and for a cluster without SSO yet.
+        # When CX_OIDC_ENABLED is on, Authentik becomes the source of truth.
+        self.auth_local_enabled = _bool("CX_AUTH_LOCAL_ENABLED", True)
+        self.bootstrap_admin_email = os.environ.get("CX_BOOTSTRAP_ADMIN_EMAIL", "admin@localhost")
+        self.bootstrap_admin_password = os.environ.get("CX_BOOTSTRAP_ADMIN_PASSWORD", "")
 
-    # --- integrations ------------------------------------------------------
-    zendesk_subdomain = os.environ.get("CX_ZENDESK_SUBDOMAIN", "")
-    zendesk_email = os.environ.get("CX_ZENDESK_EMAIL", "")
-    zendesk_token = os.environ.get("CX_ZENDESK_TOKEN", "")
-    zendesk_default_public = _bool("CX_ZENDESK_PUBLIC_REPLY", True)
+        self.oidc_enabled = _bool("CX_OIDC_ENABLED", False)
+        self.oidc_issuer = os.environ.get("CX_OIDC_ISSUER", "")            # e.g. https://sso/application/o/cx-triage/
+        self.oidc_client_id = os.environ.get("CX_OIDC_CLIENT_ID", "")
+        self.oidc_client_secret = os.environ.get("CX_OIDC_CLIENT_SECRET", "")
+        self.oidc_scopes = os.environ.get("CX_OIDC_SCOPES", "openid email profile")
+        self.oidc_admin_group = os.environ.get("CX_OIDC_ADMIN_GROUP", "cx-triage-admins")
+        self.oidc_groups_claim = os.environ.get("CX_OIDC_GROUPS_CLAIM", "groups")
 
-    jira_base = os.environ.get("CX_JIRA_BASE", "")
-    jira_email = os.environ.get("CX_JIRA_EMAIL", "")
-    jira_token = os.environ.get("CX_JIRA_TOKEN", "")
-    jira_project = os.environ.get("CX_JIRA_PROJECT", "INFRA")
-    jira_issue_type = os.environ.get("CX_JIRA_ISSUE_TYPE", "Task")
+        # --- feature flags -----------------------------------------------------
+        # Sending must be switched on deliberately; a demo instance cannot email.
+        self.feature_send_enabled = _bool("CX_FEATURE_SEND_ENABLED", False)
+        self.feature_zendesk = _bool("CX_FEATURE_ZENDESK", False)
+        self.feature_jira = _bool("CX_FEATURE_JIRA", False)
+        self.feature_linkage_scan = _bool("CX_FEATURE_LINKAGE_SCAN", True)
+        self.send_daily_cap = _int("CX_SEND_DAILY_CAP", 25)
+
+        # --- integrations ------------------------------------------------------
+        self.zendesk_subdomain = os.environ.get("CX_ZENDESK_SUBDOMAIN", "")
+        self.zendesk_email = os.environ.get("CX_ZENDESK_EMAIL", "")
+        self.zendesk_token = os.environ.get("CX_ZENDESK_TOKEN", "")
+        self.zendesk_default_public = _bool("CX_ZENDESK_PUBLIC_REPLY", True)
+
+        # Jira, instance 1: the Infrahub/OpenStack side (OIE).
+        self.jira_base = os.environ.get("CX_JIRA_BASE", "")
+        self.jira_email = os.environ.get("CX_JIRA_EMAIL", "")
+        self.jira_token = os.environ.get("CX_JIRA_TOKEN", "")
+        self.jira_project = os.environ.get("CX_JIRA_PROJECT", "OIE")
+        self.jira_issue_type = os.environ.get("CX_JIRA_ISSUE_TYPE", "Task")
+
+        # Jira, instance 2: RunPod hosts and RMAs. These may be a different
+        # Atlassian site entirely, so they get their own credentials; anything left
+        # blank falls back to the values above rather than failing.
+        self.runpod_jira_base = os.environ.get("CX_RUNPOD_JIRA_BASE", "")
+        self.runpod_jira_email = os.environ.get("CX_RUNPOD_JIRA_EMAIL", "")
+        self.runpod_jira_token = os.environ.get("CX_RUNPOD_JIRA_TOKEN", "")
+        self.runpod_jira_project = os.environ.get("CX_RUNPOD_JIRA_PROJECT", "RMA")
+        self.runpod_jira_issue_type = os.environ.get("CX_RUNPOD_JIRA_ISSUE_TYPE", "Task")
+
+        # --- RunPod ------------------------------------------------------------
+        # The API key is the supported path. Email/password only reaches a console
+        # login that now requires a second factor, so it cannot run unattended -
+        # see app/runpod/client.py.
+        self.runpod_api_key = os.environ.get("CX_RUNPOD_API_KEY", "")
+        self.runpod_email = os.environ.get("CX_RUNPOD_EMAIL", "")
+        self.runpod_password = os.environ.get("CX_RUNPOD_PASSWORD", "")
+        self.runpod_team_id = os.environ.get("CX_RUNPOD_TEAM_ID", "")
+        self.runpod_totp_secret = os.environ.get("CX_RUNPOD_TOTP_SECRET", "")
+        self.feature_runpod = _bool("CX_FEATURE_RUNPOD", True)
+        self.feature_handover = _bool("CX_FEATURE_HANDOVER", True)
+        # Unlisting a machine is destructive to earnings; gated separately.
+        self.feature_runpod_write = _bool("CX_FEATURE_RUNPOD_WRITE", False)
+
+    def jira_for(self, scope: str) -> dict:
+        """Credentials for a Jira scope: 'runpod' or anything else (default)."""
+        if scope == "runpod":
+            return {
+                "base": self.runpod_jira_base or self.jira_base,
+                "email": self.runpod_jira_email or self.jira_email,
+                "token": self.runpod_jira_token or self.jira_token,
+                "project": self.runpod_jira_project,
+                "issue_type": self.runpod_jira_issue_type,
+            }
+        return {
+            "base": self.jira_base, "email": self.jira_email, "token": self.jira_token,
+            "project": self.jira_project, "issue_type": self.jira_issue_type,
+        }
+
+    @property
+    def runpod_ready(self) -> bool:
+        return bool(self.feature_runpod and (self.runpod_api_key or
+                                             (self.runpod_email and self.runpod_password)))
+
+    @property
+    def runpod_jira_ready(self) -> bool:
+        creds = self.jira_for("runpod")
+        return bool(self.feature_jira and creds["base"] and creds["email"] and creds["token"])
 
     @property
     def zendesk_ready(self) -> bool:
@@ -93,6 +151,11 @@ class Settings:
             "send_enabled": self.feature_send_enabled,
             "linkage_scan": self.feature_linkage_scan,
             "jira_project": self.jira_project if self.jira_ready else "",
+            "runpod_ready": self.runpod_ready,
+            "runpod_write": self.feature_runpod_write,
+            "runpod_jira_ready": self.runpod_jira_ready,
+            "runpod_jira_project": self.runpod_jira_project if self.runpod_jira_ready else "",
+            "handover": self.feature_handover,
         }
 
 

backend/app/delivery.py

@@ -29,7 +29,7 @@ class DeliveryError(RuntimeError):
     pass
 
 
-def _guard(kind: str) -> None:
+def _guard(kind: str, scope: str = "default") -> None:
     if not settings.feature_send_enabled:
         raise DeliveryError(
             "Sending is disabled on this instance (CX_FEATURE_SEND_ENABLED is off). "
@@ -38,9 +38,15 @@ def _guard(kind: str) -> None:
     if kind == "zendesk" and not settings.zendesk_ready:
         raise DeliveryError("Zendesk is not configured. Set CX_FEATURE_ZENDESK plus "
                             "CX_ZENDESK_SUBDOMAIN, CX_ZENDESK_EMAIL and CX_ZENDESK_TOKEN.")
-    if kind == "jira" and not settings.jira_ready:
-        raise DeliveryError("Jira is not configured. Set CX_FEATURE_JIRA plus "
-                            "CX_JIRA_BASE, CX_JIRA_EMAIL, CX_JIRA_TOKEN and CX_JIRA_PROJECT.")
+    if kind == "jira":
+        creds = settings.jira_for(scope)
+        if not (settings.feature_jira and creds["base"] and creds["email"] and creds["token"]):
+            prefix = "CX_RUNPOD_JIRA_" if scope == "runpod" else "CX_JIRA_"
+            raise DeliveryError(
+                f"Jira ({scope}) is not configured. Set CX_FEATURE_JIRA plus {prefix}BASE, "
+                f"{prefix}EMAIL and {prefix}TOKEN - or leave the {prefix}* values blank to reuse "
+                "the default instance."
+            )
 
 
 def sends_today(db: Session) -> int:
@@ -125,12 +131,20 @@ async def send_zendesk(db: Session, case: Case, actor: User, *, to: str, subject
 
 async def create_jira(db: Session, case: Case, actor: User, *, summary: str, description: str,
                       project: str = "", issue_type: str = "",
-                      labels: Optional[list[str]] = None) -> dict[str, Any]:
-    _guard("jira")
+                      labels: Optional[list[str]] = None,
+                      scope: str = "default") -> dict[str, Any]:
+    """Raise a Jira issue on the instance that owns this kind of work.
+
+    `scope="runpod"` targets the RunPod/RMA project, which may live on an
+    entirely different Atlassian site - hence separate credentials rather than
+    just a different project key.
+    """
+    _guard("jira", scope)
     _check_cap(db)
 
-    base = settings.jira_base.rstrip("/")
-    auth = (settings.jira_email, settings.jira_token)
+    creds = settings.jira_for(scope)
+    base = creds["base"].rstrip("/")
+    auth = (creds["email"], creds["token"])
     label = f"cx-triage-{case.fingerprint}"
     all_labels = sorted(set((labels or []) + ["cx-triage", label]))
 
@@ -147,9 +161,9 @@ async def create_jira(db: Session, case: Case, actor: User, *, summary: str, des
             return {"ok": True, "key": key, "url": url, "action": "existing"}
 
         payload = {"fields": {
-            "project": {"key": project or settings.jira_project},
+            "project": {"key": project or creds["project"]},
             "summary": summary[:250],
-            "issuetype": {"name": issue_type or settings.jira_issue_type},
+            "issuetype": {"name": issue_type or creds["issue_type"]},
             "labels": all_labels,
             "description": {
                 "type": "doc", "version": 1,

backend/app/main.py

@@ -13,11 +13,15 @@ from .auth import ensure_bootstrap_admin
 from .config import get_settings
 from .db import SessionLocal, init_db
 from .routers import (actions_router, alerts_router, auth_router, cases_router,
-                      linkage_router, settings_router)
+                      handover_router, linkage_router, runpod_router, settings_router)
 
 settings = get_settings()
 
 
+def _bool_env(name: str) -> bool:
+    return str(os.environ.get(name, "")).strip().lower() in {"1", "true", "yes", "on"}
+
+
 @asynccontextmanager
 async def lifespan(app: FastAPI):
     init_db()
@@ -28,6 +32,17 @@ async def lifespan(app: FastAPI):
             print(f"[auth] {message}")
     finally:
         db.close()
+    if _bool_env("CX_SEED_DEMO"):
+        db = SessionLocal()
+        try:
+            from .seed.loader import run as seed_run
+            for line in seed_run(db):
+                print(f"[seed] {line}")
+        except Exception as exc:
+            print(f"[seed] skipped: {type(exc).__name__}: {exc}")
+        finally:
+            db.close()
+
     print(f"[startup] {settings.app_name} {VERSION}")
     print(f"[startup] prometheus: {settings.prometheus_base}")
     print(f"[startup] sso: {'on' if settings.oidc_enabled else 'off'} | "
@@ -41,7 +56,7 @@ async def lifespan(app: FastAPI):
 app = FastAPI(title=settings.app_name, version=VERSION, lifespan=lifespan)
 
 for module in (auth_router, alerts_router, cases_router, settings_router,
-               linkage_router, actions_router):
+               linkage_router, actions_router, handover_router, runpod_router):
     app.include_router(module.router)
 
 

backend/app/models.py

@@ -12,7 +12,7 @@ import datetime as dt
 import enum
 from typing import Any
 
-from sqlalchemy import (JSON, Boolean, DateTime, Enum, ForeignKey, Index, Integer,
+from sqlalchemy import (JSON, Boolean, Date, DateTime, Enum, ForeignKey, Index, Integer,
                         String, Text, UniqueConstraint)
 from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, relationship
 
@@ -25,6 +25,18 @@ def _now() -> dt.datetime:
     return dt.datetime.now(dt.timezone.utc)
 
 
+def _as_utc(value: dt.datetime | None) -> dt.datetime | None:
+    """Re-attach UTC to a value read back from the database.
+
+    SQLite has no timezone type, so a datetime stored as aware comes back naive
+    and cannot be compared with `now()`. Postgres round-trips correctly; this
+    keeps both behaving the same.
+    """
+    if value is None:
+        return None
+    return value if value.tzinfo else value.replace(tzinfo=dt.timezone.utc)
+
+
 class CaseStatus(str, enum.Enum):
     """Where a piece of work has got to."""
 
@@ -206,3 +218,249 @@ class AppSetting(Base):
     key: Mapped[str] = mapped_column(String(80), primary_key=True)
     value: Mapped[dict[str, Any] | None] = mapped_column(JSON, nullable=True)
     updated_at: Mapped[dt.datetime] = mapped_column(DateTime(timezone=True), default=_now, onupdate=_now)
+
+
+# --- shift handover ---------------------------------------------------------
+
+class ShiftName(str, enum.Enum):
+    APAC = "APAC"
+    EMEA = "EMEA"
+    AMER = "AMER"
+
+
+class HandoverStatus(str, enum.Enum):
+    DRAFT = "draft"
+    HANDED_OVER = "handed_over"
+
+
+class ItemState(str, enum.Enum):
+    """The status line CX writes at the bottom of each key update."""
+
+    PENDING_INFRA = "pending_infra"
+    PENDING_CUSTOMER = "pending_customer"
+    PENDING_RUNPOD = "pending_runpod"
+    PENDING_RMA = "pending_rma"
+    PENDING_CX = "pending_cx"
+    IN_PROGRESS = "in_progress"
+    MONITORING = "monitoring"
+    NO_FURTHER_ENGAGEMENT = "no_further_engagement"
+    DONE = "done"
+
+
+class Handover(Base):
+    """One shift handover document."""
+
+    __tablename__ = "handovers"
+    __table_args__ = (UniqueConstraint("shift_date", "shift", name="uq_handover_date_shift"),)
+
+    id: Mapped[int] = mapped_column(primary_key=True)
+    shift_date: Mapped[dt.date] = mapped_column(Date, index=True)
+    shift: Mapped[ShiftName] = mapped_column(Enum(ShiftName), default=ShiftName.APAC)
+    handing_to: Mapped[str] = mapped_column(String(40), default="")   # "EMEA" in "APAC -> EMEA"
+    status: Mapped[HandoverStatus] = mapped_column(Enum(HandoverStatus), default=HandoverStatus.DRAFT)
+
+    team_members: Mapped[str] = mapped_column(String(400), default="")
+    # The tick-boxes at the top of the Confluence page.
+    significant_issues_checked: Mapped[bool] = mapped_column(Boolean, default=False)
+    hubspot_checked: Mapped[bool] = mapped_column(Boolean, default=False)
+    total_open_tickets: Mapped[int | None] = mapped_column(Integer, nullable=True)
+    # [{"name": "Parham", "hs_checked": true, "jira_checked": true}, ...]
+    member_checks: Mapped[list[dict[str, Any]] | None] = mapped_column(JSON, default=list)
+
+    other_comments: Mapped[str] = mapped_column(Text, default="")
+    reviewed_by: Mapped[str] = mapped_column(String(200), default="")
+    reviewed_at_utc: Mapped[str] = mapped_column(String(40), default="")
+    following_shift_checked: Mapped[bool] = mapped_column(Boolean, default=False)
+
+    created_by_id: Mapped[int | None] = mapped_column(ForeignKey("users.id"), nullable=True)
+    created_by: Mapped[User | None] = relationship(lazy="joined")
+    created_at: Mapped[dt.datetime] = mapped_column(DateTime(timezone=True), default=_now)
+    updated_at: Mapped[dt.datetime] = mapped_column(DateTime(timezone=True), default=_now, onupdate=_now)
+
+    items: Mapped[list["HandoverItem"]] = relationship(
+        back_populates="handover", cascade="all, delete-orphan", order_by="HandoverItem.position")
+
+    def to_json(self, with_items: bool = False) -> dict[str, Any]:
+        data = {
+            "id": self.id,
+            "shift_date": self.shift_date.isoformat() if self.shift_date else None,
+            "shift": self.shift.value,
+            "handing_to": self.handing_to,
+            "status": self.status.value,
+            "title": f"{self.shift.value} {self.shift_date.strftime('%d %B %Y')}" if self.shift_date else "",
+            "team_members": self.team_members,
+            "significant_issues_checked": self.significant_issues_checked,
+            "hubspot_checked": self.hubspot_checked,
+            "total_open_tickets": self.total_open_tickets,
+            "member_checks": self.member_checks or [],
+            "other_comments": self.other_comments,
+            "reviewed_by": self.reviewed_by,
+            "reviewed_at_utc": self.reviewed_at_utc,
+            "following_shift_checked": self.following_shift_checked,
+            "created_by": self.created_by.to_json() if self.created_by else None,
+            "updated_at": self.updated_at.isoformat() if self.updated_at else None,
+            "item_count": len(self.items),
+        }
+        if with_items:
+            data["items"] = [i.to_json() for i in self.items]
+        return data
+
+
+class HandoverItem(Base):
+    """One key update: the Zendesk/Jira refs, the narrative, and where it stands.
+
+    Carried forward between shifts rather than retyped - `carried_from_id`
+    records where it came from, so an issue keeps one thread across days.
+    """
+
+    __tablename__ = "handover_items"
+
+    id: Mapped[int] = mapped_column(primary_key=True)
+    handover_id: Mapped[int] = mapped_column(ForeignKey("handovers.id"), index=True)
+    handover: Mapped[Handover] = relationship(back_populates="items")
+
+    position: Mapped[int] = mapped_column(Integer, default=0)
+    title: Mapped[str] = mapped_column(String(400), default="")
+    zendesk_tickets: Mapped[str] = mapped_column(String(300), default="")  # "#8495, #8453"
+    jira_key: Mapped[str] = mapped_column(String(60), default="")
+    jira_status: Mapped[str] = mapped_column(String(60), default="")
+    body: Mapped[str] = mapped_column(Text, default="")
+    links: Mapped[list[str] | None] = mapped_column(JSON, default=list)
+    state: Mapped[ItemState] = mapped_column(Enum(ItemState), default=ItemState.IN_PROGRESS)
+    # "Remove at end of shift" in the doc.
+    remove_at_end_of_shift: Mapped[bool] = mapped_column(Boolean, default=False)
+
+    carried_from_id: Mapped[int | None] = mapped_column(ForeignKey("handover_items.id"), nullable=True)
+    first_raised_on: Mapped[dt.date | None] = mapped_column(Date, nullable=True)
+    created_at: Mapped[dt.datetime] = mapped_column(DateTime(timezone=True), default=_now)
+
+    def to_json(self) -> dict[str, Any]:
+        return {
+            "id": self.id, "position": self.position, "title": self.title,
+            "zendesk_tickets": self.zendesk_tickets, "jira_key": self.jira_key,
+            "jira_status": self.jira_status, "body": self.body, "links": self.links or [],
+            "state": self.state.value,
+            "remove_at_end_of_shift": self.remove_at_end_of_shift,
+            "carried_from_id": self.carried_from_id,
+            "first_raised_on": self.first_raised_on.isoformat() if self.first_raised_on else None,
+        }
+
+
+# --- RunPod -----------------------------------------------------------------
+
+class RunpodColour(str, enum.Enum):
+    """The colour key CX uses on the handover doc, kept verbatim."""
+
+    RED = "red"        # blocked from relisting, recurring issue
+    PURPLE = "purple"  # waiting on RunPod
+    YELLOW = "yellow"  # waiting on Infrastructure or the DC team
+    BLUE = "blue"      # pending removal from the RunPod platform
+    GREEN = "green"    # stress testing >24h, GPUs may look reserved
+    WHITE = "white"    # actionable by CX
+
+
+class RunpodEventType(str, enum.Enum):
+    UNLISTED = "unlisted"
+    LISTED = "listed"
+    MAINTENANCE_SCHEDULED = "maintenance_scheduled"
+    MAINTENANCE_ENDED = "maintenance_ended"
+    DRAINED = "drained"            # unlisted and rented GPUs reached zero
+    NOTE = "note"
+    ZENDESK_TICKET = "zendesk_ticket"
+    JIRA_LINKED = "jira_linked"
+
+
+class RunpodHost(Base):
+    """A RunPod machine, and where CX has got to with it."""
+
+    __tablename__ = "runpod_hosts"
+
+    id: Mapped[int] = mapped_column(primary_key=True)
+    machine_id: Mapped[str] = mapped_column(String(40), unique=True, index=True)
+    name: Mapped[str] = mapped_column(String(200), default="", index=True)
+
+    listed: Mapped[bool] = mapped_column(Boolean, default=True, index=True)
+    gpu_reserved: Mapped[int] = mapped_column(Integer, default=0)
+    gpu_total: Mapped[int] = mapped_column(Integer, default=0)
+    gpu_type: Mapped[str] = mapped_column(String(80), default="")
+    data_center: Mapped[str] = mapped_column(String(40), default="")
+    uptime_one_week: Mapped[float | None] = mapped_column(nullable=True)
+    maintenance_start: Mapped[str] = mapped_column(String(40), default="")
+    maintenance_end: Mapped[str] = mapped_column(String(40), default="")
+
+    # CX-side state, mirrored onto the handover doc.
+    colour: Mapped[RunpodColour] = mapped_column(Enum(RunpodColour), default=RunpodColour.WHITE)
+    zendesk_ticket: Mapped[str] = mapped_column(String(40), default="")
+    jira_key: Mapped[str] = mapped_column(String(60), default="")
+    jira_status: Mapped[str] = mapped_column(String(60), default="")
+    last_error: Mapped[str] = mapped_column(Text, default="")     # hint from the RunPod email
+    next_steps: Mapped[str] = mapped_column(Text, default="")
+
+    # How often this machine has been unlisted; the "problem host" ranking.
+    unlist_count: Mapped[int] = mapped_column(Integer, default=0, index=True)
+    historic_count: Mapped[int] = mapped_column(Integer, default=0)
+    unlisted_at: Mapped[dt.datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
+    last_listed_at: Mapped[dt.datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
+    first_seen_at: Mapped[dt.datetime] = mapped_column(DateTime(timezone=True), default=_now)
+    updated_at: Mapped[dt.datetime] = mapped_column(DateTime(timezone=True), default=_now, onupdate=_now)
+
+    events: Mapped[list["RunpodEvent"]] = relationship(
+        back_populates="host", cascade="all, delete-orphan",
+        order_by="RunpodEvent.occurred_at.desc()")
+
+    @property
+    def hours_unlisted(self) -> float | None:
+        started = _as_utc(self.unlisted_at)
+        if self.listed or not started:
+            return None
+        return round((_now() - started).total_seconds() / 3600, 1)
+
+    def to_json(self, with_events: bool = False) -> dict[str, Any]:
+        data = {
+            "id": self.id, "machine_id": self.machine_id, "name": self.name,
+            "listed": self.listed, "gpu_reserved": self.gpu_reserved, "gpu_total": self.gpu_total,
+            "gpu_type": self.gpu_type, "data_center": self.data_center,
+            "uptime_one_week": self.uptime_one_week,
+            "maintenance_start": self.maintenance_start, "maintenance_end": self.maintenance_end,
+            "colour": self.colour.value, "zendesk_ticket": self.zendesk_ticket,
+            "jira_key": self.jira_key, "jira_status": self.jira_status,
+            "last_error": self.last_error, "next_steps": self.next_steps,
+            "unlist_count": self.unlist_count, "historic_count": self.historic_count,
+            "unlisted_at": self.unlisted_at.isoformat() if self.unlisted_at else None,
+            "hours_unlisted": self.hours_unlisted,
+            "event_count": len(self.events),
+        }
+        if with_events:
+            data["events"] = [e.to_json() for e in self.events]
+        return data
+
+
+class RunpodEvent(Base):
+    """Append-only history for a machine: listings, unlistings and who did what."""
+
+    __tablename__ = "runpod_events"
+
+    id: Mapped[int] = mapped_column(primary_key=True)
+    host_id: Mapped[int] = mapped_column(ForeignKey("runpod_hosts.id"), index=True)
+    host: Mapped[RunpodHost] = relationship(back_populates="events")
+
+    event_type: Mapped[RunpodEventType] = mapped_column(Enum(RunpodEventType), index=True)
+    # "runpod" when RunPod unlisted it automatically, otherwise the operator.
+    actor: Mapped[str] = mapped_column(String(200), default="system")
+    detail: Mapped[str] = mapped_column(Text, default="")
+    error_hint: Mapped[str] = mapped_column(Text, default="")   # parsed from the unlisting email
+    zendesk_ticket: Mapped[str] = mapped_column(String(40), default="")
+    jira_key: Mapped[str] = mapped_column(String(60), default="")
+    gpu_reserved: Mapped[int | None] = mapped_column(Integer, nullable=True)
+    payload: Mapped[dict[str, Any] | None] = mapped_column(JSON, nullable=True)
+    occurred_at: Mapped[dt.datetime] = mapped_column(DateTime(timezone=True), default=_now, index=True)
+
+    def to_json(self) -> dict[str, Any]:
+        return {
+            "id": self.id, "event_type": self.event_type.value, "actor": self.actor,
+            "detail": self.detail, "error_hint": self.error_hint,
+            "zendesk_ticket": self.zendesk_ticket, "jira_key": self.jira_key,
+            "gpu_reserved": self.gpu_reserved,
+            "occurred_at": self.occurred_at.isoformat() if self.occurred_at else None,
+            "payload": self.payload,
+        }

backend/app/routers/handover_router.py

@@ -0,0 +1,222 @@
+"""Shift handover: the document CX fills in at the end of every shift."""
+from __future__ import annotations
+
+import datetime as dt
+
+from fastapi import APIRouter, Depends, HTTPException, status
+from pydantic import BaseModel
+from sqlalchemy import select
+from sqlalchemy.orm import Session
+
+from ..auth import current_user
+from ..db import get_db
+from ..models import (Handover, HandoverItem, HandoverStatus, ItemState, RunpodColour,
+                      RunpodHost, ShiftName, User)
+
+router = APIRouter(prefix="/api/handover", tags=["handover"])
+
+
+class HandoverBody(BaseModel):
+    shift_date: str | None = None
+    shift: str = "APAC"
+    handing_to: str = ""
+    team_members: str = ""
+    significant_issues_checked: bool = False
+    hubspot_checked: bool = False
+    total_open_tickets: int | None = None
+    member_checks: list[dict] = []
+    other_comments: str = ""
+    reviewed_by: str = ""
+    reviewed_at_utc: str = ""
+    following_shift_checked: bool = False
+
+
+class ItemBody(BaseModel):
+    title: str = ""
+    zendesk_tickets: str = ""
+    jira_key: str = ""
+    jira_status: str = ""
+    body: str = ""
+    links: list[str] = []
+    state: str = "in_progress"
+    remove_at_end_of_shift: bool = False
+    position: int | None = None
+
+
+def _get(db: Session, handover_id: int) -> Handover:
+    found = db.get(Handover, handover_id)
+    if not found:
+        raise HTTPException(status.HTTP_404_NOT_FOUND, "No such handover")
+    return found
+
+
+@router.get("")
+def list_handovers(limit: int = 30, db: Session = Depends(get_db),
+                   user: User = Depends(current_user)):
+    rows = db.scalars(select(Handover).order_by(Handover.shift_date.desc(), Handover.id.desc())
+                      .limit(max(1, min(limit, 200)))).all()
+    return {
+        "handovers": [h.to_json() for h in rows],
+        "shifts": [s.value for s in ShiftName],
+        "states": [s.value for s in ItemState],
+    }
+
+
+@router.get("/current")
+def current(db: Session = Depends(get_db), user: User = Depends(current_user)):
+    """The newest draft, or the newest handover of any kind."""
+    row = db.scalars(select(Handover).where(Handover.status == HandoverStatus.DRAFT)
+                     .order_by(Handover.shift_date.desc(), Handover.id.desc()).limit(1)).first()
+    if row is None:
+        row = db.scalars(select(Handover).order_by(Handover.shift_date.desc(),
+                                                   Handover.id.desc()).limit(1)).first()
+    if row is None:
+        return {"handover": None}
+    return {"handover": row.to_json(with_items=True), "runpod": _runpod_section(db)}
+
+
+@router.get("/{handover_id}")
+def get_one(handover_id: int, db: Session = Depends(get_db), user: User = Depends(current_user)):
+    return {"handover": _get(db, handover_id).to_json(with_items=True), "runpod": _runpod_section(db)}
+
+
+def _runpod_section(db: Session) -> list[dict]:
+    """The RunPod table on the handover, straight from live host state.
+
+    This is the part that used to be retyped by hand every shift.
+    """
+    rows = db.scalars(select(RunpodHost).where(RunpodHost.listed.is_(False))
+                      .order_by(RunpodHost.unlist_count.desc(), RunpodHost.name)).all()
+    return [{
+        "machine_id": h.machine_id, "name": h.name, "colour": h.colour.value,
+        "zendesk_ticket": h.zendesk_ticket, "jira_key": h.jira_key, "jira_status": h.jira_status,
+        "gpu_reserved": h.gpu_reserved, "gpu_total": h.gpu_total,
+        "last_error": h.last_error, "next_steps": h.next_steps,
+        "hours_unlisted": h.hours_unlisted, "unlist_count": h.unlist_count,
+    } for h in rows]
+
+
+@router.post("")
+def create(body: HandoverBody, db: Session = Depends(get_db), user: User = Depends(current_user)):
+    shift_date = dt.date.fromisoformat(body.shift_date) if body.shift_date else dt.date.today()
+    try:
+        shift = ShiftName(body.shift)
+    except ValueError as exc:
+        raise HTTPException(status.HTTP_400_BAD_REQUEST, f"Unknown shift '{body.shift}'") from exc
+
+    existing = db.scalars(select(Handover).where(Handover.shift_date == shift_date,
+                                                 Handover.shift == shift)).first()
+    if existing:
+        raise HTTPException(status.HTTP_409_CONFLICT,
+                            f"A {shift.value} handover already exists for {shift_date}")
+
+    row = Handover(shift_date=shift_date, shift=shift, handing_to=body.handing_to,
+                   team_members=body.team_members, created_by_id=user.id)
+    db.add(row)
+    db.commit()
+    return row.to_json(with_items=True)
+
+
+@router.post("/{handover_id}")
+def update(handover_id: int, body: HandoverBody, db: Session = Depends(get_db),
+           user: User = Depends(current_user)):
+    row = _get(db, handover_id)
+    for field in ("handing_to", "team_members", "significant_issues_checked", "hubspot_checked",
+                  "total_open_tickets", "member_checks", "other_comments", "reviewed_by",
+                  "reviewed_at_utc", "following_shift_checked"):
+        setattr(row, field, getattr(body, field))
+    if body.shift_date:
+        row.shift_date = dt.date.fromisoformat(body.shift_date)
+    db.commit()
+    return row.to_json(with_items=True)
+
+
+@router.post("/{handover_id}/hand-over")
+def hand_over(handover_id: int, db: Session = Depends(get_db), user: User = Depends(current_user)):
+    """Close the shift and start the next one, carrying the live items forward.
+
+    Items flagged "remove at end of shift", and anything already done, are left
+    behind - which is the manual step this replaces.
+    """
+    row = _get(db, handover_id)
+    row.status = HandoverStatus.HANDED_OVER
+
+    order = [ShiftName.APAC, ShiftName.EMEA, ShiftName.AMER]
+    next_shift = order[(order.index(row.shift) + 1) % len(order)]
+    next_date = row.shift_date + dt.timedelta(days=1) if next_shift == ShiftName.APAC else row.shift_date
+
+    following = db.scalars(select(Handover).where(Handover.shift_date == next_date,
+                                                  Handover.shift == next_shift)).first()
+    if following is None:
+        following = Handover(shift_date=next_date, shift=next_shift,
+                             handing_to=order[(order.index(next_shift) + 1) % len(order)].value,
+                             created_by_id=user.id)
+        db.add(following)
+        db.flush()
+
+    carried = 0
+    closed = {ItemState.DONE, ItemState.NO_FURTHER_ENGAGEMENT}
+    for item in row.items:
+        if item.remove_at_end_of_shift or item.state in closed:
+            continue
+        following.items.append(HandoverItem(
+            position=item.position, title=item.title, zendesk_tickets=item.zendesk_tickets,
+            jira_key=item.jira_key, jira_status=item.jira_status, body=item.body,
+            links=list(item.links or []), state=item.state,
+            carried_from_id=item.id,
+            first_raised_on=item.first_raised_on or row.shift_date,
+        ))
+        carried += 1
+
+    db.commit()
+    return {"closed": row.to_json(), "next": following.to_json(with_items=True), "carried": carried}
+
+
+@router.post("/{handover_id}/items")
+def add_item(handover_id: int, body: ItemBody, db: Session = Depends(get_db),
+             user: User = Depends(current_user)):
+    row = _get(db, handover_id)
+    try:
+        state = ItemState(body.state)
+    except ValueError as exc:
+        raise HTTPException(status.HTTP_400_BAD_REQUEST, f"Unknown state '{body.state}'") from exc
+    item = HandoverItem(
+        position=body.position if body.position is not None else len(row.items),
+        title=body.title, zendesk_tickets=body.zendesk_tickets, jira_key=body.jira_key,
+        jira_status=body.jira_status, body=body.body, links=body.links, state=state,
+        remove_at_end_of_shift=body.remove_at_end_of_shift, first_raised_on=row.shift_date,
+    )
+    row.items.append(item)
+    db.commit()
+    return row.to_json(with_items=True)
+
+
+@router.post("/{handover_id}/items/{item_id}")
+def update_item(handover_id: int, item_id: int, body: ItemBody, db: Session = Depends(get_db),
+                user: User = Depends(current_user)):
+    row = _get(db, handover_id)
+    item = next((i for i in row.items if i.id == item_id), None)
+    if item is None:
+        raise HTTPException(status.HTTP_404_NOT_FOUND, "No such item on this handover")
+    try:
+        item.state = ItemState(body.state)
+    except ValueError as exc:
+        raise HTTPException(status.HTTP_400_BAD_REQUEST, f"Unknown state '{body.state}'") from exc
+    for field in ("title", "zendesk_tickets", "jira_key", "jira_status", "body", "links",
+                  "remove_at_end_of_shift"):
+        setattr(item, field, getattr(body, field))
+    if body.position is not None:
+        item.position = body.position
+    db.commit()
+    return row.to_json(with_items=True)
+
+
+@router.delete("/{handover_id}/items/{item_id}")
+def delete_item(handover_id: int, item_id: int, db: Session = Depends(get_db),
+                user: User = Depends(current_user)):
+    row = _get(db, handover_id)
+    item = next((i for i in row.items if i.id == item_id), None)
+    if item is not None:
+        db.delete(item)
+        db.commit()
+    return _get(db, handover_id).to_json(with_items=True)

backend/app/routers/runpod_router.py

@@ -0,0 +1,262 @@
+"""RunPod machines: current state, per-host history, and problem ranking."""
+from __future__ import annotations
+
+import datetime as dt
+
+from fastapi import APIRouter, Depends, HTTPException, status
+from pydantic import BaseModel
+from sqlalchemy import func, select
+from sqlalchemy.orm import Session
+
+from ..auth import current_user
+from ..config import get_settings
+from ..db import get_db
+from ..models import RunpodColour, RunpodEvent, RunpodEventType, RunpodHost, User
+from ..runpod.client import RunPodClient, RunPodError
+from ..runpod.email_parse import parse as parse_email
+from ..runpod.delivery import raise_rma_issue, raise_unlisting_ticket, unlisting_ticket
+from ..runpod.service import record_event, sync_machines
+from ..delivery import DeliveryError
+
+router = APIRouter(prefix="/api/runpod", tags=["runpod"])
+settings = get_settings()
+
+
+def _client() -> RunPodClient:
+    return RunPodClient(
+        api_key=settings.runpod_api_key, email=settings.runpod_email,
+        password=settings.runpod_password, team_id=settings.runpod_team_id,
+        totp_secret=settings.runpod_totp_secret,
+    )
+
+
+class HostPatch(BaseModel):
+    colour: str | None = None
+    zendesk_ticket: str | None = None
+    jira_key: str | None = None
+    jira_status: str | None = None
+    next_steps: str | None = None
+    last_error: str | None = None
+
+
+class NoteBody(BaseModel):
+    detail: str
+    event_type: str = "note"
+    zendesk_ticket: str = ""
+    jira_key: str = ""
+
+
+class EmailBody(BaseModel):
+    raw: str
+    subject: str = ""
+
+
+class TicketBody(BaseModel):
+    error_text: str = ""
+    requester: str = ""
+    subject: str = ""
+    body: str = ""
+
+
+class RmaBody(BaseModel):
+    summary: str = ""
+    description: str = ""
+
+
+@router.get("/status")
+def runpod_status(db: Session = Depends(get_db), user: User = Depends(current_user)):
+    hosts = db.scalars(select(RunpodHost)).all()
+    unlisted = [h for h in hosts if not h.listed]
+    return {
+        "configured": settings.runpod_ready,
+        "mode": _client().mode,
+        "write_enabled": settings.feature_runpod_write,
+        "totals": {
+            "machines": len(hosts),
+            "listed": len(hosts) - len(unlisted),
+            "unlisted": len(unlisted),
+            "gpus_rented": sum(h.gpu_reserved for h in hosts),
+            "gpus_total": sum(h.gpu_total for h in hosts),
+        },
+        "colours": [c.value for c in RunpodColour],
+        "event_types": [e.value for e in RunpodEventType],
+    }
+
+
+@router.get("/hosts")
+def hosts(unlisted_only: bool = False, db: Session = Depends(get_db),
+          user: User = Depends(current_user)):
+    query = select(RunpodHost)
+    if unlisted_only:
+        query = query.where(RunpodHost.listed.is_(False))
+    rows = db.scalars(query.order_by(RunpodHost.listed, RunpodHost.name)).all()
+    return {"hosts": [h.to_json() for h in rows]}
+
+
+@router.get("/hosts/{machine_id}")
+def host_detail(machine_id: str, db: Session = Depends(get_db), user: User = Depends(current_user)):
+    row = db.scalars(select(RunpodHost).where(RunpodHost.machine_id == machine_id)).first()
+    if row is None:
+        raise HTTPException(status.HTTP_404_NOT_FOUND, "No such machine")
+    return row.to_json(with_events=True)
+
+
+@router.get("/problem-hosts")
+def problem_hosts(limit: int = 40, db: Session = Depends(get_db),
+                  user: User = Depends(current_user)):
+    """Machines ranked by how often they have been unlisted.
+
+    The repeat offenders are the ones worth an RMA conversation rather than
+    another burn-in, which is what the old script's historical table showed.
+    """
+    unlist_counts = (
+        select(RunpodEvent.host_id, func.count(RunpodEvent.id).label("n"),
+               func.max(RunpodEvent.occurred_at).label("last"))
+        .where(RunpodEvent.event_type == RunpodEventType.UNLISTED)
+        .group_by(RunpodEvent.host_id).subquery()
+    )
+    rows = db.execute(
+        select(RunpodHost, unlist_counts.c.n, unlist_counts.c.last)
+        .join(unlist_counts, unlist_counts.c.host_id == RunpodHost.id, isouter=True)
+        .order_by(func.coalesce(unlist_counts.c.n, 0).desc(), RunpodHost.historic_count.desc())
+        .limit(max(1, min(limit, 200)))
+    ).all()
+
+    out = []
+    for host, count, last in rows:
+        total = int(count or 0) or host.historic_count
+        if not total:
+            continue
+        payload = host.to_json()
+        payload["unlist_events"] = int(count or 0)
+        payload["effective_count"] = total
+        payload["last_unlisted"] = last.isoformat() if last else (
+            host.unlisted_at.isoformat() if host.unlisted_at else None)
+        out.append(payload)
+    return {"hosts": out}
+
+
+@router.post("/sync")
+def sync(db: Session = Depends(get_db), user: User = Depends(current_user)):
+    if not settings.runpod_ready:
+        raise HTTPException(status.HTTP_400_BAD_REQUEST,
+                            "RunPod is not configured. Set CX_RUNPOD_API_KEY.")
+    try:
+        result = sync_machines(db, _client(), actor=user.email)
+    except RunPodError as exc:
+        raise HTTPException(status.HTTP_502_BAD_GATEWAY, str(exc)) from exc
+    return result
+
+
+@router.post("/hosts/{machine_id}")
+def patch_host(machine_id: str, body: HostPatch, db: Session = Depends(get_db),
+               user: User = Depends(current_user)):
+    row = db.scalars(select(RunpodHost).where(RunpodHost.machine_id == machine_id)).first()
+    if row is None:
+        raise HTTPException(status.HTTP_404_NOT_FOUND, "No such machine")
+    changes = []
+    if body.colour is not None:
+        try:
+            row.colour = RunpodColour(body.colour)
+        except ValueError as exc:
+            raise HTTPException(status.HTTP_400_BAD_REQUEST, f"Unknown colour '{body.colour}'") from exc
+        changes.append(f"colour={body.colour}")
+    for field in ("zendesk_ticket", "jira_key", "jira_status", "next_steps", "last_error"):
+        value = getattr(body, field)
+        if value is not None:
+            setattr(row, field, value)
+            changes.append(field)
+    if changes:
+        record_event(db, row, RunpodEventType.NOTE, actor=user.email,
+                     detail="Updated " + ", ".join(changes))
+    db.commit()
+    return row.to_json(with_events=True)
+
+
+@router.post("/hosts/{machine_id}/events")
+def add_event(machine_id: str, body: NoteBody, db: Session = Depends(get_db),
+              user: User = Depends(current_user)):
+    row = db.scalars(select(RunpodHost).where(RunpodHost.machine_id == machine_id)).first()
+    if row is None:
+        raise HTTPException(status.HTTP_404_NOT_FOUND, "No such machine")
+    try:
+        event_type = RunpodEventType(body.event_type)
+    except ValueError as exc:
+        raise HTTPException(status.HTTP_400_BAD_REQUEST, f"Unknown event type") from exc
+    record_event(db, row, event_type, actor=user.email, detail=body.detail,
+                 zendesk_ticket=body.zendesk_ticket, jira_key=body.jira_key)
+    db.commit()
+    return row.to_json(with_events=True)
+
+
+@router.get("/hosts/{machine_id}/ticket-preview")
+def ticket_preview(machine_id: str, db: Session = Depends(get_db),
+                   user: User = Depends(current_user)):
+    """The Zendesk ticket that would be raised, without raising it."""
+    row = db.scalars(select(RunpodHost).where(RunpodHost.machine_id == machine_id)).first()
+    if row is None:
+        raise HTTPException(status.HTTP_404_NOT_FOUND, "No such machine")
+    return {
+        **unlisting_ticket(row),
+        "zendesk_ready": settings.zendesk_ready,
+        "send_enabled": settings.feature_send_enabled,
+        "runpod_jira_ready": settings.runpod_jira_ready,
+        "existing_ticket": row.zendesk_ticket,
+        "existing_jira": row.jira_key,
+    }
+
+
+@router.post("/hosts/{machine_id}/zendesk")
+async def zendesk_ticket(machine_id: str, body: TicketBody, db: Session = Depends(get_db),
+                         user: User = Depends(current_user)):
+    row = db.scalars(select(RunpodHost).where(RunpodHost.machine_id == machine_id)).first()
+    if row is None:
+        raise HTTPException(status.HTTP_404_NOT_FOUND, "No such machine")
+    try:
+        return await raise_unlisting_ticket(db, row, user, error_text=body.error_text,
+                                            requester=body.requester, subject=body.subject,
+                                            body=body.body)
+    except DeliveryError as exc:
+        raise HTTPException(status.HTTP_400_BAD_REQUEST, str(exc)) from exc
+
+
+@router.post("/hosts/{machine_id}/rma")
+async def rma_issue(machine_id: str, body: RmaBody, db: Session = Depends(get_db),
+                    user: User = Depends(current_user)):
+    row = db.scalars(select(RunpodHost).where(RunpodHost.machine_id == machine_id)).first()
+    if row is None:
+        raise HTTPException(status.HTTP_404_NOT_FOUND, "No such machine")
+    try:
+        return await raise_rma_issue(db, row, user, summary=body.summary,
+                                     description=body.description)
+    except DeliveryError as exc:
+        raise HTTPException(status.HTTP_400_BAD_REQUEST, str(exc)) from exc
+
+
+@router.post("/ingest-email")
+def ingest_email(body: EmailBody, db: Session = Depends(get_db), user: User = Depends(current_user)):
+    """Take a RunPod unlisting email and attach its error hint to the machine.
+
+    The email is the only place the actual failure reason appears, so it is
+    worth capturing even when the unlisting itself came through the API.
+    """
+    parsed = parse_email(body.raw, body.subject)
+    if not parsed["machine_id"] and not parsed["host"]:
+        raise HTTPException(status.HTTP_400_BAD_REQUEST,
+                            "Could not find a machine in that email.")
+    query = select(RunpodHost)
+    row = db.scalars(query.where(RunpodHost.machine_id == parsed["machine_id"])).first() \
+        if parsed["machine_id"] else None
+    if row is None and parsed["host"]:
+        row = db.scalars(query.where(RunpodHost.name == parsed["host"])).first()
+    if row is None:
+        return {"parsed": parsed, "matched": False,
+                "detail": "Parsed the email, but no machine in the database matches."}
+
+    row.last_error = parsed["error_text"] or row.last_error
+    record_event(db, row, RunpodEventType.UNLISTED, actor="runpod",
+                 detail=f"{parsed['signature']} — {parsed['suggested_action']}",
+                 error_hint=parsed["error_text"], gpu_reserved=parsed["gpus_rented"],
+                 payload={"source": "email", "critical": parsed["is_critical"]})
+    db.commit()
+    return {"parsed": parsed, "matched": True, "host": row.to_json(with_events=True)}

backend/app/runpod/client.py

@@ -0,0 +1,285 @@
+"""RunPod access.
+
+Two transports, deliberately unequal:
+
+* **API key (default).** One header against the documented GraphQL endpoint.
+  Nothing to expire, nothing to refresh, works from a container.
+* **Console login (fallback).** The old flow: sign in to Clerk with an email and
+  password, then reuse the session JWT. Kept because it is the only path that
+  works if the API key is revoked, but it cannot run unattended.
+
+The "No active session found" failure in the old script was not a UI change:
+the account now has two-factor authentication on. Clerk verifies the password
+and then answers ``status: needs_second_factor`` with an emailed code, so no
+session is ever created. That cannot be automated without reading the mailbox.
+If the account is switched from email codes to an authenticator app, set
+CX_RUNPOD_TOTP_SECRET and this fallback can complete on its own; otherwise
+CX_RUNPOD_2FA_CODE accepts a code for a single manual run.
+
+Set CX_RUNPOD_API_KEY and this module never touches the console at all.
+"""
+from __future__ import annotations
+
+import os
+import threading
+import time
+from typing import Any, Optional
+
+import httpx
+
+GRAPHQL_URL = os.environ.get("CX_RUNPOD_GRAPHQL_URL", "https://api.runpod.io/graphql")
+CLERK_BASE = os.environ.get("CX_RUNPOD_CLERK_BASE", "https://clerk.runpod.io")
+CONSOLE_URL = "https://console.runpod.io"
+CLERK_QS = "__clerk_api_version=2025-11-10&_clerk_js_version=5.125.12"
+
+
+class RunPodError(RuntimeError):
+    pass
+
+
+# --- queries ----------------------------------------------------------------
+
+MACHINES_QUERY = """
+query getMachinesForHostDashboard {
+  myself {
+    machineQuota
+    machines {
+      id name listed registered verified
+      gpuTypeId gpuReserved gpuTotal
+      dataCenterId machineType
+      hostPricePerGpu margin
+      uptimePercentListedOneWeek uptimePercentListedFourWeek
+      maintenanceStart maintenanceEnd
+      gpuType { displayName manufacturer }
+      machineSystem { os cudaVersion kernelVersion }
+    }
+  }
+}
+"""
+
+SUMMARY_QUERY = """
+query getMyMachines {
+  myself {
+    machinesSummary {
+      id displayName listed machineType
+      gpuTypeId gpuRented gpuTotal
+      podProfitPerHr diskProfitPerHr
+      onDemandPods spotPods
+    }
+  }
+}
+"""
+
+LIST_MUTATION = """
+mutation listMachineBulk($input: MachineListBulkInput) { machineListBulk(input: $input) }
+"""
+
+UNLIST_MUTATION = """
+mutation unlistMachineBulk($input: MachineUnlistBulkInput) { machineUnlistBulk(input: $input) }
+"""
+
+MAINTENANCE_MUTATION = """
+mutation machineScheduleMaintenance($input: MachineScheduleMaintenanceInput) {
+  machineScheduleMaintenance(input: $input)
+}
+"""
+
+
+class RunPodClient:
+    def __init__(self, api_key: str = "", email: str = "", password: str = "",
+                 team_id: str = "", totp_secret: str = "", otp_code: str = "",
+                 timeout: int = 30):
+        self.api_key = (api_key or "").strip()
+        self.email = (email or "").strip()
+        self.password = password or ""
+        self.team_id = (team_id or "").strip()
+        self.totp_secret = (totp_secret or "").strip()
+        self.otp_code = (otp_code or "").strip()
+        self.timeout = timeout
+        self._jwt: Optional[str] = None
+        self._jwt_at = 0.0
+        self._lock = threading.Lock()
+
+    # --- auth ---------------------------------------------------------------
+
+    @property
+    def mode(self) -> str:
+        if self.api_key:
+            return "api_key"
+        if self.email and self.password:
+            return "console_login"
+        return "unconfigured"
+
+    def _auth_header(self) -> dict[str, str]:
+        if self.api_key:
+            return {"Authorization": f"Bearer {self.api_key}"}
+        jwt = self._console_jwt()
+        header = {"Authorization": f"Bearer {jwt}"}
+        if self.team_id:
+            header["x-team-id"] = self.team_id
+        return header
+
+    def _console_jwt(self) -> str:
+        """Sign in to the console and return a session JWT.
+
+        Only used when no API key is set. Cached for 50 minutes; Clerk tokens
+        last an hour.
+        """
+        with self._lock:
+            if self._jwt and time.time() - self._jwt_at < 3000:
+                return self._jwt
+            if not (self.email and self.password):
+                raise RunPodError(
+                    "RunPod is not configured. Set CX_RUNPOD_API_KEY, or "
+                    "CX_RUNPOD_EMAIL and CX_RUNPOD_PASSWORD for the console fallback."
+                )
+
+            with httpx.Client(timeout=self.timeout, follow_redirects=True, headers={
+                "User-Agent": "Mozilla/5.0", "Origin": CONSOLE_URL, "Referer": f"{CONSOLE_URL}/",
+            }) as client:
+                client.get(CONSOLE_URL)
+                signin = client.post(
+                    f"{CLERK_BASE}/v1/client/sign_ins?{CLERK_QS}",
+                    data={"identifier": self.email, "password": self.password, "strategy": "password"},
+                )
+                if signin.status_code >= 400:
+                    raise RunPodError(f"Console sign-in rejected ({signin.status_code}): {signin.text[:200]}")
+
+                body = signin.json().get("response") or {}
+                session_id = body.get("created_session_id")
+
+                if body.get("status") == "needs_second_factor":
+                    session_id = self._second_factor(client, body)
+
+                # The old script read the session out of this response and gave
+                # up when it was absent. Ask for the client record instead.
+                jwt, found_id = self._session_from_client(client, session_id)
+                if not jwt and found_id:
+                    jwt = self._mint_token(client, found_id)
+                if not jwt:
+                    raise RunPodError(
+                        "Console sign-in completed but no session token came back. RunPod's console auth has "
+                        "changed again - use CX_RUNPOD_API_KEY instead."
+                    )
+                self._jwt, self._jwt_at = jwt, time.time()
+                return jwt
+
+    def _second_factor(self, client: httpx.Client, body: dict[str, Any]) -> Optional[str]:
+        """Satisfy Clerk's second factor, if we have been given the means to.
+
+        The account currently uses emailed codes, which no unattended process
+        can read - that is the real reason the old login stopped working.
+        """
+        sign_in_id = body.get("id")
+        offered = [f.get("strategy") for f in (body.get("supported_second_factors") or [])]
+
+        code = ""
+        if "totp" in offered and self.totp_secret:
+            code, strategy = self._totp_code(), "totp"
+        elif self.otp_code:
+            code, strategy = self.otp_code, ("email_code" if "email_code" in offered else offered[0] if offered else "")
+            if "email_code" in offered:
+                client.post(f"{CLERK_BASE}/v1/client/sign_ins/{sign_in_id}/prepare_second_factor?{CLERK_QS}",
+                            data={"strategy": "email_code"})
+        if not code:
+            raise RunPodError(
+                "RunPod console login needs a second factor "
+                f"({', '.join(offered) or 'unknown strategy'}) and no code is available. This is why the old "
+                "script failed - the password is accepted, but the account has 2FA on. Use CX_RUNPOD_API_KEY for "
+                "unattended runs, or set CX_RUNPOD_TOTP_SECRET if you move the account to an authenticator app."
+            )
+
+        attempt = client.post(
+            f"{CLERK_BASE}/v1/client/sign_ins/{sign_in_id}/attempt_second_factor?{CLERK_QS}",
+            data={"strategy": strategy, "code": code},
+        )
+        if attempt.status_code >= 400:
+            raise RunPodError(f"Second factor rejected ({attempt.status_code}): {attempt.text[:200]}")
+        return ((attempt.json().get("response") or {}).get("created_session_id"))
+
+    def _totp_code(self) -> str:
+        """RFC 6238 code from a base32 secret - no third-party dependency."""
+        import base64
+        import hashlib
+        import hmac
+        import struct
+
+        secret = self.totp_secret.replace(" ", "").upper()
+        secret += "=" * (-len(secret) % 8)
+        key = base64.b32decode(secret, casefold=True)
+        counter = struct.pack(">Q", int(time.time()) // 30)
+        digest = hmac.new(key, counter, hashlib.sha1).digest()
+        offset = digest[-1] & 0x0F
+        value = struct.unpack(">I", digest[offset:offset + 4])[0] & 0x7FFFFFFF
+        return f"{value % 1_000_000:06d}"
+
+    def _session_from_client(self, client: httpx.Client, prefer_id: Optional[str]) -> tuple[str, str]:
+        resp = client.get(f"{CLERK_BASE}/v1/client?{CLERK_QS}")
+        if resp.status_code >= 400:
+            return "", ""
+        sessions = ((resp.json().get("response") or {}).get("sessions") or [])
+        if not sessions:
+            return "", ""
+        chosen = next((s for s in sessions if s.get("id") == prefer_id), sessions[0])
+        token = ((chosen.get("last_active_token") or {}).get("jwt")) or ""
+        return token, str(chosen.get("id") or "")
+
+    def _mint_token(self, client: httpx.Client, session_id: str) -> str:
+        resp = client.post(f"{CLERK_BASE}/v1/client/sessions/{session_id}/tokens?{CLERK_QS}")
+        if resp.status_code >= 400:
+            return ""
+        return resp.json().get("jwt", "")
+
+    # --- transport ----------------------------------------------------------
+
+    def execute(self, query: str, variables: Optional[dict[str, Any]] = None) -> dict[str, Any]:
+        headers = {"Content-Type": "application/json", **self._auth_header()}
+        payload: dict[str, Any] = {"query": query}
+        if variables is not None:
+            payload["variables"] = variables
+        with httpx.Client(timeout=self.timeout) as client:
+            resp = client.post(GRAPHQL_URL, json=payload, headers=headers)
+        if resp.status_code == 401:
+            raise RunPodError("RunPod rejected the credentials (401). Check CX_RUNPOD_API_KEY.")
+        if resp.status_code >= 400:
+            raise RunPodError(f"RunPod returned {resp.status_code}: {resp.text[:300]}")
+        body = resp.json()
+        if body.get("errors"):
+            raise RunPodError("; ".join(e.get("message", "?") for e in body["errors"])[:400])
+        data = body.get("data")
+        if data is None:
+            raise RunPodError(f"RunPod returned no data: {str(body)[:200]}")
+        return data
+
+    # --- operations ---------------------------------------------------------
+
+    def machines(self) -> list[dict[str, Any]]:
+        data = self.execute(MACHINES_QUERY)
+        return list(((data.get("myself") or {}).get("machines")) or [])
+
+    def summary(self) -> list[dict[str, Any]]:
+        data = self.execute(SUMMARY_QUERY)
+        return list(((data.get("myself") or {}).get("machinesSummary")) or [])
+
+    def list_machines(self, machine_ids: list[str]) -> dict[str, Any]:
+        return self.execute(LIST_MUTATION, {"input": {"machineIds": machine_ids}})
+
+    def unlist_machines(self, machine_ids: list[str]) -> dict[str, Any]:
+        return self.execute(UNLIST_MUTATION, {"input": {"machineIds": machine_ids}})
+
+    def schedule_maintenance(self, machine_ids: list[str], start_utc: str, minutes: int,
+                             reason: str = "EMERGENCY", destructive: bool = False) -> dict[str, Any]:
+        if reason not in ("UPGRADE", "ROUTINE", "EMERGENCY", "REMOVE"):
+            raise RunPodError(f"Unknown maintenance reason '{reason}'.")
+        return self.execute(MAINTENANCE_MUTATION, {"input": {
+            "machineIds": machine_ids, "maintenanceStartUtc": start_utc,
+            "maintenanceMinutes": minutes, "maintenanceReason": reason, "destructive": destructive,
+        }})
+
+    def check(self) -> dict[str, Any]:
+        """Cheap connectivity probe for the health endpoint."""
+        try:
+            machines = self.machines()
+            return {"ok": True, "mode": self.mode, "machines": len(machines)}
+        except RunPodError as exc:
+            return {"ok": False, "mode": self.mode, "error": str(exc)}

backend/app/runpod/delivery.py

@@ -0,0 +1,201 @@
+"""Outbound actions for RunPod hosts: Zendesk notification and Jira/RMA escalation.
+
+An unlisted machine earns nothing and may be stranding rented workloads, so the
+team wants a ticket the moment it happens. RunPod's own email already says what
+broke; this reproduces that content in Zendesk so it lands in the tracking
+platform even when the unlisting was done through the API and no email was sent.
+
+The same three gates as customer comms apply - configured, feature-flagged, and
+sending enabled globally - so a demo instance cannot raise tickets.
+"""
+from __future__ import annotations
+
+import datetime as dt
+from typing import Any, Optional
+
+import httpx
+from sqlalchemy.orm import Session
+
+from ..config import get_settings
+from ..delivery import DeliveryError, _check_cap
+from ..models import RunpodEventType, RunpodHost, User
+from .email_parse import classify
+from .service import record_event
+
+settings = get_settings()
+
+
+def unlisting_ticket(host: RunpodHost, error_text: str = "") -> dict[str, str]:
+    """Compose the ticket body, following the shape of RunPod's own email.
+
+    Keeping the same structure means whoever picks the ticket up reads the
+    familiar thing: what broke, how many GPUs it is costing, and where to look.
+    """
+    error = (error_text or host.last_error or "").strip()
+    hint = classify(error)
+    rented = f"{host.gpu_reserved} GPU(s) currently rented on this machine." if host.gpu_reserved \
+        else "No GPUs currently rented - the machine has drained."
+
+    body = "\n".join([
+        f"Machine {host.name} ({host.machine_id}) is unlisted and is not accepting new users.",
+        "",
+        "Error detected:",
+        error or "(no error text captured - check the RunPod dashboard or the notification email)",
+        "",
+        f"Impact: {rented}",
+        f"Likely cause: {hint['signature']}",
+        f"Suggested next step: {hint['suggested_action']}",
+        "",
+        "Common causes and where to look:",
+        "- GPU error or failure: check nvidia-smi for GPU health and dmesg for Xid errors.",
+        "- Unresponsive Docker daemon: check whether the docker service is running or hung.",
+        "- Pod sync errors: if df -h hangs, suspect a disk error or hung moosefs mount.",
+        "- Docker overlay storage: confirm the docker filesystem is XFS and /var/lib/docker is mounted.",
+        "- Portallocator port check: verify the publicIp ports in /etc/runpod/config.json are reachable.",
+        "",
+        "If the fix needs a reboot or hardware work, schedule maintenance from the Machines "
+        "Dashboard rather than pulling the machine abruptly - that drains workloads gracefully.",
+        "",
+        f"Unlisted {host.unlist_count} time(s) to date.",
+        "Raised automatically by CX Triage.",
+    ])
+    return {
+        "subject": f"{host.name} Unlisted - {hint['signature']}",
+        "body": body,
+        "signature": hint["signature"],
+    }
+
+
+async def raise_unlisting_ticket(db: Session, host: RunpodHost, actor: User,
+                                 *, error_text: str = "", requester: str = "",
+                                 subject: str = "", body: str = "") -> dict[str, Any]:
+    """Create (or comment on) the Zendesk ticket for an unlisted machine."""
+    if not settings.feature_send_enabled:
+        raise DeliveryError("Sending is disabled on this instance (CX_FEATURE_SEND_ENABLED is off).")
+    if not settings.zendesk_ready:
+        raise DeliveryError("Zendesk is not configured. Set CX_FEATURE_ZENDESK plus "
+                            "CX_ZENDESK_SUBDOMAIN, CX_ZENDESK_EMAIL and CX_ZENDESK_TOKEN.")
+    _check_cap(db)
+
+    composed = unlisting_ticket(host, error_text)
+    subject = subject or composed["subject"]
+    body = body or composed["body"]
+
+    base = f"https://{settings.zendesk_subdomain}.zendesk.com/api/v2"
+    auth = (f"{settings.zendesk_email}/token", settings.zendesk_token)
+    # Keyed on the machine, not the incident: one machine, one running thread.
+    external_id = f"cx-triage-runpod-{host.machine_id}"
+
+    async with httpx.AsyncClient(timeout=30) as client:
+        found = await client.get(f"{base}/search.json",
+                                 params={"query": f'type:ticket external_id:"{external_id}"'}, auth=auth)
+        existing = None
+        if found.status_code == 200:
+            results = found.json().get("results") or []
+            # Only reuse a ticket that is still open; a solved one starts a new thread.
+            existing = next((t for t in results if t.get("status") not in ("solved", "closed")), None)
+
+        # Internal note by default: this is an operations ticket, not a customer reply.
+        comment = {"body": body, "public": False}
+        if existing:
+            resp = await client.put(f"{base}/tickets/{existing['id']}.json",
+                                    json={"ticket": {"comment": comment}}, auth=auth)
+            action = "updated"
+        else:
+            ticket: dict[str, Any] = {
+                "subject": subject,
+                "comment": comment,
+                "priority": "high" if host.gpu_reserved else "normal",
+                "type": "incident",
+                "tags": ["cx-triage", "runpod", "unlisted", f"dc-{(host.data_center or 'unknown').lower()}"],
+                "external_id": external_id,
+            }
+            if requester:
+                ticket["requester"] = {"name": requester.split("@")[0], "email": requester}
+            resp = await client.post(f"{base}/tickets.json", json={"ticket": ticket}, auth=auth)
+            action = "created"
+
+    if resp.status_code not in (200, 201):
+        record_event(db, host, RunpodEventType.NOTE, actor=actor.email,
+                     detail=f"Zendesk ticket failed: HTTP {resp.status_code}")
+        db.commit()
+        raise DeliveryError(f"Zendesk returned {resp.status_code}: {resp.text[:300]}")
+
+    ticket_id = str((resp.json().get("ticket") or {}).get("id")
+                    or (existing or {}).get("id") or "")
+    url = f"https://{settings.zendesk_subdomain}.zendesk.com/agent/tickets/{ticket_id}"
+
+    host.zendesk_ticket = ticket_id or host.zendesk_ticket
+    record_event(db, host, RunpodEventType.ZENDESK_TICKET, actor=actor.email,
+                 detail=f"Zendesk ticket {ticket_id} {action} - {composed['signature']}",
+                 error_hint=error_text or host.last_error, zendesk_ticket=ticket_id)
+    db.commit()
+    return {"ok": True, "ticket_id": ticket_id, "url": url, "action": action,
+            "subject": subject, "signature": composed["signature"]}
+
+
+async def raise_rma_issue(db: Session, host: RunpodHost, actor: User, *,
+                          summary: str = "", description: str = "") -> dict[str, Any]:
+    """Open an issue on the RunPod/RMA Jira - a different instance to the OIE one."""
+    creds = settings.jira_for("runpod")
+    if not settings.feature_send_enabled:
+        raise DeliveryError("Sending is disabled on this instance (CX_FEATURE_SEND_ENABLED is off).")
+    if not (settings.feature_jira and creds["base"] and creds["email"] and creds["token"]):
+        raise DeliveryError(
+            "The RunPod Jira is not configured. Set CX_RUNPOD_JIRA_BASE, CX_RUNPOD_JIRA_EMAIL and "
+            "CX_RUNPOD_JIRA_TOKEN - or leave them blank to reuse the default instance."
+        )
+    _check_cap(db)
+
+    hint = classify(host.last_error or "")
+    summary = summary or f"RunPod host {host.name} ({host.machine_id}) - {hint['signature']}"
+    description = description or "\n".join([
+        f"Machine: {host.name} ({host.machine_id})",
+        f"Data centre: {host.data_center or 'unknown'}",
+        f"GPUs: {host.gpu_reserved}/{host.gpu_total} rented",
+        f"Unlisted {host.unlist_count} time(s) to date.",
+        "",
+        "Last error:",
+        host.last_error or "(none captured)",
+        "",
+        f"Likely cause: {hint['signature']}",
+        f"Suggested next step: {hint['suggested_action']}",
+        "",
+        f"Zendesk: {host.zendesk_ticket or '(none)'}",
+        "Raised from CX Triage.",
+    ])
+
+    base = creds["base"].rstrip("/")
+    auth = (creds["email"], creds["token"])
+    label = f"cx-triage-runpod-{host.machine_id}"
+
+    async with httpx.AsyncClient(timeout=30) as client:
+        found = await client.get(f"{base}/rest/api/3/search",
+                                 params={"jql": f'labels = "{label}"', "maxResults": 1}, auth=auth)
+        if found.status_code == 200 and (found.json().get("issues") or []):
+            issue = found.json()["issues"][0]
+            key = issue["key"]
+            host.jira_key = key
+            record_event(db, host, RunpodEventType.JIRA_LINKED, actor=actor.email,
+                         detail=f"Issue {key} already exists", jira_key=key)
+            db.commit()
+            return {"ok": True, "key": key, "url": f"{base}/browse/{key}", "action": "existing"}
+
+        resp = await client.post(f"{base}/rest/api/3/issue", json={"fields": {
+            "project": {"key": creds["project"]},
+            "summary": summary[:250],
+            "issuetype": {"name": creds["issue_type"]},
+            "labels": ["cx-triage", "runpod", label],
+            "description": {"type": "doc", "version": 1, "content": [
+                {"type": "paragraph", "content": [{"type": "text", "text": description[:30000]}]}]},
+        }}, auth=auth)
+
+    if resp.status_code not in (200, 201):
+        raise DeliveryError(f"Jira returned {resp.status_code}: {resp.text[:300]}")
+
+    key = resp.json().get("key", "")
+    host.jira_key = key
+    record_event(db, host, RunpodEventType.JIRA_LINKED, actor=actor.email,
+                 detail=f"Issue {key} created on the RunPod Jira", jira_key=key)
+    db.commit()
+    return {"ok": True, "key": key, "url": f"{base}/browse/{key}", "action": "created"}

backend/app/runpod/email_parse.py

@@ -0,0 +1,114 @@
+"""Parse RunPod's automated unlisting emails.
+
+Subject:  "<host> Unlisted - CRITICAL ERROR"
+Body:     the machine id in brackets, an "Error detected:" block, and an
+          "Impact: N GPU(s) currently rented" line.
+
+The error block is the useful part - it is the only place the actual reason
+appears, and it is what CX pastes into the Zendesk ticket and the handover.
+Written to cope with the HTML mail as well as a plain-text paste.
+"""
+from __future__ import annotations
+
+import html
+import quopri
+import re
+from typing import Any
+
+SUBJECT_RE = re.compile(r"^(?P<host>[\w.-]+)\s+Unlisted\b", re.I)
+MACHINE_RE = re.compile(r"machine\s+(?P<host>[\w.-]+)\s*\((?P<machine_id>[a-z0-9]{8,})\)", re.I)
+IMPACT_RE = re.compile(r"Impact:\s*(?P<gpus>\d+)\s*GPU", re.I)
+
+ERROR_START = re.compile(r"Error detected:\s*", re.I)
+ERROR_END = re.compile(r"(Impact:|Common causes|Need to take the machine offline)", re.I)
+
+# Recognisable failure signatures, mapped to what CX should do about them.
+SIGNATURES: list[tuple[str, str, str]] = [
+    (r"memory_remap|uncorrectable remapped memory",
+     "GPU memory remapping failure", "Hardware. Likely RMA - raise with the vendor."),
+    (r"xid", "XID error", "Check dmesg and nvidia-smi; usually a burn-in test before relisting."),
+    (r"gpu_cuda_ok.*expected 1, got 0|cuda initialization|fallen off the bus",
+     "GPU not visible to CUDA", "Check nvidia-smi and the PCIe link; reboot then burn-in."),
+    (r"nvidia-smi.*too many failures", "nvidia-smi failing repeatedly",
+     "Host-level GPU fault - stress test, then RMA if it repeats."),
+    (r"docker (service )?(unresponsive|hung)|container stuck",
+     "Docker daemon unresponsive", "Restart docker; check /var/lib/docker is XFS and mounted."),
+    (r"pod sync (failed|errors)", "Pod sync failing",
+     "Check df -h for a hung moosefs mount and network to the moosefs cluster."),
+    (r"portallocator|public port check fail", "Public port check failing",
+     "Verify the publicIp ports in /etc/runpod/config.json are reachable."),
+    (r"disk|filesystem|xfs", "Disk or filesystem error", "Check dmesg for disk failures."),
+]
+
+
+# Soft line breaks ("=\n") and "=3D" are the giveaways for a quoted-printable
+# body, which is how these mails arrive. Left undecoded, the error block comes
+# out full of "=3D" and split mid-word.
+_QP_HINT = re.compile(r"=\r?\n|=[0-9A-F]{2}")
+
+
+def _maybe_decode_qp(raw: str) -> str:
+    if not _QP_HINT.search(raw):
+        return raw
+    try:
+        return quopri.decodestring(raw.encode("utf-8", "replace")).decode("utf-8", "replace")
+    except Exception:
+        return raw
+
+
+def _text_from_html(raw: str) -> str:
+    text = re.sub(r"<(script|style)[^>]*>.*?</\1>", "", raw, flags=re.S | re.I)
+    text = re.sub(r"<br\s*/?>|</(p|div|tr|li|h[1-6]|table)>", "\n", text, flags=re.I)
+    text = re.sub(r"<[^>]+>", "", text)
+    return html.unescape(text)
+
+
+def classify(error_text: str) -> dict[str, str]:
+    """Name the failure and say what it usually means."""
+    blob = (error_text or "").lower()
+    for pattern, label, action in SIGNATURES:
+        if re.search(pattern, blob, re.I):
+            return {"signature": label, "suggested_action": action}
+    return {"signature": "Unrecognised error",
+            "suggested_action": "Read the error text and escalate if it is not obvious."}
+
+
+def parse(raw: str, subject: str = "") -> dict[str, Any]:
+    """Pull the machine, the error block and the rented-GPU impact out of an email."""
+    raw = _maybe_decode_qp(raw)
+    text = _text_from_html(raw) if "<" in raw and ">" in raw else raw
+    lines = [ln.strip() for ln in text.splitlines()]
+    body = "\n".join(ln for ln in lines if ln)
+
+    host = ""
+    machine_id = ""
+    match = MACHINE_RE.search(body)
+    if match:
+        host, machine_id = match.group("host"), match.group("machine_id")
+    if not host and subject:
+        subject_match = SUBJECT_RE.match(subject.strip())
+        if subject_match:
+            host = subject_match.group("host")
+
+    error_text = ""
+    start = ERROR_START.search(body)
+    if start:
+        rest = body[start.end():]
+        end = ERROR_END.search(rest)
+        error_text = (rest[:end.start()] if end else rest).strip()
+        # The mail repeats its own body; one copy is enough.
+        error_text = "\n".join(dict.fromkeys(ln for ln in error_text.splitlines() if ln.strip() != "---"))
+
+    gpus = None
+    impact = IMPACT_RE.search(body)
+    if impact:
+        gpus = int(impact.group("gpus"))
+
+    return {
+        "host": host,
+        "machine_id": machine_id,
+        "error_text": error_text.strip(),
+        "gpus_rented": gpus,
+        "is_critical": "critical" in (subject or body).lower(),
+        **classify(error_text),
+    }

backend/app/runpod/service.py

@@ -0,0 +1,102 @@
+"""Turning RunPod machine state into tracked history.
+
+The API only ever reports what is true now. Everything CX cares about - how
+often a machine has dropped out, when it drained, who relisted it - only exists
+if each sync writes down what changed.
+"""
+from __future__ import annotations
+
+import datetime as dt
+from typing import Any, Optional
+
+from sqlalchemy import select
+from sqlalchemy.orm import Session
+
+from ..models import RunpodColour, RunpodEvent, RunpodEventType, RunpodHost
+
+
+def record_event(db: Session, host: RunpodHost, event_type: RunpodEventType, *,
+                 actor: str = "system", detail: str = "", error_hint: str = "",
+                 zendesk_ticket: str = "", jira_key: str = "",
+                 gpu_reserved: Optional[int] = None,
+                 payload: Optional[dict[str, Any]] = None) -> RunpodEvent:
+    event = RunpodEvent(
+        event_type=event_type, actor=actor, detail=detail, error_hint=error_hint,
+        zendesk_ticket=zendesk_ticket, jira_key=jira_key,
+        gpu_reserved=host.gpu_reserved if gpu_reserved is None else gpu_reserved,
+        payload=payload,
+    )
+    # Through the relationship so an already-loaded history stays correct.
+    host.events.append(event)
+    db.add(event)
+    return event
+
+
+def sync_machines(db: Session, client: Any, actor: str = "system") -> dict[str, Any]:
+    """Pull current machines and write down every transition since last time."""
+    machines = client.machines()
+    existing = {h.machine_id: h for h in db.scalars(select(RunpodHost)).all()}
+    now = dt.datetime.now(dt.timezone.utc)
+
+    created = unlisted = relisted = drained = 0
+
+    for machine in machines:
+        machine_id = str(machine.get("id") or "")
+        if not machine_id:
+            continue
+        listed = bool(machine.get("listed"))
+        reserved = int(machine.get("gpuReserved") or 0)
+
+        host = existing.get(machine_id)
+        if host is None:
+            host = RunpodHost(machine_id=machine_id, name=str(machine.get("name") or ""))
+            db.add(host)
+            db.flush()
+            created += 1
+            record_event(db, host, RunpodEventType.LISTED if listed else RunpodEventType.UNLISTED,
+                         actor="runpod", detail="First seen by CX Triage")
+            if not listed:
+                host.unlisted_at = now
+
+        was_listed = host.listed
+        was_reserved = host.gpu_reserved
+
+        if was_listed and not listed:
+            unlisted += 1
+            host.unlisted_at = now
+            host.unlist_count += 1
+            # Nothing in the API says why; the email carries the reason and is
+            # attached separately through /ingest-email.
+            record_event(db, host, RunpodEventType.UNLISTED, actor="runpod",
+                         detail="Unlisted (detected on sync)", gpu_reserved=reserved)
+        elif not was_listed and listed:
+            relisted += 1
+            host.last_listed_at = now
+            host.unlisted_at = None
+            record_event(db, host, RunpodEventType.LISTED, actor=actor,
+                         detail="Relisted", gpu_reserved=reserved)
+
+        # Unlisted and the last renter has gone: the machine is now safe to work on.
+        if not listed and was_reserved > 0 and reserved == 0:
+            drained += 1
+            record_event(db, host, RunpodEventType.DRAINED, actor="runpod",
+                         detail="Unlisted machine has drained - no rented GPUs left",
+                         gpu_reserved=0)
+
+        gpu_type = machine.get("gpuType") or {}
+        host.name = str(machine.get("name") or host.name)
+        host.listed = listed
+        host.gpu_reserved = reserved
+        host.gpu_total = int(machine.get("gpuTotal") or 0)
+        host.gpu_type = str(gpu_type.get("displayName") or machine.get("gpuTypeId") or "")
+        host.data_center = str(machine.get("dataCenterId") or "")
+        host.uptime_one_week = machine.get("uptimePercentListedOneWeek")
+        host.maintenance_start = str(machine.get("maintenanceStart") or "")
+        host.maintenance_end = str(machine.get("maintenanceEnd") or "")
+
+    db.commit()
+    return {
+        "machines": len(machines), "created": created, "unlisted": unlisted,
+        "relisted": relisted, "drained": drained,
+        "synced_at": now.isoformat(),
+    }

backend/app/seed/data.py

@@ -0,0 +1,174 @@
+"""Seed data drawn from the real exports, so a fresh stack is worth looking at.
+
+The handover comes from APAC 06 August 2026; the RunPod hosts come from the
+monitoring script's own JSON. Only enough is transcribed to make the pages
+representative - it is sample data, not a migration.
+"""
+from __future__ import annotations
+
+HANDOVER = {
+    "shift_date": "2026-08-06",
+    "shift": "APAC",
+    "handing_to": "EMEA",
+    "team_members": "Parham",
+    "significant_issues_checked": True,
+    "hubspot_checked": True,
+    "total_open_tickets": 214,
+    "member_checks": [
+        {"name": "Prasad", "hs_checked": False, "jira_checked": False},
+        {"name": "Parham", "hs_checked": True, "jira_checked": True},
+    ],
+    "other_comments": (
+        "Weekend Coverage Annual Leave Sheet is on SharePoint.\n\n"
+        "Confluence Sanity Check: going through current docs to see what needs updating "
+        "and removing.\n\n"
+        "Kodekloud Training: more team time needed on Kubernetes CKNA. Team leads are "
+        "working on a schedule.\n\n"
+        "New runbook WIP to protect customers and infrastructure from suspicious activity - "
+        "locking a VM and, if required, detaching a public IP allocation.\n\n"
+        "For any Windmill flows that require a ticket reference, use the numeric number and "
+        "not the URL for now.\n\n"
+        "Check the Zendesk sync tool - some Hyperstack orgs are not showing in Zendesk even "
+        "though they exist in InfraInsight (e.g. org 29084, 29548)."
+    ),
+    "items": [
+        {
+            "title": "Request to detach volume from hibernated VM",
+            "zendesk_tickets": "#8495",
+            "state": "no_further_engagement",
+            "remove_at_end_of_shift": True,
+            "body": ("Details confirmed. Ran the Windmill flow but hit an error on API keys. "
+                     "Volume has since been detached successfully and the customer has been informed."),
+            "links": ["https://windmill.ngbackend.cloud/run/019fd28c-662b-3b56-199a-6af73bb1993c"],
+        },
+        {
+            "title": "Baseten node-level failure on b200-worker-frank-quetzal",
+            "zendesk_tickets": "#8453",
+            "state": "pending_infra",
+            "body": ("Aranya reached out on behalf of Baseten about a node-level failure "
+                     "(internal IP 10.32.10.2). Triage provided three times; the first two uploads "
+                     "showed no issues. Customer still reports the node is wedged. Waiting on the "
+                     "third upload for review. Asked Infra whether the nodes just need a reboot."),
+            "links": ["https://nexgen-cloud.slack.com/archives/C0AP4KV1XT5/p1785827125227079"],
+        },
+        {
+            "title": "[NO1] Investigate inbound traffic spike",
+            "zendesk_tickets": "8238",
+            "jira_key": "OIE-3213", "jira_status": "Complete",
+            "state": "pending_customer",
+            "body": ("CARMA Media Insight (org 3429) reported a transatlantic throughput collapse - "
+                     "EU fine, North America down to ~45-141 KB/s. Kheano traced it to edge-level "
+                     "congestion in NO1 affecting all tenants, top talker FIP 149.36.0.199 showing "
+                     "signs of a flood attack. Infra have confirmed WAN utilisation has normalised."),
+            "links": ["https://nexgen-cloud.slack.com/archives/C049Q9JRGM7/p1785526451626789"],
+        },
+        {
+            "title": "[EU1-BM2] Reported network latency investigation",
+            "zendesk_tickets": "#8409",
+            "jira_key": "OIE-3185", "jira_status": "In Progress",
+            "state": "pending_infra",
+            "body": "Baseten experienced a latency spike on 31 July and have requested investigation.",
+            "links": ["https://nexgen-cloud.slack.com/archives/C0AP4KV1XT5/p1784855398156009"],
+        },
+        {
+            "title": "Network storage speed",
+            "zendesk_tickets": "#8207",
+            "jira_key": "OIE-3210", "jira_status": "In Progress",
+            "state": "pending_infra",
+            "body": ("RunPod advised one of their machines is reporting network slowness to their "
+                     "storage. Confirmed against another machine in the same region using the same "
+                     "storage by creating files of random data."),
+            "links": ["https://nexgen-cloud.slack.com/archives/C049Q9JRGM7/p1785451129222409"],
+        },
+        {
+            "title": "[EU1-BM2] GPU issues on eu1-bm2-lv1-b200sxm-0(16|55)",
+            "zendesk_tickets": "#7772",
+            "jira_key": "OIE-3177", "jira_status": "Pending",
+            "state": "pending_infra",
+            "body": ("Baseten requested support on 2 nodes with XID errors. One looks transient; "
+                     "eu1-bm2-lv1-b200sxm-055 is more concerning with UECC. Raised with Lenovo for "
+                     "next steps."),
+            "links": ["https://nexgen-cloud.slack.com/archives/C0AP4KV1XT5/p1784749263687239"],
+        },
+        {
+            "title": "Shadeform locked VM - billing adjustment",
+            "zendesk_tickets": "#8215",
+            "state": "no_further_engagement",
+            "remove_at_end_of_shift": True,
+            "body": ("DevOps raised a ticket to notify the user their VM was locked (945280 / "
+                     "O6QTQyQgqDRN, Shadeform) due to unusual activity. Customer asked for it to be "
+                     "deleted; we unlocked and they deleted it. Locked for 5 days 3 hours 21 minutes "
+                     "= 7401 minutes = 123.35 hours at $1.6756/hr. Total credit $206.69 to come off "
+                     "the next invoice, since the state was never changed to shutoff/locked."),
+        },
+        {
+            "title": "185.216.20.188 network slowness - CA1-SRV-CPU6",
+            "zendesk_tickets": "#7793",
+            "jira_key": "OIE-3189", "jira_status": "Done",
+            "state": "pending_customer",
+            "body": ("Customer reported intermittent network instability on VM 298953 "
+                     "(CA1-SRV-CPU6, 185.216.20.188). The problematic VM 924706 / JqXcTaGOt has been "
+                     "siloed by Infra and the routers investigated. Network has been stable for the "
+                     "past couple of days."),
+        },
+        {
+            "title": "VM stuck - CA1-ESC8-068 host not reachable",
+            "zendesk_tickets": "#8078, #8076, #7787, #8077",
+            "jira_key": "OIE-3183", "jira_status": "In Progress",
+            "state": "pending_customer",
+            "body": ("Host maintenance scheduled 06 August 2026 at 00:00 UTC. Communication sent to "
+                     "the customer. Four VMs still on the host and reachable: vm-regA (786651), "
+                     "dubbix_instance_1 (737376), gianpaolo-dev-2 (602721), triton-backup-2 (213340). "
+                     "Follow-up emails sent. Pending customer reply on the maintenance window."),
+        },
+    ],
+}
+
+# The colour key from the handover doc, so the UI can explain itself.
+RUNPOD_COLOURS = {
+    "red": "Blocked from being relisted due to recurring issues. Needs in-depth investigation or is tied to a system issue.",
+    "purple": "Pending RunPod. We are waiting on RunPod for something.",
+    "yellow": "Unactionable by CX. Pending investigation or remediation from Infrastructure or the DC team.",
+    "blue": "Unactionable by CX. Pending removal from the RunPod platform.",
+    "green": "Stress testing for >24 hours. GPUs may appear reserved.",
+    "white": "Actionable by CX.",
+}
+
+# The unlisted table from the same handover, with the states CX had recorded.
+RUNPOD_BOARD = [
+    {"name": "no1-os1-5090-016-contract-001", "machine_id": "hedh664udd3g", "zendesk_ticket": "8405",
+     "colour": "green", "last_error": "std burn in failed on 03 Aug",
+     "next_steps": "24 Hr default burn-in test in progress. 05 Aug 2026 15:06 UTC (16:06 BST)"},
+    {"name": "no1-os1-5090-099-contract-001", "machine_id": "9bpfxa1o9our", "zendesk_ticket": "6279",
+     "colour": "red", "jira_key": "RMA-103", "jira_status": "Waiting",
+     "last_error": "nvidia-smi: too many failures. Raised for RMA",
+     "next_steps": "24 HR burn-in started 05 Aug 2026 18:38 UTC. Luis asked for multiple 24hr stress tests with full logs in Jira."},
+    {"name": "no1-os1-5090-027-contract-001", "machine_id": "seed-5090-027", "zendesk_ticket": "#8385",
+     "colour": "green", "last_error": "nvidia-smi many failures. First burn-in failed so running again",
+     "next_steps": "24HR burn-in running, started 05 Aug 2026 20:15 UTC (21:15 BST)"},
+    {"name": "no1-os1-5090-050", "machine_id": "gpc1q3vboxfo", "zendesk_ticket": "#8497",
+     "colour": "green",
+     "last_error": "dcgm-xid-check: potential XID issue detected\ngpu health check failed: metric gpu_cuda_ok: expected 1, got 0",
+     "next_steps": "24HR burn-in running, started 05 Aug 2026 17:28 UTC (18:28 BST)"},
+    {"name": "no1-os1-4090-009", "machine_id": "f2vwayvyxorv", "zendesk_ticket": "#7342",
+     "colour": "yellow", "last_error": "nvidia-smi: too many failures. 2/8 GPU in use",
+     "next_steps": "Maintenance scheduled 4 Aug 2026 19:00 UTC. Host has connectivity issues - may need escalating."},
+    {"name": "no1-os1-4090-020", "machine_id": "s7kl180cusjr", "zendesk_ticket": "#7359",
+     "colour": "yellow", "last_error": "pod sync failed 16 times. 2/8 GPU in use",
+     "next_steps": "Maintenance scheduled 04 Aug 2026 13:02 UTC for 1 day"},
+    {"name": "ca1-esc8-106", "machine_id": "x0gn8v2rthk4", "zendesk_ticket": "419195749569",
+     "colour": "red", "jira_status": "Under Test",
+     "last_error": ("dcgm-xid-check: potential XID issue detected\n"
+                    "gpu health check failed: error indicator present: "
+                    "gpu_failed{reason=memory_remap,uuid=GPU-d29f591c-63e9-5f1d-add1-ed0e2dad3660}"),
+     "next_steps": "Repeated offender - does not stay listed for more than 48 hours. Booked maintenance 17 Jun 2026 14:52 BST."},
+    {"name": "ca1-esc8-121", "machine_id": "seed-esc8-121", "zendesk_ticket": "419270908096",
+     "colour": "yellow", "jira_status": "Maintenance Scheduled",
+     "last_error": ("dcgm-xid-check: potential XID issue detected\n"
+                    "gpu health check failed: metric gpu_cuda_ok: expected 1, got 0"),
+     "next_steps": "3/8 in use. Maintenance 18 Jun 2026 19:00 UTC (20:00 BST)"},
+    {"name": "no1-os1-4090-013", "machine_id": "seed-4090-013", "zendesk_ticket": "418292796638",
+     "colour": "blue", "jira_key": "RMA-60", "jira_status": "Pending RMA",
+     "last_error": "container stuck: docker service unresponsive",
+     "next_steps": "Reboot failed, received timeout error. Pending RMA."},
+]

backend/app/seed/loader.py

@@ -0,0 +1,150 @@
+"""Populate an empty database with representative data.
+
+Runs on startup when CX_SEED_DEMO is on, and only when the relevant table is
+empty, so it never overwrites real work.
+"""
+from __future__ import annotations
+
+import datetime as dt
+import json
+import os
+import random
+from pathlib import Path
+
+from sqlalchemy import select
+from sqlalchemy.orm import Session
+
+from ..models import (AppSetting, Handover, HandoverItem, HandoverStatus, ItemState,
+                      RunpodColour, RunpodEventType, RunpodHost, ShiftName)
+from ..runpod.service import record_event
+from .data import HANDOVER, RUNPOD_BOARD, RUNPOD_COLOURS
+
+# Optional: the monitoring script's own exports, if they have been mounted in.
+RUNPOD_EXPORT_DIR = os.environ.get("CX_RUNPOD_EXPORT_DIR", "/seed/runpod")
+
+
+def seed_handover(db: Session) -> str:
+    if db.scalar(select(Handover).limit(1)):
+        return "handover: already present, left alone"
+
+    row = Handover(
+        shift_date=dt.date.fromisoformat(HANDOVER["shift_date"]),
+        shift=ShiftName(HANDOVER["shift"]),
+        handing_to=HANDOVER["handing_to"],
+        team_members=HANDOVER["team_members"],
+        significant_issues_checked=HANDOVER["significant_issues_checked"],
+        hubspot_checked=HANDOVER["hubspot_checked"],
+        total_open_tickets=HANDOVER["total_open_tickets"],
+        member_checks=HANDOVER["member_checks"],
+        other_comments=HANDOVER["other_comments"],
+        status=HandoverStatus.DRAFT,
+    )
+    for position, item in enumerate(HANDOVER["items"]):
+        row.items.append(HandoverItem(
+            position=position, title=item["title"],
+            zendesk_tickets=item.get("zendesk_tickets", ""),
+            jira_key=item.get("jira_key", ""), jira_status=item.get("jira_status", ""),
+            body=item.get("body", ""), links=item.get("links", []),
+            state=ItemState(item.get("state", "in_progress")),
+            remove_at_end_of_shift=item.get("remove_at_end_of_shift", False),
+            first_raised_on=row.shift_date,
+        ))
+    db.add(row)
+    db.commit()
+    return f"handover: seeded {row.shift.value} {row.shift_date} with {len(row.items)} items"
+
+
+def _load_export(name: str):
+    path = Path(RUNPOD_EXPORT_DIR) / name
+    if not path.is_file():
+        return None
+    try:
+        return json.loads(path.read_text())
+    except (OSError, json.JSONDecodeError):
+        return None
+
+
+def seed_runpod(db: Session) -> str:
+    if db.scalar(select(RunpodHost).limit(1)):
+        return "runpod: already present, left alone"
+
+    now = dt.datetime.now(dt.timezone.utc)
+    rng = random.Random(20260806)   # deterministic, so the demo looks the same each time
+    hosts: dict[str, RunpodHost] = {}
+
+    # 1. The board rows from the handover, with their CX state.
+    for entry in RUNPOD_BOARD:
+        host = RunpodHost(
+            machine_id=entry["machine_id"], name=entry["name"], listed=False,
+            gpu_reserved=0, gpu_total=8, gpu_type="RTX 5090" if "5090" in entry["name"] else "RTX 4090",
+            data_center="NO1" if entry["name"].startswith("no1") else "CA1",
+            colour=RunpodColour(entry.get("colour", "white")),
+            zendesk_ticket=entry.get("zendesk_ticket", ""),
+            jira_key=entry.get("jira_key", ""), jira_status=entry.get("jira_status", ""),
+            last_error=entry.get("last_error", ""), next_steps=entry.get("next_steps", ""),
+            unlisted_at=now - dt.timedelta(hours=rng.randint(6, 96)),
+        )
+        db.add(host)
+        db.flush()
+        hosts[host.machine_id] = host
+
+    # 2. Anything else the monitoring script had recorded.
+    hosts_db = _load_export("hosts_db.json") or {}
+    jira_db = _load_export("jira_db.json") or {}
+    for machine_id, record in list(hosts_db.items())[:200]:
+        if machine_id in hosts:
+            continue
+        jira = jira_db.get(machine_id, {})
+        host = RunpodHost(
+            machine_id=machine_id, name=str(record.get("name") or ""),
+            listed=bool(record.get("listed", True)),
+            gpu_reserved=int(record.get("gpuReserved") or 0), gpu_total=8,
+            historic_count=int(jira.get("historic_count") or 0),
+            jira_key=jira.get("jira_issue_key") or "", jira_status=jira.get("jira_status") or "",
+        )
+        if not host.listed:
+            host.unlisted_at = now - dt.timedelta(hours=rng.randint(2, 240))
+        db.add(host)
+        db.flush()
+        hosts[machine_id] = host
+
+    # 3. A plausible history, so the timeline and ranking have something to show.
+    for host in hosts.values():
+        rounds = max(host.historic_count, 1 if not host.listed else 0)
+        cursor = now - dt.timedelta(days=min(45, 3 + rounds * 4))
+        for _ in range(min(rounds, 12)):
+            cursor += dt.timedelta(hours=rng.randint(8, 72))
+            if cursor >= now:
+                break
+            record_event(db, host, RunpodEventType.UNLISTED, actor="runpod",
+                         detail="Unlisted automatically after a critical error",
+                         error_hint=host.last_error or "gpu health check failed",
+                         gpu_reserved=rng.choice([0, 1, 2, 8]))
+            host.events[-1].occurred_at = cursor
+            host.unlist_count += 1
+
+            cursor += dt.timedelta(hours=rng.randint(4, 48))
+            if cursor >= now or not host.listed:
+                continue
+            operator = rng.choice(["parham.monfared@nexgencloud.com", "luis.sarabando@nexgencloud.com",
+                                   "kheano.martinez@nexgencloud.com"])
+            record_event(db, host, RunpodEventType.LISTED, actor=operator,
+                         detail="Relisted after burn-in passed",
+                         zendesk_ticket=host.zendesk_ticket)
+            host.events[-1].occurred_at = cursor
+
+        if not host.listed and host.last_error:
+            record_event(db, host, RunpodEventType.UNLISTED, actor="runpod",
+                         detail="Current unlisting", error_hint=host.last_error,
+                         zendesk_ticket=host.zendesk_ticket, jira_key=host.jira_key)
+            host.events[-1].occurred_at = host.unlisted_at or now
+            host.unlist_count += 1
+
+    db.merge(AppSetting(key="runpod_colours", value=RUNPOD_COLOURS))
+    db.commit()
+    unlisted = sum(1 for h in hosts.values() if not h.listed)
+    return f"runpod: seeded {len(hosts)} machines ({unlisted} unlisted) with history"
+
+
+def run(db: Session) -> list[str]:
+    return [seed_handover(db), seed_runpod(db)]