parham.monfared/cx-ui
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add shift handover and RunPod, and make CX-Tools work in a container
Some checks failed
build-and-deploy / test (push) Has been cancelled
Details
build-and-deploy / image (push) Has been cancelled
Details
build-and-deploy / deploy (push) Has been cancelled
Details

Browse Source 
Handover
- The Confluence shift doc becomes the landing page: shift metadata, the
  top-of-page checks, key updates with their Zendesk/Jira refs and status, and
  the free-text comments. "Hand over shift" closes the shift, opens the next
  one and carries the live items across, dropping anything done or marked
  "remove at end of shift" - the retyping this replaces.
- The RunPod table on that page is read from live host state instead of being
  copied in by hand, with the six-colour key preserved.

RunPod
- GraphQL client keyed on CX_RUNPOD_API_KEY. The old console login is kept as a
  fallback but cannot run unattended: the account has 2FA, so Clerk verifies the
  password and then asks for an emailed code and never issues a session. That is
  the real cause of the "No active session found" failure, and the client now
  says so instead of failing opaquely. TOTP is supported if the account moves to
  an authenticator app.
- Hosts and their listing history are persisted, so "most problematic hosts" can
  be ranked and each machine has a timeline of who listed or unlisted it, with
  the Zendesk comment and the error hint.
- The unlisting emails are parsed for the error block (they arrive
  quoted-printable) and classified into a likely cause and a next step.

Zendesk and Jira
- Unlisting raises a Zendesk ticket that follows the format of RunPod's own
  email, keyed on the machine so one machine keeps one thread, posted as an
  internal note.
- Jira is split in two: the Infrahub/OIE instance and the RunPod/RMA one, which
  may be a different Atlassian site. Blank RunPod values fall back to the
  defaults rather than failing.

Running in a container
- CX-Tools reads its keys from 1Password, which needs a desktop app. Config is a
  dataclass whose lookups live in per-field default factories, so passing
  CX_INFRAHUB_TOKEN/CX_INFRAINSIGHT_TOKEN in means those factories never run and
  CX-Tools itself stays unmodified.
- CX-Tools reaches OpenStack with `docker exec <region>-osc`, so the image now
  carries the Docker client (the static binary, not the docker.io package) and
  compose mounts the host socket with group_add for it. Verified from inside the
  container: live OpenStack and Infrahub calls both succeed.

Also fixes Settings, which read the environment at class-definition time and so
ignored anything set afterwards; a fresh Settings() silently returned stale
values. Caught by the Jira scoping tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Parham Monfared
2026-08-06 08:21:59 +01:00
parent 1262690276
commit 8892144e0a
24 changed files with 4576 additions and 71 deletions
Download Patch File Download Diff File Expand all files Collapse all files

0
backend/app/seed/__init__.py Normal file
View File

174
backend/app/seed/data.py Normal file
View File

@@ -0,0 +1,174 @@
"""Seed data drawn from the real exports, so a fresh stack is worth looking at.
The handover comes from APAC 06 August 2026; the RunPod hosts come from the
monitoring script's own JSON. Only enough is transcribed to make the pages
representative - it is sample data, not a migration.
"""
from __future__ import annotations
HANDOVER = {
"shift_date": "2026-08-06",
"shift": "APAC",
"handing_to": "EMEA",
"team_members": "Parham",
"significant_issues_checked": True,
"hubspot_checked": True,
"total_open_tickets": 214,
"member_checks": [
{"name": "Prasad", "hs_checked": False, "jira_checked": False},
{"name": "Parham", "hs_checked": True, "jira_checked": True},
],
"other_comments": (
"Weekend Coverage Annual Leave Sheet is on SharePoint.\n\n"
"Confluence Sanity Check: going through current docs to see what needs updating "
"and removing.\n\n"
"Kodekloud Training: more team time needed on Kubernetes CKNA. Team leads are "
"working on a schedule.\n\n"
"New runbook WIP to protect customers and infrastructure from suspicious activity - "
"locking a VM and, if required, detaching a public IP allocation.\n\n"
"For any Windmill flows that require a ticket reference, use the numeric number and "
"not the URL for now.\n\n"
"Check the Zendesk sync tool - some Hyperstack orgs are not showing in Zendesk even "
"though they exist in InfraInsight (e.g. org 29084, 29548)."
),
"items": [
{
"title": "Request to detach volume from hibernated VM",
"zendesk_tickets": "#8495",
"state": "no_further_engagement",
"remove_at_end_of_shift": True,
"body": ("Details confirmed. Ran the Windmill flow but hit an error on API keys. "
"Volume has since been detached successfully and the customer has been informed."),
"links": ["https://windmill.ngbackend.cloud/run/019fd28c-662b-3b56-199a-6af73bb1993c"],
},
{
"title": "Baseten node-level failure on b200-worker-frank-quetzal",
"zendesk_tickets": "#8453",
"state": "pending_infra",
"body": ("Aranya reached out on behalf of Baseten about a node-level failure "
"(internal IP 10.32.10.2). Triage provided three times; the first two uploads "
"showed no issues. Customer still reports the node is wedged. Waiting on the "
"third upload for review. Asked Infra whether the nodes just need a reboot."),
"links": ["https://nexgen-cloud.slack.com/archives/C0AP4KV1XT5/p1785827125227079"],
},
{
"title": "[NO1] Investigate inbound traffic spike",
"zendesk_tickets": "8238",
"jira_key": "OIE-3213", "jira_status": "Complete",
"state": "pending_customer",
"body": ("CARMA Media Insight (org 3429) reported a transatlantic throughput collapse - "
"EU fine, North America down to ~45-141 KB/s. Kheano traced it to edge-level "
"congestion in NO1 affecting all tenants, top talker FIP 149.36.0.199 showing "
"signs of a flood attack. Infra have confirmed WAN utilisation has normalised."),
"links": ["https://nexgen-cloud.slack.com/archives/C049Q9JRGM7/p1785526451626789"],
},
{
"title": "[EU1-BM2] Reported network latency investigation",
"zendesk_tickets": "#8409",
"jira_key": "OIE-3185", "jira_status": "In Progress",
"state": "pending_infra",
"body": "Baseten experienced a latency spike on 31 July and have requested investigation.",
"links": ["https://nexgen-cloud.slack.com/archives/C0AP4KV1XT5/p1784855398156009"],
},
{
"title": "Network storage speed",
"zendesk_tickets": "#8207",
"jira_key": "OIE-3210", "jira_status": "In Progress",
"state": "pending_infra",
"body": ("RunPod advised one of their machines is reporting network slowness to their "
"storage. Confirmed against another machine in the same region using the same "
"storage by creating files of random data."),
"links": ["https://nexgen-cloud.slack.com/archives/C049Q9JRGM7/p1785451129222409"],
},
{
"title": "[EU1-BM2] GPU issues on eu1-bm2-lv1-b200sxm-0(16|55)",
"zendesk_tickets": "#7772",
"jira_key": "OIE-3177", "jira_status": "Pending",
"state": "pending_infra",
"body": ("Baseten requested support on 2 nodes with XID errors. One looks transient; "
"eu1-bm2-lv1-b200sxm-055 is more concerning with UECC. Raised with Lenovo for "
"next steps."),
"links": ["https://nexgen-cloud.slack.com/archives/C0AP4KV1XT5/p1784749263687239"],
},
{
"title": "Shadeform locked VM - billing adjustment",
"zendesk_tickets": "#8215",
"state": "no_further_engagement",
"remove_at_end_of_shift": True,
"body": ("DevOps raised a ticket to notify the user their VM was locked (945280 / "
"O6QTQyQgqDRN, Shadeform) due to unusual activity. Customer asked for it to be "
"deleted; we unlocked and they deleted it. Locked for 5 days 3 hours 21 minutes "
"= 7401 minutes = 123.35 hours at $1.6756/hr. Total credit $206.69 to come off "
"the next invoice, since the state was never changed to shutoff/locked."),
},
{
"title": "185.216.20.188 network slowness - CA1-SRV-CPU6",
"zendesk_tickets": "#7793",
"jira_key": "OIE-3189", "jira_status": "Done",
"state": "pending_customer",
"body": ("Customer reported intermittent network instability on VM 298953 "
"(CA1-SRV-CPU6, 185.216.20.188). The problematic VM 924706 / JqXcTaGOt has been "
"siloed by Infra and the routers investigated. Network has been stable for the "
"past couple of days."),
},
{
"title": "VM stuck - CA1-ESC8-068 host not reachable",
"zendesk_tickets": "#8078, #8076, #7787, #8077",
"jira_key": "OIE-3183", "jira_status": "In Progress",
"state": "pending_customer",
"body": ("Host maintenance scheduled 06 August 2026 at 00:00 UTC. Communication sent to "
"the customer. Four VMs still on the host and reachable: vm-regA (786651), "
"dubbix_instance_1 (737376), gianpaolo-dev-2 (602721), triton-backup-2 (213340). "
"Follow-up emails sent. Pending customer reply on the maintenance window."),
},
],
}
# The colour key from the handover doc, so the UI can explain itself.
RUNPOD_COLOURS = {
"red": "Blocked from being relisted due to recurring issues. Needs in-depth investigation or is tied to a system issue.",
"purple": "Pending RunPod. We are waiting on RunPod for something.",
"yellow": "Unactionable by CX. Pending investigation or remediation from Infrastructure or the DC team.",
"blue": "Unactionable by CX. Pending removal from the RunPod platform.",
"green": "Stress testing for >24 hours. GPUs may appear reserved.",
"white": "Actionable by CX.",
}
# The unlisted table from the same handover, with the states CX had recorded.
RUNPOD_BOARD = [
{"name": "no1-os1-5090-016-contract-001", "machine_id": "hedh664udd3g", "zendesk_ticket": "8405",
"colour": "green", "last_error": "std burn in failed on 03 Aug",
"next_steps": "24 Hr default burn-in test in progress. 05 Aug 2026 15:06 UTC (16:06 BST)"},
{"name": "no1-os1-5090-099-contract-001", "machine_id": "9bpfxa1o9our", "zendesk_ticket": "6279",
"colour": "red", "jira_key": "RMA-103", "jira_status": "Waiting",
"last_error": "nvidia-smi: too many failures. Raised for RMA",
"next_steps": "24 HR burn-in started 05 Aug 2026 18:38 UTC. Luis asked for multiple 24hr stress tests with full logs in Jira."},
{"name": "no1-os1-5090-027-contract-001", "machine_id": "seed-5090-027", "zendesk_ticket": "#8385",
"colour": "green", "last_error": "nvidia-smi many failures. First burn-in failed so running again",
"next_steps": "24HR burn-in running, started 05 Aug 2026 20:15 UTC (21:15 BST)"},
{"name": "no1-os1-5090-050", "machine_id": "gpc1q3vboxfo", "zendesk_ticket": "#8497",
"colour": "green",
"last_error": "dcgm-xid-check: potential XID issue detected\ngpu health check failed: metric gpu_cuda_ok: expected 1, got 0",
"next_steps": "24HR burn-in running, started 05 Aug 2026 17:28 UTC (18:28 BST)"},
{"name": "no1-os1-4090-009", "machine_id": "f2vwayvyxorv", "zendesk_ticket": "#7342",
"colour": "yellow", "last_error": "nvidia-smi: too many failures. 2/8 GPU in use",
"next_steps": "Maintenance scheduled 4 Aug 2026 19:00 UTC. Host has connectivity issues - may need escalating."},
{"name": "no1-os1-4090-020", "machine_id": "s7kl180cusjr", "zendesk_ticket": "#7359",
"colour": "yellow", "last_error": "pod sync failed 16 times. 2/8 GPU in use",
"next_steps": "Maintenance scheduled 04 Aug 2026 13:02 UTC for 1 day"},
{"name": "ca1-esc8-106", "machine_id": "x0gn8v2rthk4", "zendesk_ticket": "419195749569",
"colour": "red", "jira_status": "Under Test",
"last_error": ("dcgm-xid-check: potential XID issue detected\n"
"gpu health check failed: error indicator present: "
"gpu_failed{reason=memory_remap,uuid=GPU-d29f591c-63e9-5f1d-add1-ed0e2dad3660}"),
"next_steps": "Repeated offender - does not stay listed for more than 48 hours. Booked maintenance 17 Jun 2026 14:52 BST."},
{"name": "ca1-esc8-121", "machine_id": "seed-esc8-121", "zendesk_ticket": "419270908096",
"colour": "yellow", "jira_status": "Maintenance Scheduled",
"last_error": ("dcgm-xid-check: potential XID issue detected\n"
"gpu health check failed: metric gpu_cuda_ok: expected 1, got 0"),
"next_steps": "3/8 in use. Maintenance 18 Jun 2026 19:00 UTC (20:00 BST)"},
{"name": "no1-os1-4090-013", "machine_id": "seed-4090-013", "zendesk_ticket": "418292796638",
"colour": "blue", "jira_key": "RMA-60", "jira_status": "Pending RMA",
"last_error": "container stuck: docker service unresponsive",
"next_steps": "Reboot failed, received timeout error. Pending RMA."},
]

150
backend/app/seed/loader.py Normal file
View File

@@ -0,0 +1,150 @@
"""Populate an empty database with representative data.
Runs on startup when CX_SEED_DEMO is on, and only when the relevant table is
empty, so it never overwrites real work.
"""
from __future__ import annotations
import datetime as dt
import json
import os
import random
from pathlib import Path
from sqlalchemy import select
from sqlalchemy.orm import Session
from ..models import (AppSetting, Handover, HandoverItem, HandoverStatus, ItemState,
RunpodColour, RunpodEventType, RunpodHost, ShiftName)
from ..runpod.service import record_event
from .data import HANDOVER, RUNPOD_BOARD, RUNPOD_COLOURS
# Optional: the monitoring script's own exports, if they have been mounted in.
RUNPOD_EXPORT_DIR = os.environ.get("CX_RUNPOD_EXPORT_DIR", "/seed/runpod")
def seed_handover(db: Session) -> str:
if db.scalar(select(Handover).limit(1)):
return "handover: already present, left alone"
row = Handover(
shift_date=dt.date.fromisoformat(HANDOVER["shift_date"]),
shift=ShiftName(HANDOVER["shift"]),
handing_to=HANDOVER["handing_to"],
team_members=HANDOVER["team_members"],
significant_issues_checked=HANDOVER["significant_issues_checked"],
hubspot_checked=HANDOVER["hubspot_checked"],
total_open_tickets=HANDOVER["total_open_tickets"],
member_checks=HANDOVER["member_checks"],
other_comments=HANDOVER["other_comments"],
status=HandoverStatus.DRAFT,
)
for position, item in enumerate(HANDOVER["items"]):
row.items.append(HandoverItem(
position=position, title=item["title"],
zendesk_tickets=item.get("zendesk_tickets", ""),
jira_key=item.get("jira_key", ""), jira_status=item.get("jira_status", ""),
body=item.get("body", ""), links=item.get("links", []),
state=ItemState(item.get("state", "in_progress")),
remove_at_end_of_shift=item.get("remove_at_end_of_shift", False),
first_raised_on=row.shift_date,
))
db.add(row)
db.commit()
return f"handover: seeded {row.shift.value} {row.shift_date} with {len(row.items)} items"
def _load_export(name: str):
path = Path(RUNPOD_EXPORT_DIR) / name
if not path.is_file():
return None
try:
return json.loads(path.read_text())
except (OSError, json.JSONDecodeError):
return None
def seed_runpod(db: Session) -> str:
if db.scalar(select(RunpodHost).limit(1)):
return "runpod: already present, left alone"
now = dt.datetime.now(dt.timezone.utc)
rng = random.Random(20260806) # deterministic, so the demo looks the same each time
hosts: dict[str, RunpodHost] = {}
# 1. The board rows from the handover, with their CX state.
for entry in RUNPOD_BOARD:
host = RunpodHost(
machine_id=entry["machine_id"], name=entry["name"], listed=False,
gpu_reserved=0, gpu_total=8, gpu_type="RTX 5090" if "5090" in entry["name"] else "RTX 4090",
data_center="NO1" if entry["name"].startswith("no1") else "CA1",
colour=RunpodColour(entry.get("colour", "white")),
zendesk_ticket=entry.get("zendesk_ticket", ""),
jira_key=entry.get("jira_key", ""), jira_status=entry.get("jira_status", ""),
last_error=entry.get("last_error", ""), next_steps=entry.get("next_steps", ""),
unlisted_at=now - dt.timedelta(hours=rng.randint(6, 96)),
)
db.add(host)
db.flush()
hosts[host.machine_id] = host
# 2. Anything else the monitoring script had recorded.
hosts_db = _load_export("hosts_db.json") or {}
jira_db = _load_export("jira_db.json") or {}
for machine_id, record in list(hosts_db.items())[:200]:
if machine_id in hosts:
continue
jira = jira_db.get(machine_id, {})
host = RunpodHost(
machine_id=machine_id, name=str(record.get("name") or ""),
listed=bool(record.get("listed", True)),
gpu_reserved=int(record.get("gpuReserved") or 0), gpu_total=8,
historic_count=int(jira.get("historic_count") or 0),
jira_key=jira.get("jira_issue_key") or "", jira_status=jira.get("jira_status") or "",
)
if not host.listed:
host.unlisted_at = now - dt.timedelta(hours=rng.randint(2, 240))
db.add(host)
db.flush()
hosts[machine_id] = host
# 3. A plausible history, so the timeline and ranking have something to show.
for host in hosts.values():
rounds = max(host.historic_count, 1 if not host.listed else 0)
cursor = now - dt.timedelta(days=min(45, 3 + rounds * 4))
for _ in range(min(rounds, 12)):
cursor += dt.timedelta(hours=rng.randint(8, 72))
if cursor >= now:
break
record_event(db, host, RunpodEventType.UNLISTED, actor="runpod",
detail="Unlisted automatically after a critical error",
error_hint=host.last_error or "gpu health check failed",
gpu_reserved=rng.choice([0, 1, 2, 8]))
host.events[-1].occurred_at = cursor
host.unlist_count += 1
cursor += dt.timedelta(hours=rng.randint(4, 48))
if cursor >= now or not host.listed:
continue
operator = rng.choice(["parham.monfared@nexgencloud.com", "luis.sarabando@nexgencloud.com",
"kheano.martinez@nexgencloud.com"])
record_event(db, host, RunpodEventType.LISTED, actor=operator,
detail="Relisted after burn-in passed",
zendesk_ticket=host.zendesk_ticket)
host.events[-1].occurred_at = cursor
if not host.listed and host.last_error:
record_event(db, host, RunpodEventType.UNLISTED, actor="runpod",
detail="Current unlisting", error_hint=host.last_error,
zendesk_ticket=host.zendesk_ticket, jira_key=host.jira_key)
host.events[-1].occurred_at = host.unlisted_at or now
host.unlist_count += 1
db.merge(AppSetting(key="runpod_colours", value=RUNPOD_COLOURS))
db.commit()
unlisted = sum(1 for h in hosts.values() if not h.listed)
return f"runpod: seeded {len(hosts)} machines ({unlisted} unlisted) with history"
def run(db: Session) -> list[str]:
return [seed_handover(db), seed_runpod(db)]