CX Triage: alert diagnosis over the CX-Tools collectors

Read-only triage for the Infrahub error alerts. Pulls the Prometheus alert
queue, re-checks each alert's condition against live state to separate real
work from noise, diagnoses it using the CX runbooks, and drafts the customer
comms with contacts resolved from Infrahub.

Findings from validating against production:
- "Suspected Rogue VM" fires on spare GPU capacity, not rogue VMs: In_Use_Gpus
  equals the physical count on 71 of 75 firing hosts, so the rule reduces to
  "this host has a free GPU". Verified against OpenStack on 10 hosts.
- "Exists in Infrahub but does not exist in OpenStack" matches every VM because
  openstack_nova_server_status returns no series; excluded as a rule defect.
- Prometheus activeAt is reset several times a day by dips in the Resources
  metric, so alert ages are recovered from ALERTS history instead.

Takes ~2,650 firing alerts down to ~20 that need a decision.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-06 06:48:34 +01:00
commit a039e0b5fd
23 changed files with 7192 additions and 0 deletions

429
triagelib/ui.py Normal file
View File

@@ -0,0 +1,429 @@
"""The single-page UI, served inline so the app needs no assets or CDN."""
from __future__ import annotations
PAGE = r"""<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>CX Triage</title>
<style>
:root {
--bg: #0f1115; --panel: #171a21; --panel2: #1d2129; --line: #2a2f3a;
--fg: #e6e9ef; --muted: #8b93a3; --accent: #5b9cf8;
--ok: #3fb950; --warn: #d29922; --bad: #f85149; --info: #58a6ff;
--mono: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
}
@media (prefers-color-scheme: light) {
:root {
--bg: #f6f7f9; --panel: #ffffff; --panel2: #f0f2f5; --line: #dfe3e8;
--fg: #1a1d23; --muted: #5c6472; --accent: #1f6feb;
--ok: #1a7f37; --warn: #9a6700; --bad: #cf222e; --info: #0969da;
}
}
* { box-sizing: border-box; }
body {
margin: 0; background: var(--bg); color: var(--fg);
font: 14px/1.5 -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;
}
header {
display: flex; align-items: center; gap: 14px; flex-wrap: wrap;
padding: 10px 20px; border-bottom: 1px solid var(--line); background: var(--panel);
position: sticky; top: 0; z-index: 10;
}
h1 { font-size: 15px; margin: 0; font-weight: 650; }
h1 span { color: var(--muted); font-weight: 400; }
.pills { display: flex; gap: 6px; flex-wrap: wrap; margin-left: auto; }
.pill { font-size: 11px; padding: 3px 9px; border-radius: 999px; border: 1px solid var(--line);
background: var(--panel2); color: var(--muted); white-space: nowrap; }
.pill.ok { color: var(--ok); border-color: color-mix(in srgb, var(--ok) 40%, transparent); }
.pill.bad { color: var(--bad); border-color: color-mix(in srgb, var(--bad) 40%, transparent); }
button { font: inherit; font-size: 13px; padding: 5px 12px; border-radius: 6px;
border: 1px solid var(--line); background: var(--panel2); color: var(--fg); cursor: pointer; }
button:hover { border-color: var(--accent); }
button.primary { background: var(--accent); border-color: var(--accent); color: #fff; }
main { display: grid; grid-template-columns: minmax(340px, 32%) 1fr; min-height: calc(100vh - 52px); }
@media (max-width: 900px) { main { grid-template-columns: 1fr; } }
#queue { border-right: 1px solid var(--line); background: var(--panel);
overflow-y: auto; max-height: calc(100vh - 52px); }
#detail { padding: 20px 24px; overflow-y: auto; max-height: calc(100vh - 52px); }
.tabs { display: flex; border-bottom: 1px solid var(--line); }
.tab { flex: 1; padding: 10px 8px; text-align: center; cursor: pointer; font-size: 13px;
color: var(--muted); border-bottom: 2px solid transparent; }
.tab.on { color: var(--fg); border-bottom-color: var(--accent); font-weight: 600; }
.tab b { font-weight: 650; }
.qhead { padding: 10px 14px; border-bottom: 1px solid var(--line);
display: flex; flex-direction: column; gap: 8px; }
.qhead input, .qhead select, textarea {
font: inherit; width: 100%; padding: 6px 9px; border-radius: 6px;
border: 1px solid var(--line); background: var(--bg); color: var(--fg);
}
textarea { font-family: var(--mono); font-size: 12px; min-height: 62px; resize: vertical; }
.row { display: flex; align-items: center; gap: 8px; font-size: 12px; color: var(--muted); }
.banner { margin: 10px 14px; padding: 9px 11px; border-radius: 7px; font-size: 12px;
border: 1px solid color-mix(in srgb, var(--warn) 45%, transparent);
background: color-mix(in srgb, var(--warn) 10%, transparent); color: var(--fg); }
.group { border-bottom: 1px solid var(--line); }
.ghead { padding: 9px 14px; display: flex; align-items: center; gap: 8px;
cursor: pointer; user-select: none; background: var(--panel2); }
.ghead:hover { background: color-mix(in srgb, var(--accent) 8%, var(--panel2)); }
.caret { color: var(--muted); font-size: 10px; width: 10px; transition: transform .12s; }
.group.closed .caret { transform: rotate(-90deg); }
.group.closed .glist { display: none; }
.gtitle { font-weight: 620; font-size: 13px; flex: 1; }
.gcount { font-size: 11px; padding: 1px 7px; border-radius: 999px; background: var(--bg); color: var(--muted); }
.gcount.live { color: #fff; background: var(--bad); }
.alert { padding: 9px 14px 9px 26px; border-top: 1px solid var(--line); cursor: pointer; }
.alert:hover { background: var(--panel2); }
.alert.sel { background: color-mix(in srgb, var(--accent) 14%, transparent); box-shadow: inset 3px 0 var(--accent); }
.alert.muted { opacity: .55; }
.alert .t { font-weight: 600; font-size: 13px; word-break: break-all; }
.alert .m { color: var(--muted); font-size: 12px; }
.alert .why { font-size: 11.5px; margin-top: 3px; }
.age { font-family: var(--mono); font-size: 11.5px; color: var(--muted); }
.tag { font-size: 10px; padding: 1px 6px; border-radius: 4px; border: 1px solid var(--line); color: var(--muted); }
.v-real { color: var(--bad); border-color: color-mix(in srgb, var(--bad) 45%, transparent); }
.v-unverified { color: var(--warn); border-color: color-mix(in srgb, var(--warn) 45%, transparent); }
.v-chronic, .v-low_impact, .v-pending { color: var(--muted); }
.v-resolved { color: var(--ok); border-color: color-mix(in srgb, var(--ok) 40%, transparent); }
.card { background: var(--panel); border: 1px solid var(--line); border-radius: 10px;
padding: 16px 18px; margin-bottom: 16px; }
.card h2 { font-size: 12px; text-transform: uppercase; letter-spacing: .6px; color: var(--muted);
margin: 0 0 12px; font-weight: 600; }
.verdict { font-size: 17px; font-weight: 650; line-height: 1.35; margin-bottom: 8px; }
.assessment { color: var(--muted); }
.meta { display: flex; gap: 8px; flex-wrap: wrap; margin-top: 12px; align-items: center; }
table { width: 100%; border-collapse: collapse; }
td { padding: 5px 8px; border-bottom: 1px solid var(--line); vertical-align: top; font-size: 13px; }
tr:last-child td { border-bottom: none; }
td.k { color: var(--muted); width: 190px; white-space: nowrap; }
td.v { font-family: var(--mono); font-size: 12.5px; word-break: break-word; }
.tone-ok { color: var(--ok); } .tone-warn { color: var(--warn); }
.tone-bad { color: var(--bad); } .tone-info { color: var(--fg); }
.det { display: block; color: var(--muted); font-family: inherit; font-size: 12px; margin-top: 3px; }
ol.actions { margin: 0; padding-left: 0; list-style: none; counter-reset: a; }
ol.actions li { counter-increment: a; padding: 9px 0 9px 30px; border-bottom: 1px solid var(--line); position: relative; }
ol.actions li:last-child { border-bottom: none; }
ol.actions li::before {
content: counter(a); position: absolute; left: 0; top: 9px; width: 20px; height: 20px;
border-radius: 50%; border: 1px solid var(--line); font-size: 11px; color: var(--muted);
display: grid; place-items: center;
}
ol.actions li.done::before { content: "\2713"; color: var(--ok); border-color: var(--ok); }
.owner { font-size: 11px; padding: 1px 6px; border-radius: 4px; background: var(--panel2);
color: var(--muted); margin-left: 6px; }
.owner.infra { color: var(--warn); }
.kind { font-size: 10px; text-transform: uppercase; letter-spacing: .5px; color: var(--muted); margin-left: 6px; }
.guide { display: block; font-size: 12px; color: var(--muted); margin-top: 2px; }
.draft { border: 1px solid var(--line); border-radius: 8px; margin-bottom: 12px; overflow: hidden; }
.draft .dh { padding: 9px 12px; background: var(--panel2); display: flex; align-items: center; gap: 10px; flex-wrap: wrap; }
.draft .dl { font-weight: 600; font-size: 13px; flex: 1; min-width: 180px; }
.draft .dw { padding: 8px 12px; color: var(--warn); font-size: 12px; border-bottom: 1px solid var(--line); }
.draft pre { margin: 0; padding: 12px; white-space: pre-wrap; font-family: var(--mono); font-size: 12.5px; }
.contact { font-family: var(--mono); font-size: 12.5px; }
.notes li { color: var(--warn); margin-bottom: 5px; font-size: 13px; }
.empty { color: var(--muted); padding: 40px 20px; text-align: center; }
.spinner { width: 14px; height: 14px; border: 2px solid var(--line); border-top-color: var(--accent);
border-radius: 50%; display: inline-block; animation: spin .7s linear infinite; vertical-align: -2px; }
@keyframes spin { to { transform: rotate(360deg); } }
details.raw summary { cursor: pointer; color: var(--muted); font-size: 12px; }
details.raw pre { background: var(--panel2); padding: 10px; border-radius: 6px; overflow-x: auto; font-size: 11.5px; }
.ro { font-size: 11px; color: var(--muted); border: 1px dashed var(--line); padding: 2px 8px; border-radius: 999px; }
.infsec { border-bottom: 1px solid var(--line); }
.infsec .ghead { background: var(--panel2); }
.infrow { padding: 6px 14px 6px 26px; border-top: 1px solid var(--line); display: flex; gap: 8px; font-size: 12.5px; }
.infrow .n { flex: 1; }
</style>
</head>
<body>
<header>
<h1>CX Triage <span>alert diagnosis over CX-Tools</span></h1>
<span class="ro">read-only &middot; suggests, never acts</span>
<div class="pills" id="pills"></div>
<button id="refresh">Refresh</button>
</header>
<main>
<section id="queue">
<div class="tabs">
<div class="tab on" data-tab="cx" id="tabCx">CX runbooks</div>
<div class="tab" data-tab="infra" id="tabInfra">Infrastructure</div>
</div>
<div id="warnings"></div>
<div class="qhead">
<input id="search" type="search" placeholder="Filter by name, host, IP, org, ID...">
<label class="row"><input type="checkbox" id="showNoise" style="width:auto">
Show screened-out (chronic / pending / resolved) <span id="noiseCount"></span></label>
<div class="row" id="excludedNote"></div>
<details>
<summary style="cursor:pointer;color:var(--muted);font-size:12px">Paste an alert manually</summary>
<textarea id="paste" placeholder='Paste an ALERTS{...} line or a Prometheus graph URL'></textarea>
<button id="parseBtn" style="margin-top:6px">Diagnose pasted alert</button>
</details>
</div>
<div id="list"></div>
</section>
<section id="detail"><div class="empty">Pick an alert to diagnose.</div></section>
</main>
<script>
const $ = (s) => document.querySelector(s);
const esc = (s) => String(s == null ? "" : s).replace(/[&<>"']/g, c => (
{"&":"&amp;","<":"&lt;",">":"&gt;",'"':"&quot;","'":"&#39;"}[c]));
let STATE = { data: null, tab: "cx", selected: null, polling: null, closed: {}, lastBody: null };
const api = (p, o) => fetch(p, o).then(r => r.json());
async function loadHealth() {
const h = await api("/api/health");
$("#pills").innerHTML = (h.checks || []).map(c =>
`<span class="pill ${c.ok ? "ok" : "bad"}" title="${esc(c.detail)}">${esc(c.name)}</span>`).join("");
}
async function loadAlerts(force) {
if (!STATE.data) $("#list").innerHTML = '<div class="empty"><span class="spinner"></span> Loading alerts…</div>';
const d = await api("/api/alerts" + (force ? "?force=1" : ""));
STATE.data = d;
if (d.error) $("#list").innerHTML = `<div class="empty tone-bad">${esc(d.error)}</div>`;
const t = d.totals || {};
const s = d.summary || {};
$("#tabCx").innerHTML = `CX runbooks <b>${s.actionable || 0}</b>`;
$("#tabInfra").innerHTML = `Infrastructure <b>${t.infrastructure || 0}</b>`;
$("#noiseCount").innerHTML = `<span class="tag">${s.screened_out || 0} screened out</span>`;
$("#excludedNote").textContent = d.excluded_note || "";
$("#warnings").innerHTML = (d.warnings || []).map(w => `<div class="banner">${esc(w)}</div>`).join("");
render();
}
function matches(a, q) {
if (!q) return true;
return [a.instance_name, a.host, a.floating_ip, a.openstack_id, a.org_name, a.org_id, a.title, a.alertname]
.join(" ").toLowerCase().includes(q);
}
function render() {
document.querySelectorAll(".tab").forEach(t => t.classList.toggle("on", t.dataset.tab === STATE.tab));
STATE.tab === "cx" ? renderCx() : renderInfra();
}
function renderCx() {
const d = STATE.data || {};
const q = $("#search").value.trim().toLowerCase();
const showNoise = $("#showNoise").checked;
const out = [];
for (const g of d.groups || []) {
let rows = (g.alerts || []).filter(a => matches(a, q));
if (!showNoise) rows = rows.filter(a => a.screen && a.screen.actionable);
if (!rows.length) continue;
const live = rows.filter(a => a.screen && a.screen.actionable).length;
const closed = STATE.closed[g.kind] === true;
out.push(`<div class="group ${closed ? "closed" : ""}" data-kind="${esc(g.kind)}">
<div class="ghead">
<span class="caret">&#9660;</span>
<span class="gtitle">${esc(g.title)}</span>
<span class="gcount ${live ? "live" : ""}">${live ? live + " to action" : "0 to action"}</span>
${g.noise ? `<span class="gcount" title="screened out: chronic, pending, low impact or already resolved">${g.noise} not new</span>` : ""}
</div>
<div class="glist">${rows.map(a => alertRow(a)).join("")}</div>
</div>`);
}
$("#list").innerHTML = out.length ? out.join("") :
`<div class="empty">${(d.summary || {}).actionable === 0
? "Nothing new needs action. " + ((d.summary || {}).screened_out || 0) + " alert(s) screened out \u2014 tick the box above to see them."
: "No matching alerts."}</div>`;
document.querySelectorAll(".group .ghead").forEach(el => el.onclick = () => {
const kind = el.parentElement.dataset.kind;
STATE.closed[kind] = !STATE.closed[kind];
render();
});
document.querySelectorAll(".alert").forEach(el =>
el.onclick = () => triage({ alert_id: el.dataset.id }, el.dataset.id));
}
function alertRow(a) {
const subject = a.kind === "duplicate_ip" ? a.floating_ip
: (a.kind === "rogue_vm" || a.kind === "total_gpus" || a.kind === "orphan_vm") ? a.host
: (a.instance_name || a.openstack_id || "unknown");
const bits = [a.region_label || a.region, a.status, a.org_name].filter(Boolean);
const sc = a.screen || {};
return `<div class="alert ${STATE.selected === a.id ? "sel" : ""} ${sc.actionable ? "" : "muted"}" data-id="${esc(a.id)}">
<div style="display:flex;gap:8px;align-items:baseline">
<div class="t" style="flex:1">${esc(subject)}</div>
<div class="age" title="${a.age_is_reset ? "condition has held this long; Prometheus says only " + esc(a.age_text) + " because a pipeline dip reset it" : "how long the condition has held"}">${
esc(a.effective_age_text || "")}${a.age_is_reset ? "*" : ""}</div>
</div>
<div class="m">${esc(bits.join(" · "))}</div>
<div class="why"><span class="tag v-${esc(sc.verdict || "unverified")}">${esc(sc.label || "")}</span>
<span style="color:var(--muted)"> ${esc(sc.reason || "")}</span></div>
</div>`;
}
function renderInfra() {
const d = STATE.data || {};
const q = $("#search").value.trim().toLowerCase();
const out = [`<div class="banner">These are not CX runbook alerts &mdash; they are host and platform
rules, shown here so they stay out of the triage queue.</div>`];
for (const s of d.infrastructure || []) {
const names = (s.by_alertname || []).filter(n => !q || n.name.toLowerCase().includes(q));
if (!names.length) continue;
const closed = STATE.closed["inf:" + s.source] === true;
out.push(`<div class="infsec group ${closed ? "closed" : ""}" data-kind="inf:${esc(s.source)}">
<div class="ghead">
<span class="caret">&#9660;</span>
<span class="gtitle">${esc(s.label)}</span>
<span class="gcount">${s.total} firing</span>
</div>
<div class="glist">${names.map(n =>
`<div class="infrow"><span class="n">${esc(n.name)}</span><span class="tag">${n.count}</span></div>`
).join("")}</div>
</div>`);
}
$("#list").innerHTML = out.join("");
document.querySelectorAll(".infsec .ghead").forEach(el => el.onclick = () => {
const k = el.parentElement.dataset.kind;
STATE.closed[k] = !STATE.closed[k];
render();
});
}
async function triage(body, id) {
STATE.selected = id || null;
STATE.lastBody = body;
render();
if (STATE.polling) { clearInterval(STATE.polling); STATE.polling = null; }
$("#detail").innerHTML = '<div class="empty"><span class="spinner"></span> Checking Infrahub, OpenStack and InfraInsight…<br><small>Host-wide checks can take up to a minute.</small></div>';
const started = await api("/api/triage", {
method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify(body) });
if (started.error) { $("#detail").innerHTML = `<div class="empty tone-bad">${esc(started.error)}</div>`; return; }
STATE.polling = setInterval(async () => {
const job = await api("/api/jobs/" + started.job_id);
if (job.state === "running") return;
clearInterval(STATE.polling); STATE.polling = null;
if (job.state === "error") {
$("#detail").innerHTML = `<div class="card"><h2>Triage failed</h2><pre>${esc(job.error)}</pre></div>`;
return;
}
renderDiagnosis(job.result);
}, 900);
}
function diagnoseAnyway() {
triage({ ...(STATE.lastBody || {}), force: true }, STATE.selected);
}
function renderDiagnosis(d) {
const a = d.alert || {};
const sc = a.screen || {};
const out = [];
out.push(`<div class="card">
<h2>${esc(a.title)} &middot; ${esc(a.region_label || a.region || "region unknown")}</h2>
<div class="verdict">${esc(d.verdict || d.error || "No verdict")}</div>
${d.assessment ? `<div class="assessment">${esc(d.assessment)}</div>` : ""}
<div class="meta">
<span class="tag v-${esc(sc.verdict || "unverified")}">${esc(sc.label || "")}</span>
<span class="tag">priority ${esc(a.priority)}</span>
<span class="tag">confidence ${esc(d.confidence)}</span>
<span class="tag">ETTR ${esc(a.ettr)}</span>
<span class="tag">held ${esc(a.effective_age_text || "?")}</span>
${a.age_is_reset ? `<span class="tag" title="a metric-pipeline dip reset Prometheus' activeAt">prometheus says ${esc(a.age_text)}</span>` : ""}
${d.elapsed_seconds ? `<span class="tag">diagnosed in ${d.elapsed_seconds}s</span>` : ""}
${a.is_kubernetes ? '<span class="tag">likely K8s node</span>' : ""}
${!sc.actionable ? '<button onclick="diagnoseAnyway()">Diagnose anyway</button>' : ""}
</div>
</div>`);
if (d.error) out.push(`<div class="card"><h2>Lookup problem</h2><div class="tone-bad">${esc(d.error)}</div></div>`);
if ((d.notes || []).length)
out.push(`<div class="card"><h2>Caveats</h2><ul class="notes">${
d.notes.map(n => `<li>${esc(n)}</li>`).join("")}</ul></div>`);
if ((d.findings || []).length)
out.push(`<div class="card"><h2>What the platforms say</h2><table>${
d.findings.map(f => `<tr><td class="k">${esc(f.label)}</td><td class="v tone-${esc(f.tone)}">${
esc(f.value)}${f.detail ? `<span class="det">${esc(f.detail)}</span>` : ""}</td></tr>`).join("")
}</table></div>`);
if ((d.actions || []).length)
out.push(`<div class="card"><h2>Next steps</h2><ol class="actions">${
d.actions.map(x => `<li class="${x.status === "done" ? "done" : ""}">${esc(x.text)}
<span class="owner ${x.owner !== "CX" ? "infra" : ""}">${esc(x.owner)}</span>
<span class="kind">${esc(x.kind)}</span>
${x.guide ? `<span class="guide">Guide: ${esc(x.guide)}</span>` : ""}
${x.detail ? `<span class="guide">${esc(x.detail)}</span>` : ""}</li>`).join("")
}</ol></div>`);
const c = d.contacts || {};
if ((d.drafts || []).length)
out.push(`<div class="card">
<h2>Suggested customer comms</h2>
<div style="margin-bottom:12px">
${c.organization ? `<div class="contact">Org: ${esc(c.organization)}</div>` : ""}
${(c.owners || []).length ? c.owners.map(o => `<div class="contact">${esc(o)}</div>`).join("")
: '<div class="tone-warn">No owner contacts resolved — look the org up in the Admin Portal.</div>'}
</div>
${d.drafts.map((x, i) => `<div class="draft">
<div class="dh"><span class="dl">${esc(x.label)}</span>
<span class="tag">${esc(x.channel)}</span>
<button onclick="copyDraft(${i})" id="cp${i}">Copy</button></div>
${x.when ? `<div class="dw">When: ${esc(x.when)}</div>` : ""}
${x.unfilled.length ? `<div class="dw">Still to fill in: ${esc(x.unfilled.join(", "))}</div>` : ""}
<pre>${esc(x.body)}</pre></div>`).join("")}
<div style="color:var(--muted);font-size:12px">Send from HubSpot. This app never contacts anyone.</div>
</div>`);
out.push(`<div class="card"><h2>Evidence</h2>
<details class="raw"><summary>Alert labels, annotations and screening</summary><pre>${
esc(JSON.stringify({ rule: { file: a.rule_file, group: a.rule_group, for_seconds: a.for_seconds },
screen: a.screen, labels: a.labels, annotations: a.annotations }, null, 2))}</pre></details>
<details class="raw" style="margin-top:8px"><summary>Raw CX-Tools output</summary><pre>${
esc(JSON.stringify(d.evidence, null, 2))}</pre></details></div>`);
$("#detail").innerHTML = out.join("");
window.__drafts = d.drafts || [];
}
function copyDraft(i) {
navigator.clipboard.writeText((window.__drafts[i] || {}).body || "").then(() => {
const b = $("#cp" + i), old = b.textContent;
b.textContent = "Copied"; setTimeout(() => b.textContent = old, 1200);
});
}
$("#parseBtn").onclick = async () => {
const text = $("#paste").value.trim();
if (!text) return;
const res = await api("/api/parse", {
method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ text }) });
if (res.error) { $("#detail").innerHTML = `<div class="empty tone-bad">${esc(res.error)}</div>`; return; }
const first = (res.parsed || [])[0];
if (!first) return;
if (!first.supported) {
$("#detail").innerHTML = `<div class="empty tone-warn">Parsed "${esc(first.alert.alertname)}" but no CX runbook covers it.</div>`;
return;
}
triage({ labels: first.alert.labels, annotations: first.alert.annotations }, null);
};
document.querySelectorAll(".tab").forEach(t => t.onclick = () => { STATE.tab = t.dataset.tab; render(); });
$("#refresh").onclick = () => { loadHealth(); loadAlerts(true); };
$("#search").oninput = render;
$("#showNoise").onchange = render;
loadHealth();
loadAlerts(false);
setInterval(() => loadAlerts(false), 60000);
</script>
</body>
</html>
"""