Files

CX Triage: alert diagnosis over the CX-Tools collectors

Read-only triage for the Infrahub error alerts. Pulls the Prometheus alert
queue, re-checks each alert's condition against live state to separate real
work from noise, diagnoses it using the CX runbooks, and drafts the customer
comms with contacts resolved from Infrahub.

Findings from validating against production:
- "Suspected Rogue VM" fires on spare GPU capacity, not rogue VMs: In_Use_Gpus
  equals the physical count on 71 of 75 firing hosts, so the rule reduces to
  "this host has a free GPU". Verified against OpenStack on 10 hosts.
- "Exists in Infrahub but does not exist in OpenStack" matches every VM because
  openstack_nova_server_status returns no series; excluded as a rule defect.
- Prometheus activeAt is reset several times a day by dips in the Resources
  metric, so alert ages are recovered from ALERTS history instead.

Takes ~2,650 firing alerts down to ~20 that need a decision.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

2026-08-06 06:48:34 +01:00

3.9 KiB

Raw Blame History

Wiring Zendesk and Jira

"Not wired" meant: the app builds the full API payload and shows it to you, but the code path that would POST it to Zendesk deliberately returns an error. Nothing can reach a customer today, even by accident. Here is what is needed to change that.

What I need from you

Zendesk

Thing	Where it comes from	Example
Subdomain	your Zendesk URL `https://<subdomain>.zendesk.com`	`nexgencloud`
API email	the agent account tickets are created as	`cx-bot@nexgencloud.com`
API token	Zendesk Admin Center → Apps and integrations → APIs → Zendesk API → Add API token	40-char string

Plus four decisions:

Which agent account should own these tickets? A dedicated cx-triage agent is better than a person's account — the audit trail stays clear.
Public reply or internal note on first send? The runbook wording is written for the customer, so public — but confirm.
Requester matching. Infrahub gives us the owner's name and email. If they are not already a Zendesk user, should we create them, or fail and ask a human?
Do you want a tag convention beyond the cx-triage, alert-<kind>, region-<x> tags I currently set?

Jira

Thing	Where it comes from	Example
Base URL	your Atlassian site	`https://nexgencloud.atlassian.net`
API email	Atlassian account	`cx-bot@nexgencloud.com`
API token	https://id.atlassian.com/manage-profile/security/api-tokens	token string
Project key	the Infrastructure project	`INFRA`?
Issue type	must exist in that project	`Task`? `Bug`?

Confirm the project key and issue type — I guessed INFRA / Task.

Where to put the credentials

Don't paste them to me. Two options:

Preferred — 1Password. Create two items in the Employee vault, e.g. Zendesk API (CX Triage) and Jira API (CX Triage), each with the token in the password field and the email in a username field. I extend the existing CX-Tools secrets loader to read them, exactly like the Infrahub key. Nothing touches disk.
Quick and dirty — environment variables in your shell before launching: CX_ZENDESK_SUBDOMAIN, CX_ZENDESK_EMAIL, CX_ZENDESK_TOKEN, CX_JIRA_BASE, CX_JIRA_EMAIL, CX_JIRA_TOKEN, CX_JIRA_PROJECT. Fine for a trial, worse for a shared tool.

What I build once I have that

Delivery — replace the refusal in App.send_zendesk with a real POST, and add the same for Jira.
Search before create — GET /api/v2/search?query=external_id:cx-triage-<fp> so re-diagnosing an alert comments on the existing ticket rather than opening a second one. Same for Jira via a cx-triage-<fp> label.
A send gate — the server refuses to send unless started with --allow-send. A demo instance then physically cannot email a customer, no matter what is clicked. Default stays off.
Audit — every send appended to vmc-audit.log with who, what, and the resulting ticket URL, and the URL shown back on the case.
Rate limiting — Zendesk allows 700 req/min; a simple per-minute cap plus a refusal to send more than N tickets in one session, so a bad loop cannot mail a hundred customers.

Guard rails that stay regardless

Every send needs a human click plus a confirm naming the recipient.
No auto-send, ever — a verdict never triggers an email on its own.
No bulk send in v1. The 13 overdue ERROR alerts are tempting, but one wrong template across 13 customers is a bad first outing.
The message stays editable before sending.
Deleting, shelving and InfraInsight edits remain copy-a-command. The read-only guarantee is what makes this safe against production.

Suggested first run

Point it at a Zendesk sandbox first, or send the first real ticket to your own address by editing the To field. Once one round-trip looks right in Zendesk, turn it on for real.

3.9 KiB Raw Blame History