Parham Monfared
a039e0b5fd
Read-only triage for the Infrahub error alerts. Pulls the Prometheus alert queue, re-checks each alert's condition against live state to separate real work from noise, diagnoses it using the CX runbooks, and drafts the customer comms with contacts resolved from Infrahub. Findings from validating against production: - "Suspected Rogue VM" fires on spare GPU capacity, not rogue VMs: In_Use_Gpus equals the physical count on 71 of 75 firing hosts, so the rule reduces to "this host has a free GPU". Verified against OpenStack on 10 hosts. - "Exists in Infrahub but does not exist in OpenStack" matches every VM because openstack_nova_server_status returns no series; excluded as a rule defect. - Prometheus activeAt is reset several times a day by dips in the Resources metric, so alert ages are recovered from ALERTS history instead. Takes ~2,650 firing alerts down to ~20 that need a decision. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
90 lines
3.9 KiB
Markdown
90 lines
3.9 KiB
Markdown
# Wiring Zendesk and Jira
|
|
|
|
"Not wired" meant: the app **builds** the full API payload and shows it to you,
|
|
but the code path that would POST it to Zendesk deliberately returns an error.
|
|
Nothing can reach a customer today, even by accident. Here is what is needed to
|
|
change that.
|
|
|
|
---
|
|
|
|
## What I need from you
|
|
|
|
### Zendesk
|
|
|
|
| Thing | Where it comes from | Example |
|
|
|---|---|---|
|
|
| Subdomain | your Zendesk URL `https://<subdomain>.zendesk.com` | `nexgencloud` |
|
|
| API email | the agent account tickets are created as | `cx-bot@nexgencloud.com` |
|
|
| API token | Zendesk **Admin Center → Apps and integrations → APIs → Zendesk API → Add API token** | 40-char string |
|
|
|
|
Plus four decisions:
|
|
|
|
1. **Which agent account should own these tickets?** A dedicated `cx-triage` agent
|
|
is better than a person's account — the audit trail stays clear.
|
|
2. **Public reply or internal note on first send?** The runbook wording is written
|
|
for the customer, so public — but confirm.
|
|
3. **Requester matching.** Infrahub gives us the owner's name and email. If they
|
|
are not already a Zendesk user, should we create them, or fail and ask a human?
|
|
4. **Do you want a tag convention** beyond the `cx-triage`, `alert-<kind>`,
|
|
`region-<x>` tags I currently set?
|
|
|
|
### Jira
|
|
|
|
| Thing | Where it comes from | Example |
|
|
|---|---|---|
|
|
| Base URL | your Atlassian site | `https://nexgencloud.atlassian.net` |
|
|
| API email | Atlassian account | `cx-bot@nexgencloud.com` |
|
|
| API token | <https://id.atlassian.com/manage-profile/security/api-tokens> | token string |
|
|
| Project key | the Infrastructure project | `INFRA`? |
|
|
| Issue type | must exist in that project | `Task`? `Bug`? |
|
|
|
|
Confirm the project key and issue type — I guessed `INFRA` / `Task`.
|
|
|
|
### Where to put the credentials
|
|
|
|
**Don't paste them to me.** Two options:
|
|
|
|
- **Preferred — 1Password.** Create two items in the `Employee` vault, e.g.
|
|
`Zendesk API (CX Triage)` and `Jira API (CX Triage)`, each with the token in the
|
|
`password` field and the email in a `username` field. I extend the existing
|
|
CX-Tools secrets loader to read them, exactly like the Infrahub key. Nothing
|
|
touches disk.
|
|
- **Quick and dirty** — environment variables in your shell before launching:
|
|
`CX_ZENDESK_SUBDOMAIN`, `CX_ZENDESK_EMAIL`, `CX_ZENDESK_TOKEN`,
|
|
`CX_JIRA_BASE`, `CX_JIRA_EMAIL`, `CX_JIRA_TOKEN`, `CX_JIRA_PROJECT`.
|
|
Fine for a trial, worse for a shared tool.
|
|
|
|
---
|
|
|
|
## What I build once I have that
|
|
|
|
1. **Delivery** — replace the refusal in `App.send_zendesk` with a real POST, and
|
|
add the same for Jira.
|
|
2. **Search before create** — `GET /api/v2/search?query=external_id:cx-triage-<fp>`
|
|
so re-diagnosing an alert comments on the existing ticket rather than opening a
|
|
second one. Same for Jira via a `cx-triage-<fp>` label.
|
|
3. **A send gate** — the server refuses to send unless started with `--allow-send`.
|
|
A demo instance then physically cannot email a customer, no matter what is
|
|
clicked. Default stays off.
|
|
4. **Audit** — every send appended to `vmc-audit.log` with who, what, and the
|
|
resulting ticket URL, and the URL shown back on the case.
|
|
5. **Rate limiting** — Zendesk allows 700 req/min; a simple per-minute cap plus a
|
|
refusal to send more than N tickets in one session, so a bad loop cannot mail
|
|
a hundred customers.
|
|
|
|
## Guard rails that stay regardless
|
|
|
|
- Every send needs a human click plus a confirm naming the recipient.
|
|
- No auto-send, ever — a verdict never triggers an email on its own.
|
|
- No bulk send in v1. The 13 overdue ERROR alerts are tempting, but one wrong
|
|
template across 13 customers is a bad first outing.
|
|
- The message stays editable before sending.
|
|
- Deleting, shelving and InfraInsight edits remain copy-a-command. The read-only
|
|
guarantee is what makes this safe against production.
|
|
|
|
## Suggested first run
|
|
|
|
Point it at a **Zendesk sandbox** first, or send the first real ticket to your own
|
|
address by editing the To field. Once one round-trip looks right in Zendesk, turn
|
|
it on for real.
|