Monitoring as an AI Employee

Design doc for replacing human on-call rotation with an AI agent that triages alerts, dismisses false alarms, executes runbook autofixes, and pages humans only when the situation genuinely warrants a human in the loop.

Why

Manual PagerDuty rotation is the wrong shape for a fast-evolving system like Mortar. The bottleneck is not “does someone wake up” — the bottleneck is “does the responder remember what p95 > 2s on /tables/*/rows actually means this week, given that last sprint we changed the rows pagination path.”

Runbook automation captures that knowledge in code; rotation paperwork merely schedules its forgetting. We invest in a bot that reads the runbook every time rather than a rotation that trusts humans to.

Architecture

Aliyun SLS query   →   alert webhook   →   oncall-triage (mortar.compute)
                                                │
                                                ├── fetch log window (last 5min)
                                                ├── fetch trace context (LogProject + TraceID)
                                                ├── load runbook.md from git
                                                ▼
                                           Claude Sonnet 4.6
                                                │
                                                ▼
                                  classify + decide (JSON)
                                                │
                ┌───────────────┬───────────────┴────────────────┐
                ▼               ▼                                ▼
          false-alarm     known-incident                   new-incident
          log + dismiss   run autofix(runbook)        Lark/DingTalk webhook
                                                       (page human with analysis)

SLS metric queries fire on threshold breach. The webhook is signed with HMAC; the bot rejects unsigned payloads. Log window + trace context are pulled through the SLS GetLogs API using a scoped AK that can read but not write.

Classification taxonomy

Classification is only the above three labels — there is no “investigate further” middle state, because middle states are how on-call bots become noise generators.

Implementation

The bot is a Mortar compute function:

await mortar.compute.deploy("oncall-triage", {
  source: "./oncall-triage.ts",
  trigger: { http: { path: "/webhook/sls" } },
  env: { ANTHROPIC_API_KEY: secret("anthropic"), RUNBOOK_REPO: "..." },
});

The runbook lives in git as runbook.md. Claude reads it via tool calls (fs_read / fs_grep) rather than having the whole file inlined, so the prompt stays small and the runbook can grow without retuning.

Cost model

Two orders of magnitude cheaper, and the bot’s median response time is ~4s.

Failure modes

• Bot can’t reach Claude API → fall through to paging human directly. No silent drops; the bot’s own health check fires its own alert.
• Bot in retry loop → circuit breaker after 3 attempts → page human and mark the bot itself as degraded.
• Claude classifies wrong → all false-alarm and known-incident decisions are written via the existing internal/audit package. A human reviews dismissed alerts weekly through mortar audit list --filter=ai-dismissed; precision regressions update the runbook (not the prompt — runbook is the source of truth).

Example

Structured prompt sent to Claude:

ALERT: p95_latency = 2.4s for 5min on /v1/proj_xyz/db/tables/users/rows
WINDOW: 2026-05-13T03:10Z → 03:15Z
TRACE: arms-cn-hangzhou:f3a91c8e
TOP LOGS (3):
  03:12:41 WARN pg slow query (1.8s) SELECT * FROM users WHERE ...
  03:13:02 WARN connection pool 95% (49/50)
  03:14:18 WARN pg slow query (2.1s) SELECT * FROM users WHERE ...
RUNBOOK: docs/runbook.md
TASK: classify + decide. Reply JSON only.

Claude’s response:

{
  "class": "known-incident",
  "match": "runbook.md#users-rows-pool-exhaustion",
  "action": "scale connection_pool max_conn 50→100",
  "reasoning": "matches runbook signature: slow query + pool >90% on /tables/users/rows"
}

The bot looks up users-rows-pool-exhaustion, runs the documented mortar db scale-pool command, and posts the outcome back into the alert thread.

Roadmap

• Today — false-alarm dismissal only. Bot never executes remediations; everything that isn’t a false alarm pages a human. Lowest blast radius to validate classification precision.
• Planned — runbook-driven autofix for known-incident. Remediations are whitelisted to idempotent commands (restart, scale, flush).
• Roadmap — cross-incident pattern learning. Claude reads the last 30 days of internal/audit records weekly and suggests new dashboard rules (e.g. “the pool-exhaustion alert fires every Tuesday at 3am — propose a scheduled scale-up instead of reactive paging”).