AI monitor playbook library

Pairs with monitoring-ai-employee: once the AI monitor triggers an alert, it picks the right playbook from this library. Each playbook is a markdown block the AI reads via tool calls + then decides autofix or page a human.

Playbook design principles:

• Explicit trigger condition: metric query + threshold, easy for the AI to grep
• Binary handling: autofix (AI handles it) or escalate (page the on-call human)
• Reversibility first: autofix actions must be rollbackable (restart, not drop)
• Rate limit: every playbook carries its own “per N min run K times” cap to avoid loops

Playbook index

api-latency-spike

Trigger: avg(api_p95_latency_ms) by (route) > 2000 for 5 consecutive minutes.

Action: ESCALATE to a human.

AI pre-investigation:

1. Read the last 100 lines of mortar-api.log for the matching route
2. Check the Aliyun status page (RDS + Redis + OSS health)
3. Check recent deploys: git log --since="1 hour ago"
4. Compute baseline (median p95 over the last 7 days) — surface the % deviation

Page message template:

⚠ Latency spike: <route> p95 = <current> ms (baseline <baseline> ms, +<delta>%)
Started: <UTC time>
Recent deploys: <git short list>
Aliyun: <status — green / amber / red>
Last 5 error lines: <excerpt>
Playbook: oncall-playbooks.md#api-latency-spike

Not an autofix because: latency causes are heterogeneous (slow query / connection pool / cache miss / GC pause / upstream cloud) and require human triage.

pg-connection-flood

Trigger: pg_stat_activity.count > 0.8 * max_connections for 2 min.

Action: AUTOFIX — restart the application’s connection pool (forces idle connections to drop).

Steps:

1. Pull current pg_stat_activity snapshot for the audit log
2. Send SIGHUP to each Mortar replica → triggers a connection-pool reset (graceful drain — in-flight requests finish)
3. Wait 60 sec
4. Re-check; if still > 80%, escalate

Limit: max 3 runs per hour. Beyond → escalate (suggests a slow query holding connections).

oss-error-rate

Trigger: oss_response_5xx_rate > 0.05 for 5 min.

Action: ESCALATE (likely Aliyun-side; we can’t fix).

AI pre-investigation:

1. Check the Aliyun OSS status page (https://status.aliyun.com)
2. Sample 10 failing requests — extract bucket + key + status
3. Check if isolated to one region

Note: OSS errors usually self-recover in < 30 min. Don’t autofix unless we know the root cause is on our side (bucket policy misconfig, etc.).

redis-oom

Trigger: redis_evicted_keys_per_sec > <baseline> * 10 for 10 min.

Action: AUTOFIX — bump Tair capacity to next tier.

Steps:

1. Read current Tair tier + memory utilization
2. Aliyun API call to upgrade tier (e.g. tiny → small)
3. Tair handles online resize transparently (no downtime)
4. Verify post-upgrade evict rate drops within 5 min

Cost impact: alert next-day bill review (cost may jump 2×). Limit: max 1 upgrade / day. Beyond → escalate (suggests a capacity planning issue, not a transient spike).

compute-cold-start

Trigger: compute_cold_start_p95 > 5000 for 30 min.

Action: ESCALATE — capacity planning; the AI doesn’t have authority to bump idle pool size.

AI pre-investigation:

1. Which function? (group by function_name)
2. How many cold starts in the window?
3. Cost of bumping the idle pool: estimate from FC pricing

credit-low

Trigger: credit_balance_yuan < 1.0 for any project.

Action: AUTOFIX — emit credit_low audit event + (if auto_top_up mode) trigger charge.

Steps:

1. Write an audit_log row kind=credit_low with project + owner + balance
2. If credit_mode = auto_top_up: trigger top-up via payment.Multi
3. Owner notification ships out of the audit stream by whatever channel the deploy operator has wired downstream (Slack webhook, log aggregator alert, in-app banner). Mortar itself no longer sends email — the email feature was removed.

Limit: 1 audit event per project per 24 hours (dedup on kind + project).

bundle-error

Trigger: metric to be defined when Keel Update server emits Prometheus stats — planned. Placeholder; the playbook below is the intended response shape once the metric ships.

Action: AUTOFIX — keel rollback --to=<previous-promoted-hash>.

Steps:

1. Look up which project is affected (group by project)
2. Find the previous promoted hash by querying Keel’s bundles table for the prior entry under the same (project, platform, channel, runtime_version) tuple — the in-flight rollout’s staged_hash in currents is the current target; the row in bundles with the most recent uploaded_at before that is the rollback target. See keel/cloud/internal/update/store.go for the schema.
3. Run keel rollback for that project (server-side: Rollback() on ManifestStore)
4. Emit a bundle_rollback audit event for downstream owner notification

Limit: 1 auto-rollback per project per 24 hours. Multiple rollbacks suggest a bigger problem — escalate.

cert-expiry

Trigger: cron daily; cert.notAfter < now + 30 days.

Action: ESCALATE (Caddy auto-renew is normally fine; this alert is for paranoia).

Special: If Caddy hasn’t attempted renewal in the last 24h, page immediately (Caddy may be stuck).

sql-injection

Trigger: SQL syntax error patterns in the audit log (e.g. UNION SELECT, DROP TABLE, ;-- in user input).

Action: AUTOFIX — temporary IP block + alert security team.

Steps:

1. Extract source IP from the request log
2. Add to the rate-limiter denylist for 24 hours
3. Record in the audit log with severity=critical
4. Page security on-call (not the normal on-call rotation)

Special: Don’t autofix if the same IP appears > 100 times — that’s a distributed attack and needs human + Aliyun anti-DDoS coordination.

cost-spike

Trigger: project_credits_used_today > 2 * project_credits_used_yesterday AND today_credits > 100 yuan.

Action: ESCALATE + freeze project (autofix).

Steps:

1. Emit a cost_anomaly audit event (owner notification routes through whatever channel the operator has wired on the audit stream)
2. Set project credit_mode = hard_budget + log incident; user must intervene to top up
3. Subsequent API calls hit credit-gate 402
4. Page customer success team — manual review

Why autofix the freeze: Mortar’s hard-budget gate is the safety net against rogue infinite-loop functions burning a customer’s account. Pause first, ask later. (Current credit model is single-dimension: only credit exists; no monthly_cap / hourly cap fields — top-up is the only path back to writable.)

Adding a new playbook

When you add a new alert rule, write the playbook here. Format:

## <slug>

**Trigger**: <metric expression>
**Action**: AUTOFIX | ESCALATE
**Steps**: numbered list of actions
**Limit**: rate-limit + "beyond → escalate" rule
**Why** (if autofix): justify the reversibility

The AI monitor reads the # heading list in oncall-playbooks.md via a tool call, matches the slug, and acts on the “Action” field. Adding a new playbook here means the AI uses it on the next alert — no redeploy required.

Extension plan (planned)

• Cross-product playbooks (Fabric / Keel / Mortar shared)
• Playbook versioning — log every autofix as “ran v3 of playbook X”
• Monthly incident review auto-summary + playbook improvements