AI 监控员 Playbook 库

跟 monitoring-ai-employee.md 配对： AI 监控员触发告警后，按这个 playbook 库决定怎么处理。每个 playbook 是一段 markdown，AI 通过 tool call 读它 + 决定 autofix 或 page 人。

Playbook 设计原则：

  • 明确触发条件：metric query + 阈值，AI 容易 grep
  • 二元处理：autofix（AI 自己能搞）或 escalate（page 值班人）
  • 可逆性优先：autofix 操作必须可回退（restart 而不是 drop）
  • 限速：每个 playbook 自带”per N min run K times”上限，避免循环

Playbook index

触发严重度处理Doc
API p95 latency > 2s for 5minHighescalate#api-latency-spike
Postgres connections > 80% poolHighautofix (restart 连接池)#pg-connection-flood
OSS upload error rate > 5%Mediumescalate to Aliyun ticket#oss-error-rate
Redis OOM evict rate > 10% / minMediumautofix (扩容到下一 tier)#redis-oom
Compute cold-start p95 > 5sLowescalate (capacity planning)#compute-cold-start
Credit balance per project < ¥1Infoautofix (audit_log warn + auto top-up if enabled)#credit-low
Keel bundle download error > 1% (planned)Mediumautofix (回滚到 stable)#bundle-error
Cert expiry < 30 daysLowescalate (Caddy 一般自动续)#cert-expiry
New SQL injection attempt logHighautofix (block IP + audit)#sql-injection
Cost anomaly: project > 200% prev dayHighescalate + freeze project#cost-spike

api-latency-spike

Trigger: avg(api_p95_latency_ms) by (route) > 2000 for 5 consecutive minutes.

Action: ESCALATE to human.

AI pre-investigation:

  1. Read last 100 lines of mortar-api.log for the matching route
  2. Check Aliyun status page (RDS + Redis + OSS health)
  3. Check recent deploys: git log --since="1 hour ago"
  4. Calculate baseline (median p95 last 7 days) — surfacing % deviation

Page message template:

⚠ Latency spike: <route> p95 = <current> ms (baseline <baseline> ms, +<delta>%)
Started: <UTC time>
Recent deploys: <git short list>
Aliyun: <status — green / amber / red>
Last 5 error lines: <excerpt>
Playbook: oncall-playbooks.md#api-latency-spike

Not autofix because: latency causes are heterogeneous (slow query / connection pool / cache miss / GC pause / upstream cloud) and require human triage.

pg-connection-flood

Trigger: pg_stat_activity.count > 0.8 * max_connections for 2 min.

Action: AUTOFIX — restart the application’s connection pool (forces idle connections to drop).

Steps:

  1. Pull current pg_stat_activity snapshot for audit log
  2. Send SIGHUP to each Mortar replica → triggers connection pool reset (uses graceful drain — in-flight requests finish)
  3. Wait 60 sec
  4. Re-check; if still > 80%, escalate

Limit: max 3 runs per hour. Beyond → escalate (suggests slow query holding connections).

oss-error-rate

Trigger: oss_response_5xx_rate > 0.05 for 5 min.

Action: ESCALATE (likely Aliyun-side; we can’t fix).

AI pre-investigation:

  1. Check Aliyun OSS status page (https://status.aliyun.com)
  2. Sample 10 failing requests — extract bucket + key + status
  3. Check if isolated to one region

Note: OSS errors usually self-recover in < 30 min. Don’t autofix unless we know root cause is on our side (bucket policy misconfig etc.).

redis-oom

Trigger: redis_evicted_keys_per_sec > <baseline> * 10 for 10 min.

Action: AUTOFIX — bump Tair capacity to next tier.

Steps:

  1. Read current Tair tier + memory utilization
  2. Aliyun API call to upgrade tier (e.g. tiny → small)
  3. Tair handles online resize transparently (no downtime)
  4. Verify post-upgrade evict rate drops within 5 min

Cost impact: alert next-day bill review (cost may jump 2x). Limit: max 1 upgrade / day. Beyond → escalate (suggests capacity planning issue, not transient spike).

compute-cold-start

Trigger: compute_cold_start_p95 > 5000 for 30 min.

Action: ESCALATE — capacity planning, AI doesn’t have authority to bump idle pool size.

AI pre-investigation:

  1. Which function? (group by function_name)
  2. How many cold starts in window?
  3. Cost of bumping idle pool: estimate from FC pricing

credit-low

Trigger: credit_balance_yuan < 1.0 for any project.

Action: AUTOFIX — emit credit_low audit event + (if auto_top_up mode) trigger charge.

Steps:

  1. Write audit_log row kind=credit_low with project + owner + balance
  2. If credit_mode = auto_top_up: trigger top-up via payment.Multi
  3. Owner notification ships out of the audit stream by whatever channel the deploy operator has wired downstream (Slack webhook, log aggregator alert, in-app banner). Mortar itself no longer sends email — the email feature was removed.

Limit: 1 audit event per project per 24 hours (dedup on kind + project).

bundle-error

Trigger: metric to be defined when Keel Update server emits Prometheus stats — planned. Placeholder; the playbook below is the intended response shape once the metric ships.

Action: AUTOFIX — keel rollback --to=<previous-promoted-hash>.

Steps:

  1. Look up which project is affected (group by project)
  2. Find previous promoted hash by querying Keel’s bundles table for the prior entry under the same (project, platform, channel, runtime_version) tuple — the in-flight rollout’s staged_hash in currents is the current target; the row in bundles with the most recent uploaded_at before that is the rollback target. See keel/cloud/internal/update/store.go for the schema.
  3. Run keel rollback for that project (server-side: Rollback() on ManifestStore)
  4. Emit bundle_rollback audit event for downstream owner notification

Limit: 1 auto-rollback per project per 24 hours. Multiple rollbacks suggest bigger problem — escalate.

cert-expiry

Trigger: cron daily; cert.notAfter < now + 30 days.

Action: ESCALATE (Caddy auto-renew is normally fine; alert is for paranoia).

Special: If Caddy hasn’t attempted renewal in last 24h, page immediately (Caddy might be stuck).

sql-injection

Trigger: SQL syntax error patterns in audit log (e.g. UNION SELECT, DROP TABLE, ;-- in user input).

Action: AUTOFIX — temporary IP block + alert security team.

Steps:

  1. Extract source IP from request log
  2. Add to rate-limiter denylist for 24 hours
  3. Record in audit log with severity=critical
  4. Page security on-call (not normal on-call)

Special: Don’t autofix if same IP appears > 100 times — that’s distributed attack, needs human + 阿里云 anti-DDoS coordination.

cost-spike

Trigger: project_credits_used_today > 2 * project_credits_used_yesterday AND today_credits > 100 yuan.

Action: ESCALATE + freeze project (autofix).

Steps:

  1. Emit cost_anomaly audit event (owner notification routes through whatever channel the operator has wired on the audit stream)
  2. Set project credit_mode = hard_budget + log incident; user must intervene to top up
  3. Subsequent API calls hit credit-gate 402
  4. Page customer success team — manual review

Why autofix freeze: Mortar’s hard-budget gate is the safety net against rogue infinite-loop functions burning customer’s account. Pause first, ask later. (Current credit model is single-dimension: only credit exists; no monthly_cap / hourly cap fields — top-up is the only path back to writable.)

Adding a new playbook

When you add a new alert rule, write the playbook here. Format:

## <slug>

**Trigger**: <metric expression>
**Action**: AUTOFIX | ESCALATE
**Steps**: numbered list of actions
**Limit**: rate-limit + "beyond → escalate" rule
**Why** (if autofix): justify the reversibility

AI 监控员通过 tool call 读 oncall-playbooks.md 的 # heading 列 表，匹配 slug，按 “Action” 字段决定行为。新 playbook 加到这里 + AI 下次告警就用上，不需要 redeploy。

扩展计划（规划中）

  • 跨产品 playbook（Fabric / Keel / Mortar 共享）
  • Playbook 版本化 — 记录每次 autofix 的”按 v3 of playbook X”
  • 月度 incident review 自动总结 + 改进 playbook

在 GitHub 上编辑 →