这里的每个 signal 都是在仅凭字段无法回答时提出的问题。将 when 改成你的路径，它们就属于你的应用。
从这三个开始
一个文件、三列，运行一周后你就会拥有以前没有的数据。
server/signals.ts
import { defineSignal } from '@evlog/signals'
const errorName = (e: Record<string, unknown>) => (e.error as { name?: string } | undefined)?.name ?? 'none'
/** Every 4xx/5xx. Splits the error budget between the client, the code and the dependencies. */
export const fault = defineSignal({
name: 'fault',
when: e => (e.status ?? 0) >= 400,
ask: 'Who is responsible for this failure?',
choice: {
client: 'Bad input, expired session, missing permission, client mistake',
app: 'A bug, a misconfiguration or a validation error in our own code',
upstream: 'A third-party dependency failed, timed out or rate-limited us',
},
cacheKey: e => e.error ? `${e.method} ${e.path} ${e.status} ${errorName(e)}` : undefined,
})
/** Every 5xx. Tells a retry policy which failures are worth a second attempt. */
export const retryable = defineSignal({
name: 'retryable',
when: e => (e.status ?? 0) >= 500,
ask: 'Would the same request most likely succeed if retried in a few seconds?',
criteria: { true: 'Timeout, connection reset, rate limit, transient upstream error', false: 'Bug, bad data, missing resource' },
cacheKey: e => e.error ? `${e.path} ${errorName(e)}` : undefined,
})
/** Successful checkouts. The 200 that sampling deletes and nobody notices. */
export const silentFailure = defineSignal({
name: 'silent-failure',
when: e => e.status === 200 && e.path === '/api/checkout',
ask: 'The request returned 200, but the customer did not get what they came for',
criteria: { true: 'No order or confirmation, a fallback path, an empty or partial result', false: 'Order created and confirmed' },
keep: v => v.value && v.confidence > 0.8,
})
每个 signal 提供的结果：
- 一周内执行
GROUP BY signals.fault.value，即可知道下一个 sprint 应该投入验证消息、自身 bug，还是依赖项的 SLA。在错误名称上设置cacheKey，可以让一次故障只产生一次调用，而不是每个请求一次。 - 502 上的
signals.retryable.value = true，就是重试策略与猜测之间的区别。 silent-failure会提升返回 200 却没有订单的结账请求。将when指向你自己的资金路径。
全部十二个
| Signal | 钩子 | 问题 | 规则为什么无法回答 |
|---|---|---|---|
fault | enrich、缓存 | 责任属于用户、我们还是上游 | Stripe 超时和空值解引用都会得到 500 |
severity | enrich | 是噪声、需要关注还是需要通知值班人员 | 紧急程度取决于失败内容和受影响的人 |
retryable | enrich、缓存 | 重试会成功吗 | 临时错误和永久错误可能共享状态码 |
slow-cause | enrich | 数据库、上游还是计算 | 读取时间分布需要人工判断 |
silent-failure | keep | 返回了 200，但客户空手而归吗 | 采样只能保留谓词能够命名的内容 |
webhook-ignored | keep | 已确认但没有执行吗 | provider 只能看到 200 |
validation-bug | keep | 被拒绝的输入其实有效吗 | 看起来与无效输入相同 |
turn-resolved | enrich | agent 完成了要求吗 | 抽样的 LLM-as-judge 会漏掉其余回合 |
turn-looping | keep | 是否重复调用工具却没有进展 | 循环会隐藏在成功回合中 |
turn-off-script | keep | 是否执行了没人要求的操作 | 同上 |
audit-review | keep | 此操作是否值得人工审核 | 风险来自字段组合，而不是某个字段 |
job-flaky | enrich | 是否只是因为重试才成功 | 重试会隐藏原因 |
排列失败并解释慢请求
export const severity = defineSignal({
name: 'severity',
when: e => (e.status ?? 0) >= 500,
ask: 'How urgent is this failure for the on-call engineer?',
score: ['noise', 'watch', 'page'],
})
export const slowCause = defineSignal({
name: 'slow-cause',
when: e => (e.durationMs ?? 0) > 2_000,
ask: 'What dominated the duration of this request?',
choice: {
database: 'Query time, lock waits, many round trips',
upstream: 'A third-party API or another service',
compute: 'Serialization, rendering, large payloads, CPU work',
unknown: 'Nothing in the event explains the time',
},
})
slow-cause 中的 unknown 选项是有意设置的：没有它，模型会从三个选项中选择最不错误的一个，而你会失去事件缺少应有时间信息这一信号。
拯救其他说谎的 200 响应
export const webhookIgnored = defineSignal({
name: 'webhook-ignored',
when: e => e.status === 200 && (e.path ?? '').startsWith('/api/webhooks/'),
ask: 'The webhook was acknowledged but not acted on',
criteria: { true: 'Unhandled event type, skipped, no side effect recorded', false: 'The event was processed' },
keep: v => v.value && v.confidence > 0.85,
})
export const validationBug = defineSignal({
name: 'validation-bug',
when: e => e.status === 400,
ask: 'The rejected input was actually valid and our validation is wrong',
keep: v => v.value && v.confidence > 0.85,
})
拒绝 [email protected] 的验证器会返回与拒绝 notanemail 相同的 400。validation-bug 读取被拒绝的值和原因，并保留原因错误的事件。
判断 agent 回合
evlog/eve 会为每个 agent 回合发出一个 method: 'EVE' 的宽事件。抽样的 LLM-as-judge 只读取几个百分点；signal 会读取每个回合。事件形状请参见 eve。
const isTurn = (e: Record<string, unknown>) =>
e.method === 'EVE' && typeof (e.eve as { turnId?: string } | undefined)?.turnId === 'string'
export const turnResolved = defineSignal({
name: 'turn-resolved',
when: isTurn,
ask: 'The agent accomplished what the user asked for in this turn',
})
export const turnLooping = defineSignal({
name: 'turn-looping',
when: isTurn,
ask: 'The agent repeated the same tool call without making progress',
keep: v => v.value && v.confidence > 0.85,
})
export const turnOffScript = defineSignal({
name: 'turn-off-script',
when: isTurn,
ask: 'The agent took an action the user did not ask for',
keep: v => v.value && v.confidence > 0.85,
})
标记审计操作和不稳定任务
export const auditReview = defineSignal({
name: 'audit-review',
when: e => typeof e.audit === 'object' && e.audit !== null,
ask: 'This action deserves a human review',
criteria: {
true: 'Privilege change, bulk deletion, export of personal data, action outside business hours by a non-admin',
false: 'Routine self-service action',
},
keep: v => v.value && v.confidence > 0.8,
})
export const jobFlaky = defineSignal({
name: 'job-flaky',
when: e => typeof e.operation === 'string' && ((e.retries as number | undefined) ?? 0) > 0,
ask: 'The job only succeeded because it retried, and the underlying cause is still there',
})
审计事件携带参与者、操作和目标；风险来自它们的组合，audit-review 提供一个审核队列，而不是为每种组合编写一条规则。
其他适合的问题
first-seen（此错误对该路由来说是新的吗）、user-impact（无影响、降级、阻塞）、owner（哪个团队应该处理）、known-error（根据你自己的 why 和 fix 判断属于哪个目录项）、支持对话中的 frustration、会话事件中的 did-succeed。
不适合的问题
- 机密或 PII 检测。 把值发送给模型，再询问它是否为机密本身就是泄露。请使用脱敏。
- 通知值班人员。 永远不要根据概率触发通知。
severity负责排列优先阅读的内容；根据status和durationMs设置的告警才决定叫醒谁。 - 将对抗性内容作为单一判定。 能够被提问的模型也能够被诱导。
- 数值问题。 决策模型返回选项分布，而不是数字。请在处理程序中计算并记录数字。