Open app
Moonborn — Developers

Drift threshold tuning

Pick a workspace default, override per persona, sanity-check with a labeled sample. Concrete numbers for support, brand, creative surfaces.

The default engine.pipeline.drift_detection.threshold = 0.30 is a calibrated middle ground. Tighten or loosen per surface.

Suggested defaults

SurfaceThreshold
Customer support, regulated voice, healthcare0.20
General product chat, brand-adjacent0.30
Creative play, NPCs in low-stakes scenes0.45

Workspace default

await client.config.setItem({
  key: 'engine.pipeline.drift_detection.threshold',
  value: 0.20,
  scope: 'workspace',
  scopeId: 'ws_...',
});

Per-persona override

The runtime contract on each persona accepts a driftDetection.threshold field — overrides the workspace default for that persona only.

Validate before rolling out

Run a labeled sample through your candidate threshold:

const replies = await fetchReplyCorpus(50); // your last 50 prod replies
let truePositive = 0, falsePositive = 0;
for (const r of replies) {
  const drift = await client.consistency.scoreReply({
    personaId: r.personaId,
    reply: r.content,
  });
  const flagged = drift.score >= 0.25; // your candidate
  if (flagged && r.humanLabel === 'off-voice') truePositive++;
  if (flagged && r.humanLabel === 'in-voice') falsePositive++;
}

Targets: TP ≥ 80%, FP ≤ 10%. Adjust threshold up if FP is high, down if TP is low.

Common mistake

Tightening the threshold without action_on_alert = auto_recover — you'll just spam your QA queue. Pair every drop with the recovery path or accept the alert volume.

Related