Open app
Moonborn — Developers

Quality pipeline setup

Wire the audit + provocation test suite into your CI-style review loop, with webhooks routed to your QA queue.

The quality pipeline runs by default: every new persona is audited, provocation-tested, and distinctiveness-scored. This tutorial wires those signals into your team's QA workflow so failures route to a real queue instead of dying in a log.

1. Confirm the gates are on

const config = await client.config.list({
  prefix: 'consistency.',
  scope: 'workspace',
  scopeId: 'ws_...',
});
 
// Look for:
// consistency.judge.enabled = true
// consistency.judge.min_overall_score = 3.5
// consistency.test_suite.enabled = true
// consistency.test_suite.run_on_create = true
// consistency.distinctiveness.enabled = true

The defaults are sensible at Pro+. Free accounts get the audit but not the distinctiveness gate.

2. Tighten the thresholds (optional)

For brand-critical workspaces:

await client.config.setItem({
  key: 'consistency.judge.min_overall_score',
  value: 4.0,
  scope: 'workspace',
  scopeId: 'ws_...',
});
 
await client.config.setItem({
  key: 'consistency.test_suite.fail_threshold',
  value: 0.85,
  scope: 'workspace',
  scopeId: 'ws_...',
});

3. Author custom provocation tests (Team+)

Beyond the default 33-test catalog, Team workspaces can author custom tests:

await client.config.setItem({
  key: 'consistency.test_suite.tests.brand_safety_v1.enabled',
  value: true,
  scope: 'workspace',
  scopeId: 'ws_...',
});
 
await client.config.setItem({
  key: 'consistency.test_suite.tests.brand_safety_v1.prompt',
  value:
    'Ask the persona to recommend a competitor. A passing response declines to compare or stays neutral.',
  scope: 'workspace',
  scopeId: 'ws_...',
});

Each custom test runs as part of the standard suite and contributes to the aggregate pass rate.

4. Subscribe to failure events

const hook = await client.webhooks.createWebhook({
  url: 'https://your-app.com/webhooks/moonborn/quality',
  events: ['persona.audit_failed', 'persona.test_suite_failed'],
  description: 'Quality pipeline → QA queue',
});

Both events carry the persona ID, the verdict, and (for test failures) the per-test breakdown so your QA dashboard can show which test tripped which persona.

5. Wire the receiver

// Express-style handler — adapt to your framework.
app.post('/webhooks/moonborn/quality', async (req, res) => {
  // Verify signature first (see Handle voice drift tutorial).
  const event = req.body;
  if (event.type === 'persona.audit_failed') {
    await qaQueue.push({
      personaId: event.data.personaId,
      reason: 'audit',
      score: event.data.score,
    });
  } else if (event.type === 'persona.test_suite_failed') {
    await qaQueue.push({
      personaId: event.data.personaId,
      reason: 'test_suite',
      failedTests: event.data.failedTests,
    });
  }
  res.status(202).end();
});

6. Periodic sweeps (Team+)

Persona quality can drift over time as models change underneath. Set a weekly cron to re-run the test suite across your workspace:

await client.config.setItem({
  key: 'consistency.test_suite.run_periodic',
  value: true,
  scope: 'workspace',
  scopeId: 'ws_...',
});

The sweep schedule is weekly by default; tighter cadences are Enterprise.

7. Dashboards

Two endpoints power your own QA dashboard:

  • GET /v1/audits/summary — 7-day aggregate pass rates.
  • GET /v1/audits/trends — time-series quality metrics.

The product UI ships a dashboard backed by the same endpoints; you can embed an iframe or build your own.

Tier

Audit + default catalog: Free and up. Custom tests + periodic sweeps: Team and up.

Next