Eval (songbird-eval)
LLM evaluation for the Songbird copilot. The eval drives the real in-app
chat loop in-process — ml.chat against the production dispatch handlers
(classifier, model tiers, orchestrator, nudges, read-before-write guard,
splice handlers, deletion guard, receipts). There is no mocked loop: a
change to the chat pipeline is automatically a change to what gets
evaluated.
The crate lives at rust/crates/ml/songbird-eval; the prompt set and the
baseline project fixture are in its assets/.
Running
~/.tivra/settings.json (same as the app);
GEMINI_API_KEY overrides for the judge. Don’t run two evals concurrently
with one key — the judge rate-limits, and zeroed judgments are flagged
JUDGE-FAILED rather than silently scored.
Scoring (max 10 per prompt)
The judge defaults to
gemini-3-flash-preview (the Pro preview shares a
daily quota pool with the auto router’s Thinker tier) and requests
structured JSON output, with brace extraction as the parse fallback.
Creative categories are graded for musical coherence against the request’s
explicit constraints (“many valid realizations”) rather than against the
prompt’s notes; param-heavy categories (mix_control, sound_design,
fx_chain, external_plugin, plugin_management) are graded against a
measured state diff vs baseline instead of inferred plausibility.
Hard checks reported per run: structural violations (baseline tracks
destroyed → validity 0 — on explicit “delete/remove … track” prompts only
the track named in the prompt is exempt; any other missing track still
violates), assertions (assert specs in prompts.json: chord
brackets, note moved/removed/added, velocity and pitch bounds, plus
unchanged — read-only prompts must leave the bird text and every
track’s notes untouched — and anyNotesChanged {tracks} — at least one
listed track’s notes changed — all delta-based against the baseline state
snapshot, since real parsed state has one clip per section), and silent
completions (tools executed, no final user-facing message).
Prompts marked "holdout": true (ids 136+) are excluded from default
sweeps and run only via --holdout — a release-gate set that prompts and
tool descriptions must not be tuned against.
Each prompt runs --repeat times (default 3); the median run is kept and
the spread persisted. The report shows pass@1 (mean fraction of a
prompt’s non-failed runs scoring >= 6), pass^k (fraction of prompts
whose non-failed runs all scored >= 6), a high-variance list (repeat
spread >= 3), and median tool rounds overall and per category. With
--repeat 1 only pass@1 is printed.
Why in-process, not mocked
The previous JS harness (eval/run_eval.js, retired 2026-06-11)
re-implemented the loop, tools, and guards as mocks, and they drifted: the
mock taught the wrong chord syntax, couldn’t observe state destruction, and
never exercised intent routing. The in-process eval found two production
bugs in its first hour — a mixer intent misroute (“lower the kick volume”
edited note velocities) and a panic on malformed MCP trackId input that
poisoned the state mutex. The bar: if the eval can’t fail the way
production fails, it isn’t evaluating production. Historical JS-era reports
remain archived under eval/.