> ## Documentation Index
> Fetch the complete documentation index at: https://songbird.studiocollective.xyz/llms.txt
> Use this file to discover all available pages before exploring further.

# Eval (`songbird-eval`)

# Eval (`songbird-eval`)

LLM evaluation for the Songbird copilot. The eval drives the **real in-app
chat loop in-process** — `ml.chat` against the production dispatch handlers
(classifier, model tiers, orchestrator, nudges, read-before-write guard,
splice handlers, deletion guard, receipts). There is no mocked loop: a
change to the chat pipeline is automatically a change to what gets
evaluated.

The crate lives at `rust/crates/ml/songbird-eval`; the prompt set and the
baseline project fixture are in its `assets/`.

## Running

```bash theme={null}
# Full sweep against the in-app loop (Gemini tiers, auto-routed)
cargo run -p songbird-eval --release

# Specific prompts / explicit model tier
cargo run -p songbird-eval --release -- --ids 1,2,3 --model pro

# Repeats: each prompt runs 3 times by default (median kept, spread
# recorded) — a full default sweep is 3x the API calls. Use --repeat 1
# for a cheap smoke sweep (pass^k then needs repeats and is omitted).
cargo run -p songbird-eval --release -- --repeat 1

# Claude via the Anthropic API — same in-app loop/guards/nudges as the
# Gemini tiers, so it's the fair model A/B. Key: `anthropic-api-key` in
# ~/.tivra/settings.json or ANTHROPIC_API_KEY.
cargo run -p songbird-eval --release -- --model claude-sonnet-5
cargo run -p songbird-eval --release -- --pipeline v3-claude   # Opus plans, Sonnet executes, Haiku speed

# Claude Code backend (relays through the user's `claude` CLI — Claude
# Code's own harness, NOT the in-app loop; tool calls come back via
# songbird-mcp → the same dispatch handlers)
cargo build --release -p songbird-mcp
cargo run -p songbird-eval --release -- --model claude-code

# Regenerate a report, or compare two result sets
cargo run -p songbird-eval --release -- --report-only --results-dir crates/ml/songbird-eval/results/auto
cargo run -p songbird-eval --release -- --compare results/auto results/claude-code
```

The Gemini key resolves from `~/.tivra/settings.json` (same as the app);
`GEMINI_API_KEY` overrides for the judge. Don't run two evals concurrently
with one key — the judge rate-limits, and zeroed judgments are flagged
`JUDGE-FAILED` rather than silently scored.

## Scoring (max 10 per prompt)

| Dimension | Range | Source                                                                                                                                                                           |
| --------- | ----- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| validity  | 0–3   | deterministic — did a parse-clean edit actually land in real state                                                                                                               |
| relevance | 0–4   | LLM judge — capped at 2 if the run was silent (tools ran, no final message) or the final text contradicts the tool log (judged `summary_accuracy` 0, flagged `SUMMARY-MISMATCH`) |
| quality   | 0–3   | LLM judge — capped at 1 if the prompt's deterministic assertion failed                                                                                                           |

The judge defaults to `gemini-3-flash-preview` (the Pro preview shares a
daily quota pool with the auto router's Thinker tier) and requests
structured JSON output, with brace extraction as the parse fallback.
Creative categories are graded for musical coherence against the request's
explicit constraints ("many valid realizations") rather than against the
prompt's `notes`; param-heavy categories (mix\_control, sound\_design,
fx\_chain, external\_plugin, plugin\_management) are graded against a
measured state diff vs baseline instead of inferred plausibility.

Hard checks reported per run: **structural violations** (baseline tracks
destroyed → validity 0 — on explicit "delete/remove … track" prompts only
the track named in the prompt is exempt; any other missing track still
violates), **assertions** (`assert` specs in `prompts.json`: chord
brackets, note moved/removed/added, velocity and pitch bounds, plus
`unchanged` — read-only prompts must leave the bird text and every
track's notes untouched — and `anyNotesChanged {tracks}` — at least one
listed track's notes changed — all delta-based against the baseline state
snapshot, since real parsed state has one clip per section), and **silent
completions** (tools executed, no final user-facing message).

Prompts marked `"holdout": true` (ids 136+) are excluded from default
sweeps and run only via `--holdout` — a release-gate set that prompts and
tool descriptions must not be tuned against.

Each prompt runs `--repeat` times (default 3); the median run is kept and
the spread persisted. The report shows **pass\@1** (mean fraction of a
prompt's non-failed runs scoring >= 6), **pass^k** (fraction of prompts
whose non-failed runs *all* scored >= 6), a high-variance list (repeat
spread >= 3), and median tool rounds overall and per category. With
`--repeat 1` only pass\@1 is printed.

## Why in-process, not mocked

The previous JS harness (`eval/run_eval.js`, retired 2026-06-11)
re-implemented the loop, tools, and guards as mocks, and they drifted: the
mock taught the wrong chord syntax, couldn't observe state destruction, and
never exercised intent routing. The in-process eval found two production
bugs in its first hour — a mixer intent misroute ("lower the kick volume"
edited note velocities) and a panic on malformed MCP `trackId` input that
poisoned the state mutex. The bar: if the eval can't fail the way
production fails, it isn't evaluating production. Historical JS-era reports
remain archived under `eval/`.