> ## Documentation Index
> Fetch the complete documentation index at: https://songbird.studiocollective.xyz/llms.txt
> Use this file to discover all available pages before exploring further.

# Ai copilot

# AI Copilot Architecture

> For humans and LLMs contributing to Songbird's AI features.

## Overview

Songbird's AI copilot is a **realtime music production assistant** that lives inside the DAW. It routes user messages to specialized agents, assembles context from the current project state, and provides suggestions driven by DAW events and learned user preferences.

The architecture is organized into six subsystems:

```
react_ui/src/lib/ai/
├── core/           — Copilot orchestrator + intent router
├── agents/         — Specialized agents (MixEngineer, Composer, SoundDesigner)
├── hooks/          — Event-driven copilot behaviors (DAW event → suggestion)
├── instincts/      — Per-user/per-project learned preferences
├── context/        — Tiered context assembly (hot/warm/cold)
├── pipeline/       — Generate-then-refine two-pass pipeline
├── timeline/       — Timeline-driven pre-computation
├── providers/      — LLM provider abstraction (Gemini)
├── tools/          — Tool declarations and executor
├── cache/          — Context caching for large prompts
├── bird/           — .bird notation utilities
├── routing/        — Intent classification
├── docs/           — Prompt templates and documentation
├── types.ts        — Shared types (Intent, ModelTier, etc.)
└── index.ts        — Barrel exports
```

***

## Core Pipeline

### Request Flow

```
User message
  → Intent classification (LLM + regex fallback)
  → Agent routing (intent → specialized agent)
  → Tiered context assembly (hot/warm/cold)
  → Instincts injection (learned preferences)
  → LLM call (streaming, with tools)
  → [Optional] Refinement pass (for creative bird_edit)
  → Result
```

### Intent Types

| Intent         | Description                  | Agent         | Model Tier   |
| -------------- | ---------------------------- | ------------- | ------------ |
| `bird_edit`    | Compose, edit notes, arrange | Composer      | balanced/pro |
| `mixer`        | Volume, pan, sends, routing  | MixEngineer   | fast         |
| `mix_feedback` | Mixing advice and analysis   | MixEngineer   | balanced     |
| `set_param`    | Plugin parameter changes     | Plugin Editor | fast         |
| `bpm`          | Tempo changes                | General       | fast         |
| `lyria`        | AI audio generation          | General       | balanced     |
| `shortcut`     | Direct tool invocation       | (shortcut)    | —            |
| `chat`         | General conversation         | General       | balanced     |

### Chat Modes

* **`copilot`** — Full tool-use mode. Agent can call tools to modify the project.
* **`advisor`** — Read-only conversational mode. No tools, just musical guidance.

***

## Event-Driven Hooks

**File:** `hooks/copilot-hooks.ts` + `hooks/store-integration.ts`

The hook system maps DAW events to copilot behaviors. Hooks are async, non-blocking, and never touch the audio thread.

### Hook Events

| Event                  | Fires When                     | Example Behavior             |
| ---------------------- | ------------------------------ | ---------------------------- |
| `onPlaybackStart`      | Transport starts               | Prepare timeline suggestions |
| `onPlaybackStop`       | Transport stops                | Offer mixing tips            |
| `onSectionBoundary`    | Playback crosses section       | Suggest transitions          |
| `onBpmChange`          | BPM changed                    | Adjust rhythm suggestions    |
| `onScaleChange`        | Key/scale changed              | Update harmonic context      |
| `onTrackAdded`         | New track created              | Suggest instruments/FX       |
| `onTrackRemoved`       | Track deleted                  | —                            |
| `onMixerIdle`          | No mixer changes for N seconds | Offer mixing feedback        |
| `onPatternEdit`        | Notes edited in MIDI editor    | Suggest variations           |
| `onGenerationComplete` | AI generation finishes         | Suggest refinements          |

### Architecture

```
Zustand store change
  → store-integration.ts (subscriber, edge detection)
  → fireHook(event, context)
  → copilot-hooks.ts (throttle check, find handlers)
  → handler(context) → HookSuggestion[]
  → suggestion queue (priority, TTL, max size)
  → onSuggestions callback → UI
```

**Key design decisions:**

* Hooks are throttled per event type (configurable `MIN_INTERVAL_MS`) to prevent storms
* Suggestion queue has TTL — stale suggestions are culled automatically
* The `initialized` flag inside the transport subscriber skips the initial C++ hydration diff
* Store subscriptions use dependency injection (passed as callbacks) to avoid circular imports

### Registering Custom Hooks

```typescript theme={null}
import { registerHook } from '@/lib/ai';

registerHook('onSectionBoundary', {
  id: 'suggest-transition',
  priority: 5,
  handler: async (ctx) => [{
    type: 'transition',
    title: 'Add transition fill',
    description: `Transition from ${ctx.payload.prevSection} to ${ctx.payload.newSection}`,
    confidence: 0.7,
  }],
});
```

***

## Per-User/Per-Project Instincts

**File:** `instincts/instinct-engine.ts`

Tracks learned user preferences with confidence scoring. Inspired by ECC's continuous learning pattern.

### Instinct Categories

| Category            | What It Tracks                           |
| ------------------- | ---------------------------------------- |
| `velocity_range`    | Preferred velocity ranges per instrument |
| `voicing_style`     | Open vs closed voicings, inversions      |
| `rhythm_density`    | Sparse vs dense patterns                 |
| `plugin_preference` | Preferred plugins per role               |
| `genre_tendency`    | Detected genre patterns                  |
| `scale_preference`  | Preferred scales/modes                   |
| `mix_style`         | Mixing tendencies                        |
| `arrangement_style` | Arrangement patterns                     |

### Confidence Mechanics

* **Initial confidence:** 0.5
* **On acceptance:** +0.1 (capped at 0.95)
* **On rejection:** -0.15 (floor at 0.1)
* **Prompt threshold:** Only instincts >= 0.3 confidence are injected into prompts
* **Global promotion:** Instincts appearing in 2+ projects with >= 0.6 confidence are promoted to global scope

### Storage

JSON-serialized via juceBridge under key `songbird-instincts`. Debounced saves (3s).

### Pattern Key Matching

When `observe()` is called with a `patternKey`, it's stored as `pattern._key` in the instinct object. Future `observe()` calls match by `category + _key`, enabling stable reinforcement across sessions without needing exact pattern JSON equality.

***

## Tiered Context Loading

**File:** `context/tiered-context.ts`

Instead of dumping the entire project state on every LLM call, context is assembled in tiers with per-intent token budgets.

### Tiers

| Tier     | Content                                                             | When Included    |
| -------- | ------------------------------------------------------------------- | ---------------- |
| **Hot**  | Transport state, recent actions (last 30s), active instincts        | Always           |
| **Warm** | Track list with instruments/FX, arrangement structure, mixer levels | Most intents     |
| **Cold** | Full .bird file, historical edits                                   | Only `bird_edit` |

### Token Budgets by Intent

| Intent         | Budget | Tiers             |
| -------------- | ------ | ----------------- |
| `shortcut`     | 200    | hot               |
| `bpm`          | 500    | hot               |
| `mixer`        | 1500   | hot + warm        |
| `set_param`    | 2000   | hot + warm        |
| `lyria`        | 1500   | hot + warm        |
| `chat`         | 3000   | hot + warm        |
| `mix_feedback` | 4000   | hot + warm        |
| `bird_edit`    | 6000   | hot + warm + cold |

### Assembly Logic

1. Always include hot context
2. Add warm blocks if required by intent OR if \< 50% of budget used
3. Add cold blocks if required by intent OR if \< 30% of budget used
4. Each block is individually checked against remaining budget before inclusion
5. `tiersUsed` accurately reports which tiers had blocks actually included

***

## Generate-Then-Refine Pipeline

**File:** `pipeline/generate-refine.ts`

A two-pass pipeline for creative content: let the model be creative first, then evaluate against musical constraints.

### When It Triggers

Only for `bird_edit` intent with creative keywords: compose, write, create, fill, extend, generate, add, build, arrange, orchestrate, harmonize, improvise.

### Pass 1: Generate (Creative)

Standard copilot call — unconstrained generation using the balanced/pro model.

### Pass 2: Refine (Evaluation)

A fast-model call that evaluates the generated content against:

| Check              | What It Catches                                      |
| ------------------ | ---------------------------------------------------- |
| `key_mismatch`     | Notes outside the project's key signature            |
| `scale_violation`  | Notes outside the active scale                       |
| `rhythm_clash`     | Rhythmic patterns that conflict with existing tracks |
| `density_mismatch` | Too dense or too sparse relative to context          |
| `range_issue`      | Notes outside playable range for the instrument      |
| `style_mismatch`   | Doesn't match user's instinct preferences            |

### Output

```typescript theme={null}
interface PipelineResult {
  finalContent: string;    // refined content (or original if no issues)
  wasRefined: boolean;     // whether refinement changed anything
  issues: RefinementIssue[];  // what was found and fixed
  summary: string;         // human-readable summary
}
```

***

## Timeline-Driven Pre-Computation

**File:** `timeline/pre-compute.ts`

Uses playback position and arrangement structure to predict what the user needs next and pre-generate suggestions before they're needed.

### How It Works

```
Transport position update (during playback)
  → Calculate bars remaining in current section
  → If within lookAheadBars (default: 4) of section boundary
  → Pre-compute suggestions for the next section
  → Cache with TTL (default: 60s)
  → Available via getPreComputedSuggestions() when user reaches that section
```

### Suggestion Types

| Type             | When Generated                         |
| ---------------- | -------------------------------------- |
| `transition`     | Approaching a section boundary         |
| `continuation`   | Current section is about to loop       |
| `variation`      | User has been in a section for a while |
| `mix_adjustment` | Section change may need mix tweaks     |

### Configuration

```typescript theme={null}
interface PreComputeConfig {
  lookAheadBars: number;      // default: 4
  maxConcurrentJobs: number;  // default: 2
  suggestionTtlMs: number;    // default: 60000
  enabled: boolean;           // default: true
}
```

### Cache Invalidation

* Suggestions are invalidated when a section is edited (`invalidateSection()`)
* TTL-based expiry for stale suggestions
* Manual `cancelAll()` available for cleanup

***

## Integration Points

### copilot.ts (Main Orchestrator)

The copilot `send()` method integrates the subsystems:

* **Step 3b**: After building the agent/standard system prompt, appends tiered context from `assembleTieredContext()` (transport + mixer + instincts)
* **Step 7**: After generation, checks `shouldUseRefine()` and optionally runs the refinement pass

### store.ts (Store Subscriptions)

Hook integration is initialized via dynamic import to avoid circular dependencies:

```typescript theme={null}
let _copilotHooksUnsub: (() => void) | null = null;
import('@/lib/ai/hooks/store-integration').then(({ initCopilotHooks }) => {
  _copilotHooksUnsub = initCopilotHooks(
    (listener) => useTransportStore.subscribe(listener),
    (listener) => useMixerStore.subscribe(listener),
    () => useTransportStore.getState(),
    () => useMixerStore.getState(),
  );
});
```

HMR cleanup uses a dedicated module-level variable (not `_unsubs` array) to avoid race conditions with async imports.

### index.ts (Barrel Exports)

All subsystems are exported from `@/lib/ai`:

```typescript theme={null}
// Hooks
import { registerHook, fireHook, onSuggestions } from '@/lib/ai';

// Instincts
import { observeInstinct, acceptInstinct, rejectInstinct } from '@/lib/ai';

// Context
import { assembleContext } from '@/lib/ai';

// Pipeline
import { shouldUseRefine, buildRefinePrompt } from '@/lib/ai';

// Pre-compute
import { onPositionUpdate, getPreComputedSuggestions } from '@/lib/ai';
```

***

## Wiring Status

The subsystems are at different stages of integration:

| Subsystem         | Module               | Wired Into                  | Status                                             |
| ----------------- | -------------------- | --------------------------- | -------------------------------------------------- |
| Hooks             | copilot-hooks.ts     | store.ts subscriptions      | Active (fires on DAW events)                       |
| Instincts         | instinct-engine.ts   | copilot.ts prompt injection | Scaffolding (observe/accept/reject not called yet) |
| Tiered Context    | tiered-context.ts    | copilot.ts Step 3b          | Active (assembles on every request)                |
| Generate-Refine   | generate-refine.ts   | copilot.ts Step 7           | Active (triggers for creative bird\_edit)          |
| Pre-Compute       | pre-compute.ts       | store.ts transport updates  | Scaffolding (compute handler not wired to LLM)     |
| Store Integration | store-integration.ts | store.ts dynamic import     | Active (fires hooks, feeds pre-compute)            |

**Next steps to fully wire:**

1. Call `observe()` / `accept()` / `reject()` on instincts from copilot response handlers
2. Wire `setComputeHandler()` to call the copilot for pre-generation
3. Track `recentActions` and pass them to `assembleContext()`
4. Replace `'current-project'` placeholder with actual project ID from .bird filename
5. Add UI for displaying hook suggestions to the user

***

## Key Files

| File                           | Role                                                          |
| ------------------------------ | ------------------------------------------------------------- |
| `core/copilot.ts`              | Main orchestrator — routes, builds prompt, calls LLM, refines |
| `core/router.ts`               | Intent classification (LLM + regex fallback)                  |
| `hooks/copilot-hooks.ts`       | Hook registry, throttling, suggestion queue                   |
| `hooks/store-integration.ts`   | Zustand → hook event wiring                                   |
| `instincts/instinct-engine.ts` | Learned preferences with confidence scoring                   |
| `context/tiered-context.ts`    | Hot/warm/cold context assembly                                |
| `pipeline/generate-refine.ts`  | Two-pass creative generation + evaluation                     |
| `timeline/pre-compute.ts`      | Predictive suggestion pre-generation                          |
| `agents/base.ts`               | Agent registry                                                |
| `agents/mix-engineer.ts`       | Mixing agent                                                  |
| `agents/composer.ts`           | Composition agent                                             |
| `agents/sound-designer.ts`     | Sound design agent                                            |
| `providers/gemini.ts`          | Gemini LLM provider                                           |
| `tools/executor.ts`            | Tool execution system                                         |
| `types.ts`                     | Shared types                                                  |
| `index.ts`                     | Barrel exports                                                |