AI Copilot Architecture

For humans and LLMs contributing to Songbird’s AI features.

Overview

Songbird’s AI copilot is a realtime music production assistant that lives inside the DAW. It routes user messages to specialized agents, assembles context from the current project state, and provides suggestions driven by DAW events and learned user preferences. The architecture is organized into six subsystems:

react_ui/src/lib/ai/
├── core/           — Copilot orchestrator + intent router
├── agents/         — Specialized agents (MixEngineer, Composer, SoundDesigner)
├── hooks/          — Event-driven copilot behaviors (DAW event → suggestion)
├── instincts/      — Per-user/per-project learned preferences
├── context/        — Tiered context assembly (hot/warm/cold)
├── pipeline/       — Generate-then-refine two-pass pipeline
├── timeline/       — Timeline-driven pre-computation
├── providers/      — LLM provider abstraction (Gemini)
├── tools/          — Tool declarations and executor
├── cache/          — Context caching for large prompts
├── bird/           — .bird notation utilities
├── routing/        — Intent classification
├── docs/           — Prompt templates and documentation
├── types.ts        — Shared types (Intent, ModelTier, etc.)
└── index.ts        — Barrel exports

Core Pipeline

Request Flow

User message
  → Intent classification (LLM + regex fallback)
  → Agent routing (intent → specialized agent)
  → Tiered context assembly (hot/warm/cold)
  → Instincts injection (learned preferences)
  → LLM call (streaming, with tools)
  → [Optional] Refinement pass (for creative bird_edit)
  → Result

Intent Types

Intent	Description	Agent	Model Tier
`bird_edit`	Compose, edit notes, arrange	Composer	balanced/pro
`mixer`	Volume, pan, sends, routing	MixEngineer	fast
`mix_feedback`	Mixing advice and analysis	MixEngineer	balanced
`set_param`	Plugin parameter changes	Plugin Editor	fast
`bpm`	Tempo changes	General	fast
`lyria`	AI audio generation	General	balanced
`shortcut`	Direct tool invocation	(shortcut)	—
`chat`	General conversation	General	balanced

Chat Modes

copilot — Full tool-use mode. Agent can call tools to modify the project.
advisor — Read-only conversational mode. No tools, just musical guidance.

Event-Driven Hooks

File: hooks/copilot-hooks.ts + hooks/store-integration.ts The hook system maps DAW events to copilot behaviors. Hooks are async, non-blocking, and never touch the audio thread.

Hook Events

Event	Fires When	Example Behavior
`onPlaybackStart`	Transport starts	Prepare timeline suggestions
`onPlaybackStop`	Transport stops	Offer mixing tips
`onSectionBoundary`	Playback crosses section	Suggest transitions
`onBpmChange`	BPM changed	Adjust rhythm suggestions
`onScaleChange`	Key/scale changed	Update harmonic context
`onTrackAdded`	New track created	Suggest instruments/FX
`onTrackRemoved`	Track deleted	—
`onMixerIdle`	No mixer changes for N seconds	Offer mixing feedback
`onPatternEdit`	Notes edited in MIDI editor	Suggest variations
`onGenerationComplete`	AI generation finishes	Suggest refinements

Architecture

Zustand store change
  → store-integration.ts (subscriber, edge detection)
  → fireHook(event, context)
  → copilot-hooks.ts (throttle check, find handlers)
  → handler(context) → HookSuggestion[]
  → suggestion queue (priority, TTL, max size)
  → onSuggestions callback → UI

Key design decisions:

Hooks are throttled per event type (configurable MIN_INTERVAL_MS) to prevent storms
Suggestion queue has TTL — stale suggestions are culled automatically
The initialized flag inside the transport subscriber skips the initial C++ hydration diff
Store subscriptions use dependency injection (passed as callbacks) to avoid circular imports

Registering Custom Hooks

import { registerHook } from '@/lib/ai';

registerHook('onSectionBoundary', {
  id: 'suggest-transition',
  priority: 5,
  handler: async (ctx) => [{
    type: 'transition',
    title: 'Add transition fill',
    description: `Transition from ${ctx.payload.prevSection} to ${ctx.payload.newSection}`,
    confidence: 0.7,
  }],
});

Per-User/Per-Project Instincts

File: instincts/instinct-engine.ts Tracks learned user preferences with confidence scoring. Inspired by ECC’s continuous learning pattern.

Instinct Categories

Category	What It Tracks
`velocity_range`	Preferred velocity ranges per instrument
`voicing_style`	Open vs closed voicings, inversions
`rhythm_density`	Sparse vs dense patterns
`plugin_preference`	Preferred plugins per role
`genre_tendency`	Detected genre patterns
`scale_preference`	Preferred scales/modes
`mix_style`	Mixing tendencies
`arrangement_style`	Arrangement patterns

Confidence Mechanics

Initial confidence: 0.5
On acceptance: +0.1 (capped at 0.95)
On rejection: -0.15 (floor at 0.1)
Prompt threshold: Only instincts >= 0.3 confidence are injected into prompts
Global promotion: Instincts appearing in 2+ projects with >= 0.6 confidence are promoted to global scope

Storage

JSON-serialized via juceBridge under key songbird-instincts. Debounced saves (3s).

Pattern Key Matching

When observe() is called with a patternKey, it’s stored as pattern._key in the instinct object. Future observe() calls match by category + _key, enabling stable reinforcement across sessions without needing exact pattern JSON equality.

Tiered Context Loading

File: context/tiered-context.ts Instead of dumping the entire project state on every LLM call, context is assembled in tiers with per-intent token budgets.

Tiers

Tier	Content	When Included
Hot	Transport state, recent actions (last 30s), active instincts	Always
Warm	Track list with instruments/FX, arrangement structure, mixer levels	Most intents
Cold	Full .bird file, historical edits	Only `bird_edit`

Token Budgets by Intent

Intent	Budget	Tiers
`shortcut`	200	hot
`bpm`	500	hot
`mixer`	1500	hot + warm
`set_param`	2000	hot + warm
`lyria`	1500	hot + warm
`chat`	3000	hot + warm
`mix_feedback`	4000	hot + warm
`bird_edit`	6000	hot + warm + cold

Assembly Logic

Always include hot context
Add warm blocks if required by intent OR if < 50% of budget used
Add cold blocks if required by intent OR if < 30% of budget used
Each block is individually checked against remaining budget before inclusion
tiersUsed accurately reports which tiers had blocks actually included

Generate-Then-Refine Pipeline

File: pipeline/generate-refine.ts A two-pass pipeline for creative content: let the model be creative first, then evaluate against musical constraints.

When It Triggers

Only for bird_edit intent with creative keywords: compose, write, create, fill, extend, generate, add, build, arrange, orchestrate, harmonize, improvise.

Pass 1: Generate (Creative)

Standard copilot call — unconstrained generation using the balanced/pro model.

Pass 2: Refine (Evaluation)

A fast-model call that evaluates the generated content against:

Check	What It Catches
`key_mismatch`	Notes outside the project’s key signature
`scale_violation`	Notes outside the active scale
`rhythm_clash`	Rhythmic patterns that conflict with existing tracks
`density_mismatch`	Too dense or too sparse relative to context
`range_issue`	Notes outside playable range for the instrument
`style_mismatch`	Doesn’t match user’s instinct preferences

Output

interface PipelineResult {
  finalContent: string;    // refined content (or original if no issues)
  wasRefined: boolean;     // whether refinement changed anything
  issues: RefinementIssue[];  // what was found and fixed
  summary: string;         // human-readable summary
}

Timeline-Driven Pre-Computation

File: timeline/pre-compute.ts Uses playback position and arrangement structure to predict what the user needs next and pre-generate suggestions before they’re needed.

How It Works

Transport position update (during playback)
  → Calculate bars remaining in current section
  → If within lookAheadBars (default: 4) of section boundary
  → Pre-compute suggestions for the next section
  → Cache with TTL (default: 60s)
  → Available via getPreComputedSuggestions() when user reaches that section

Suggestion Types

Type	When Generated
`transition`	Approaching a section boundary
`continuation`	Current section is about to loop
`variation`	User has been in a section for a while
`mix_adjustment`	Section change may need mix tweaks

Configuration

interface PreComputeConfig {
  lookAheadBars: number;      // default: 4
  maxConcurrentJobs: number;  // default: 2
  suggestionTtlMs: number;    // default: 60000
  enabled: boolean;           // default: true
}

Cache Invalidation

Suggestions are invalidated when a section is edited (invalidateSection())
TTL-based expiry for stale suggestions
Manual cancelAll() available for cleanup

Integration Points

copilot.ts (Main Orchestrator)

The copilot send() method integrates the subsystems:

Step 3b: After building the agent/standard system prompt, appends tiered context from assembleTieredContext() (transport + mixer + instincts)
Step 7: After generation, checks shouldUseRefine() and optionally runs the refinement pass

store.ts (Store Subscriptions)

Hook integration is initialized via dynamic import to avoid circular dependencies:

let _copilotHooksUnsub: (() => void) | null = null;
import('@/lib/ai/hooks/store-integration').then(({ initCopilotHooks }) => {
  _copilotHooksUnsub = initCopilotHooks(
    (listener) => useTransportStore.subscribe(listener),
    (listener) => useMixerStore.subscribe(listener),
    () => useTransportStore.getState(),
    () => useMixerStore.getState(),
  );
});

HMR cleanup uses a dedicated module-level variable (not _unsubs array) to avoid race conditions with async imports.

index.ts (Barrel Exports)

All subsystems are exported from @/lib/ai:

// Hooks
import { registerHook, fireHook, onSuggestions } from '@/lib/ai';

// Instincts
import { observeInstinct, acceptInstinct, rejectInstinct } from '@/lib/ai';

// Context
import { assembleContext } from '@/lib/ai';

// Pipeline
import { shouldUseRefine, buildRefinePrompt } from '@/lib/ai';

// Pre-compute
import { onPositionUpdate, getPreComputedSuggestions } from '@/lib/ai';

Wiring Status

The subsystems are at different stages of integration:

Subsystem	Module	Wired Into	Status
Hooks	copilot-hooks.ts	store.ts subscriptions	Active (fires on DAW events)
Instincts	instinct-engine.ts	copilot.ts prompt injection	Scaffolding (observe/accept/reject not called yet)
Tiered Context	tiered-context.ts	copilot.ts Step 3b	Active (assembles on every request)
Generate-Refine	generate-refine.ts	copilot.ts Step 7	Active (triggers for creative bird_edit)
Pre-Compute	pre-compute.ts	store.ts transport updates	Scaffolding (compute handler not wired to LLM)
Store Integration	store-integration.ts	store.ts dynamic import	Active (fires hooks, feeds pre-compute)

Next steps to fully wire:

Call observe() / accept() / reject() on instincts from copilot response handlers
Wire setComputeHandler() to call the copilot for pre-generation
Track recentActions and pass them to assembleContext()
Replace 'current-project' placeholder with actual project ID from .bird filename
Add UI for displaying hook suggestions to the user

Key Files

File	Role
`core/copilot.ts`	Main orchestrator — routes, builds prompt, calls LLM, refines
`core/router.ts`	Intent classification (LLM + regex fallback)
`hooks/copilot-hooks.ts`	Hook registry, throttling, suggestion queue
`hooks/store-integration.ts`	Zustand → hook event wiring
`instincts/instinct-engine.ts`	Learned preferences with confidence scoring
`context/tiered-context.ts`	Hot/warm/cold context assembly
`pipeline/generate-refine.ts`	Two-pass creative generation + evaluation
`timeline/pre-compute.ts`	Predictive suggestion pre-generation
`agents/base.ts`	Agent registry
`agents/mix-engineer.ts`	Mixing agent
`agents/composer.ts`	Composition agent
`agents/sound-designer.ts`	Sound design agent
`providers/gemini.ts`	Gemini LLM provider
`tools/executor.ts`	Tool execution system
`types.ts`	Shared types
`index.ts`	Barrel exports

Getting Started

User Guide

Core Concepts

AI & Copilot

Ai copilot

AI Copilot Architecture

Overview

Core Pipeline

Request Flow

Intent Types

Chat Modes

Event-Driven Hooks

Hook Events

Architecture

Registering Custom Hooks

Per-User/Per-Project Instincts

Instinct Categories

Confidence Mechanics

Storage

Pattern Key Matching

Tiered Context Loading

Tiers

Token Budgets by Intent

Assembly Logic

Generate-Then-Refine Pipeline

When It Triggers

Pass 1: Generate (Creative)

Pass 2: Refine (Evaluation)

Output

Timeline-Driven Pre-Computation

How It Works

Suggestion Types

Configuration

Cache Invalidation

Integration Points

copilot.ts (Main Orchestrator)

store.ts (Store Subscriptions)

index.ts (Barrel Exports)

Wiring Status

Key Files

​AI Copilot Architecture

​Overview

​Core Pipeline

​Request Flow

​Intent Types

​Chat Modes

​Event-Driven Hooks

​Hook Events

​Architecture

​Registering Custom Hooks

​Per-User/Per-Project Instincts

​Instinct Categories

​Confidence Mechanics

​Storage

​Pattern Key Matching

​Tiered Context Loading

​Tiers

​Token Budgets by Intent

​Assembly Logic

​Generate-Then-Refine Pipeline

​When It Triggers

​Pass 1: Generate (Creative)

​Pass 2: Refine (Evaluation)

​Output

​Timeline-Driven Pre-Computation

​How It Works

​Suggestion Types

​Configuration

​Cache Invalidation

​Integration Points

​copilot.ts (Main Orchestrator)

​store.ts (Store Subscriptions)

​index.ts (Barrel Exports)

​Wiring Status

​Key Files

AI Copilot Architecture

Overview

Core Pipeline

Request Flow

Intent Types

Chat Modes

Event-Driven Hooks

Hook Events

Architecture

Registering Custom Hooks

Per-User/Per-Project Instincts

Instinct Categories

Confidence Mechanics

Storage

Pattern Key Matching

Tiered Context Loading

Tiers

Token Budgets by Intent

Assembly Logic

Generate-Then-Refine Pipeline

When It Triggers

Pass 1: Generate (Creative)

Pass 2: Refine (Evaluation)

Output

Timeline-Driven Pre-Computation

How It Works

Suggestion Types

Configuration

Cache Invalidation

Integration Points

copilot.ts (Main Orchestrator)

store.ts (Store Subscriptions)

index.ts (Barrel Exports)

Wiring Status

Key Files