Skip to main content

​
songbird-inference

Shared ML inference infrastructure for Songbird. Provides a trait-based backend abstraction, ONNX Runtime session management, execution provider selection, model weight download/cache/verification, and progress reporting.

​
ONNX Runtime Dynamic Loading

This crate uses ort with the load-dynamic feature. The ONNX Runtime shared library (libonnxruntime.dylib / .so / .dll) is loaded at runtime, not linked at compile time. This means:
  1. The app compiles without the dylib present.
  2. The dylib must be discoverable at runtime for inference to work.
  3. If the dylib isn’t found, the app still runs — stem separation is simply unavailable (graceful degradation).

​
Setup for Development

This places the library in rust/target/onnxruntime/. The ort_init module finds it via the sibling-of-target search path (exe_dir/../onnxruntime/).

​
Setup for Production Builds

The tauri.onnxruntime.json overlay adds the dylib directory as a bundled resource. On macOS it lands in Contents/Resources/, on Linux/Windows next to the binary.

​
Search Order (ort_init::resolve_dylib_path)

  1. ORT_DYLIB_PATH environment variable (explicit override)
  2. Same directory as the running executable
  3. macOS: ../Frameworks/ and ../Resources/ relative to executable (app bundle)
  4. exe_dir/../onnxruntime/ — dev workflow (fetch-onnxruntime drops the dylib here, sibling of target/{debug,release}/)
  5. Tauri resource directory (if provided)
  6. System library paths (/usr/local/lib, /opt/homebrew/lib, etc.)

​
Supported Platforms

​
Version Compatibility

The ort crate version (2.0.0-rc.11) requires ONNX Runtime >= 1.23.x. The fetch script downloads 1.23.2 (latest patch in the minimum-required line). Using an older version causes a hard failure at ort::init_from with a clear “not compatible” error. When bumping ort, re-check the version constraint in ort_init::ORT_VERSION and the fetch script.

​
Public API

  • InferenceBackend trait — uniform interface for ONNX, llama.cpp, MLX, and remote backends
  • BackendRegistry — routes inference requests by ModelCapability
  • SamplingParams / GeneratedToken — text generation parameters and streaming output
  • InferenceSession — wraps ort::Session with provider management
  • SessionConfig / ExecutionProvider — configuration types
  • ModelStore — model weight download, caching, verification, and local model listing
    • ensure_model() — download + SHA-256 verify
    • download_model_no_verify() — generic download for arbitrary GGUF URLs (no checksum)
    • list_local_models() → Vec<LocalModelInfo> — scans cache for GGUF files and HF repo dirs
    • list_cached() — returns paths for all cached ONNX and GGUF files
  • ModelManifest — describes a downloadable model (URL, checksum, size)
  • LocalModelInfo — metadata for a locally available model (id, name, backend, path, size, status)
  • ProgressSink trait — progress callbacks for downloads and inference
  • MiniLmEmbedder — text → 384-dim f32 embedding (requires embeddings feature)
  • OnnxBackend — InferenceBackend impl with tensor compute + embedding support
  • ort_init — dynamic library discovery and initialization

​
Feature flags

  • embeddings: enables MiniLmEmbedder and the tokenizers dependency. Used by songbird-agent for prompt classification.
Depended on by songbird-separator (stem separation), songbird-agent (LLM pipeline and prompt classification), and sync-engine ml.* commands. Lives in ml/ group — not core/ (too many external deps) and not integration/ (runs locally, not an external service bridge).

​
Crate Structure