> ## Documentation Index
> Fetch the complete documentation index at: https://songbird.studiocollective.xyz/llms.txt
> Use this file to discover all available pages before exploring further.

# Webgpu

# WebGPU Rendering — Invariants

> Governs the GPU rendering layer: the renderer entry point in
> `react_ui/src/lib/gpu/GpuCanvas.tsx` and the surfaces it hosts
> (`arrangement/meshes/`, `midi-editor/meshes/`, `visualizer/scenes/`,
> `node-graph/GpuLayer.tsx`). Read this before adding a new GPU
> surface or shader.

The rendering pipeline runs on **WebGPU** (primary) and **WebGL2**
(fallback). Three.js + TSL bridge the two; this spec defines the
rules that keep that bridge intact.

***

## Invariants

These are load-bearing. Violating one is a critical bug even if the
WebGPU path looks fine.

### 1. Single renderer entry point

`react_ui/src/lib/gpu/GpuCanvas.tsx` is the **only** place that
constructs a renderer. The renderer is always `WebGPURenderer`
from `three/webgpu`, which transparently picks a `WebGPUBackend`
or a `WebGLBackend` under the hood. All GPU scenes mount inside
`<GpuCanvas>`.

* **Why.** The renderer factory enforces backend selection, async
  init, fallback logic, DPR setup, and color-space configuration in
  one place. Bypassing it means each surface re-derives those
  choices and they drift. Using a single renderer class (rather
  than the legacy `WebGLRenderer` for WebGL and `WebGPURenderer`
  for WebGPU) is what makes `NodeMaterial` and the auto-translation
  of legacy materials (`MeshBasicMaterial` → `MeshBasicNodeMaterial`)
  work uniformly on both paths.
* **How to apply.** New GPU surface → mount `<GpuCanvas>` and put the
  scene inside. Do not import `WebGPURenderer` or the legacy
  `WebGLRenderer` anywhere else.

### 2. Stay in the dual-backend TSL envelope

The features below have **no WebGL2 equivalent**. They cause silent
failures on the fallback path.

* Compute shaders (`Fn` with `compute()`, `.computeAsync()`)

* Storage buffers, storage textures

* Indirect draws

* Timestamp queries

* **Why.** Three.js will throw at material/pipeline compile, or the
  feature will no-op silently. Users on the fallback see broken UI
  with no clear error path.

* **How to apply.** If you need one of these, opt-in explicitly per
  surface (Pattern: WebGPU-only features). Keep a JS/CPU fallback
  alive for the WebGL2 path. Enforced by the lint guard.

### 3. All custom shaders use TSL, not GLSL/WGSL strings

`NodeMaterial` + TSL only. No `ShaderMaterial`, no
`RawShaderMaterial`, no inline `vertexShader: \`...\`\` strings in new
code.

* **Why.** TSL compiles to both WGSL and GLSL automatically. Raw
  shader source is one-backend-only and forces dual codepaths.
* **How to apply.** New shader → write it as TSL nodes from
  `three/tsl`. `ClipInstances.tsx` is the reference for SDF + per-clip
  attribute patterns; copy from there.

### 4. Pixel parity between backends

Both backends must produce visually-equivalent output: same colors,
same anti-aliasing, same alpha compositing, same DPR crispness.

* **Why.** The fallback isn't a "degraded mode" — it's a real path
  some users live on (Linux WebKitGTK, older macOS, driver-blocklisted
  GPUs). Quality drift on either backend is a bug.
* **How to apply.** Every PR that touches a GPU surface file must be tested
  on both `VITE_GPU_BACKEND=webgpu` and `VITE_GPU_BACKEND=webgl`. CI
  has no GPU; this is a human-eyeball check.

### 5. Per-frame work stays on the GPU

Don't rebuild `BufferGeometry` or upload large buffers per frame.
Per-frame state goes through uniforms, instance attributes, or the
camera/projection matrix — not full geometry rebuilds.

* **Why.** Geometry rebuilds are the WebGL2 stutter source and waste
  WebGPU performance. The point of GPU-side rendering is that the
  data lives there between frames.
* **How to apply.** Geometry `useMemo` deps change only on scene-
  shape changes (clip add/remove, edit). Scroll/zoom drives the
  camera. Playhead position is a uniform, not a vertex.

### 6. Frame loop is `demand` by default

Use `frameloop="demand"` on `<GpuCanvas>` and call `invalidate()` only
when the scene changes. Use `"always"` only for genuinely continuous
animation (visualizer, scrolling).

* **Why.** "Always" wastes battery and CPU for static scenes. The
  demand-driven loop is half the reason the GL port doesn't melt
  laptops.
* **How to apply.** Default to `demand`. If you reach for `always`,
  ask whether you can drive the redraw from input events instead.

***

## Patterns

### Writing a new GPU surface

1. Mount under `<GpuCanvas frameloop="demand|always" transparent>`.
2. Build geometry inside `useMemo`, keyed on scene-shape inputs.
3. Use built-in `<meshBasicMaterial>` etc. when you can; reach for
   `NodeMaterial` + TSL only when you need custom shading.
4. Per-frame state goes through `useFrame` updates to uniforms or
   the camera, not geometry rebuilds.
5. Verify on both backends before merging.

### Writing a custom shader

1. Build the material with `NodeMaterial` from `three/webgpu`.
2. Express vertex/fragment logic as TSL nodes:
   * Attributes: `attribute('aFoo', 'vec2')`
   * Uniforms: `uniform(value)` or `uniform(value, 'float')`
   * Math: `add`, `sub`, `mul`, `length`, `dot`, `smoothstep`, etc.
3. Set `material.colorNode` for the fragment color output.
4. Apply transparency / depth-test / side flags directly on the
   material.
5. Test side-by-side on both backends — colors and alpha
   compositing are the most common drift points.

### WebGPU-only features (the opt-in path)

If you genuinely need compute, storage, etc.:

1. Detect the active backend via `renderer.backend.type` (or the env
   override).
2. On WebGPU, use the compute/storage path.
3. On WebGL, fall through to a pre-existing CPU/JS path that
   produces the same visual result. The CPU path is mandatory, not
   optional.
4. Document the opt-in here as a new Decision entry, dated, with the
   surface name and the perf justification.

### Forbidden patterns (lint-guarded)

The following are blocked at validation time:

* `new WebGLRenderer(` or `new WebGPURenderer(` outside
  `GpuCanvas.tsx`
* `new ShaderMaterial(` — use NodeMaterial instead
* `new RawShaderMaterial(` — never
* `compute(`, `.computeAsync(`, `storage(`, `storageObject(` — opt-in
  required
* Inline `vertexShader:` / `fragmentShader:` strings in new code

Single-line opt-out: `// WEBGPU-OPT-IN: <reason>` allows the next
line through the guard. Force the conversation; don't bypass
silently.

***

## Decisions

### D1 — Default backend is WebGPU with WebGL2 fallback (2026-05-06)

`GpuCanvas` always constructs a `WebGPURenderer` and lets it pick
its backend: `WebGPUBackend` by default, `WebGLBackend` when
`forceWebGL: true` is passed (set when the user picks `webgl`) or
when our explicit fallback fires (see D5).

**Why.** WebGPU has lower per-frame CPU overhead and unblocks
compute-shader use cases (waveform mip generation, real-time
spectral analysis). The WebGL2 fallback covers Linux WebKitGTK,
older macOS, and driver-blocklisted GPUs. Using a single renderer
class with a swappable backend (rather than two separate renderer
classes) is what keeps NodeMaterial and TSL working on both paths.

**How to apply.** Don't assume one backend; respect the override
layers (`localStorage.GPU_BACKEND` runtime, `VITE_GPU_BACKEND` build).
Don't construct the legacy `WebGLRenderer` — it doesn't share the
node pipeline.

### D2 — CPU-side waveform polys, not compute mip generation (2026-05-06)

Waveform peak data is decimated in JS
(`react_ui/src/components/panels/arrangement/renderer/peakMipmap.ts`,
a power-of-2 mipmap of `Float32Array` peaks) and emitted per-frame as
filled polygon triangles in `WaveformMesh.tsx`, rendered with a
built-in material (no custom shader, no texture sampler).

**Why.** Polygon-tris render identically on both backends with zero
backend-specific code. GPU-side mip generation via compute would
unlock denser-than-pixel detail, but needs a CPU fallback anyway and
the current path isn't a perf bottleneck.

**How to apply.** Don't migrate waveform mip generation to compute,
and don't move the peak data into a `DataTexture` + fragment-sampler
pipeline, without an explicit opt-in (see Pattern: WebGPU-only
features) and a real perf justification.

### D3 — DOM overlay + CanvasTexture for text, not MSDF (2026-05-06)

Chrome text (panel headers, tooltips) lives in the DOM. In-scene
text (ruler labels, badges) uses `CanvasTexture` per
`RulerLabelsMesh.tsx` / `BadgeMesh.tsx`. MSDF is not adopted.

**Why.** MSDF adds a font-atlas pipeline and a custom shader for a
problem that doesn't exist at current zoom levels. CanvasTexture
works on both backends and matches CSS-rendered text.

**How to apply.** New labels follow the existing pattern. Reach for
MSDF only if a specific surface needs sharp scaling beyond what
CanvasTexture delivers, and document the exception here.

### D4 — Hit-testing stays CPU-side (2026-05-06)

Clip/note picking uses bounding-box math in
`react_ui/src/components/panels/arrangement/interaction/hitTest.ts`,
not GPU framebuffer readback.

**Why.** Framebuffer readback stalls the GPU pipeline on both
backends, and the math is cheap. WebGPU's compute path could change
this analysis later, but not yet.

**How to apply.** New hit-test paths use the same CPU-side approach.

### D5 — Explicit WebGL fallback, not just three's `getFallback` (2026-05-08)

`GpuCanvas.makeRenderer` runs an explicit two-stage fallback before
trusting `WebGPURenderer`'s built-in fallback path:

1. **Adapter probe.** Before construction, in `auto`/`webgpu` mode we
   call `navigator.gpu?.requestAdapter()`. A null result, a thrown
   exception, or missing `navigator.gpu` flips us to `forceWebGL: true`
   without ever attempting `WebGPUBackend`.
2. **Init retry.** Even with a valid adapter, `renderer.init()` is
   wrapped in `try`/`catch`. On failure we dispose the half-initialised
   renderer and construct a fresh `WebGPURenderer({ forceWebGL: true })`.

**Why.** Three's internal `getFallback` does not catch every Linux
WebKitGTK failure mode. The two states that bit users in testing:

* `navigator.gpu` exposed but `requestAdapter()` resolves null
  (WebKitGTK with WebGPU experimental flag off but the API binding
  shipped). Without the probe, this manifests as an unhandled rejection
  inside `init()` and a black canvas — the user has no recovery path.
* `requestAdapter()` succeeds, `requestDevice()` rejects (driver
  reports adapter, kernel module refuses device creation). Without the
  init-retry, same black canvas. With the retry, the user gets a
  working WebGL2 surface and a banner explaining the fallback.

The dev console banner reports the fallback reason
(`fellback: <reason>`) so support diagnostics are self-serve — the
string is the first thing to ask for when "the timeline is blank on
Linux."

**How to apply.** If you change the renderer construction sequence,
preserve both stages (probe + init-retry) and the banner reason
string. The `glCanvasFallback.test.ts` smoke test exercises the
probe's four states (no API, null, throw, success); add cases there
when you discover a new Linux failure mode.

**Per-platform init status.** Filled in as platform builds are run
against `localStorage.GPU_BACKEND=webgpu`:

| Target             | WebGPU init | Fallback fires | Notes                      |
| ------------------ | ----------- | -------------- | -------------------------- |
| macOS WKWebView    | *untested*  | *untested*     | Tauri `tauri-apps/wry` 0.x |
| Windows WebView2   | *untested*  | *untested*     | Edge channel-dependent     |
| Linux WebKitGTK    | *untested*  | *untested*     | Probe expected to reject   |
| Linux Chromium dev | *untested*  | *untested*     | For the headless harness   |

Anyone running on a fresh platform: replace the row with the
observed result and the console banner string. Don't leave *untested*
in place once a platform has shipped.

### D6 — Ideate backdrops are faithful three.js example ports (2026-06-13)

The Ideate audio-reactive backdrops
(`react_ui/src/components/panels/ideate/backdrops/`) are ports of the named
three.js examples and use whichever rendering path the original does:

* **Galaxy** (`webgpu_tsl_galaxy`), **Waves**, and **Lines**
  (`webgl_lines_colors`) → TSL node materials (`SpriteNodeMaterial` /
  `Line2NodeMaterial`) on the unified WebGPU renderer.
* **Geometry** (`webgl_buffergeometry`) → built-in `MeshPhongMaterial`.
* **Photos** → GLSL `ShaderMaterial` (the Curvy-showcase edge-warp effect),
  mounted on the **`webgl-legacy`** renderer island — same escape hatch as
  Flock.
* **Flock** (`webgl_gpgpu_birds`) → `GPUComputationRenderer` + a GLSL
  `ShaderMaterial`, so it mounts on the **`webgl-legacy`** renderer island
  (`<GpuCanvas renderer="webgl-legacy">`) — the same GLSL escape hatch the
  Visualizer uses. It's WebGL2, so it runs everywhere (no WebGPU gate).
* **Sprites** (`css3d_sprites`) → its own `CSS3DRenderer` (DOM in 3D),
  outside the GL canvas entirely.

**Animation runs on the GPU**, never a per-frame CPU geometry loop — geometry
is built once; only uniforms / transforms / the GPU sims update each frame
(Hard Rule #6). The earlier compute-shader Flock prototype was dropped in
favour of the faithful GPUComputationRenderer port. New GLSL `ShaderMaterial`
is allowed here only because it lives on the `webgl-legacy` island; anything
on the unified renderer still must use TSL.

### D7 — Blackout recovery is a four-rung ladder, not one handler (2026-06-24)

A hard sleep or battery-drain wake on macOS can darken the GL canvas, or
the *entire* WebView, and the right recovery differs by which. We layer
four mechanisms, innermost (cheapest, most local) first:

1. **GL context loss → repaint.** On `webglcontextrestored`, `GpuCanvas`'s
   `ContextLossHandler` calls r3f `invalidate()` so the scene repaints with
   the new context (`preventDefault()` on the lost event is what enables
   the restore at all). `lib/gpu/gpuRecovery.ts` is a repaint registry only.
   Note: it does **not** remount on WebGPU `device.lost` — an earlier
   version did, but WKWebView doesn't report intentional device teardown
   reliably as `reason === 'destroyed'`, so React StrictMode's
   mount/unmount/remount at launch slipped through and blanked the canvas.
   A genuinely dead surface is recovered by rung 3/4 (native reload), which
   is lossless, rather than by a canvas-level rebuild.
2. **App resume → repaint, and (only on a real wake) reflow.**
   `lib/recovery/webviewRecovery.ts`. On `visibilitychange`/`pageshow`/
   `focus` it does the *cheap, never-flashing* part — invalidate every
   canvas + heartbeat to reset the watchdog clock. The *heavier* compositor
   nudge (flush layout + synthetic `resize`) — for a page that's alive but
   showing a stale (blank) layer — fires **only on a detected system wake**,
   not on ordinary backgrounding: a heartbeat tick arriving ≥20 s late means
   the process was suspended (slept), whereas a backgrounded-but-running app
   keeps ticking. This is what keeps a normal app-switch from flashing the
   UI. It mutates **no** styles (the app-zoom transform on `<html>` survives).
3. **WebView render-process death → reload (native).**
   `webview_watchdog.rs` additively installs
   `-webViewWebContentProcessDidTerminate:` on wry's navigation-delegate
   class and reloads at the instant of death. It only *adds* the selector
   — never overrides wry's own methods (Hard Rule #1: that delegate is the
   wry IPC pipeline).
4. **WebView blackout backstop → heartbeat watchdog (native).** The main
   window pings `debug.heartbeat` \~every 2 s; a Rust thread reloads the
   WKWebView if those go silent for ≥15 s **while the window is
   occlusion-visible** (NSWindow `occlusionState`, not just `is_visible`).
   The 15 s threshold clears legitimate long main-thread blocks (Hard Rule
   \#6) and the occlusion gate excludes WebKit timer-throttling of a
   minimized / off-Space / fully-covered window — the two ways a heartbeat
   stalls without the WebView actually being dead. This catches the
   blackouts the delegate misses (frozen compositor, pre-paint death).

Rungs 3–4 only ever *reload* the WKWebView, never destroy/recreate it —
the object is wired to native state (Melodyne parent `contentView`,
`drawsBackground`) a rebuild would break, and all app state lives in the
sync engine, so a reload is lossless. The whole ladder is macOS-desktop
only; browser (WS/WASM) and popout/mobile windows skip the heartbeat.