> ## Documentation Index
> Fetch the complete documentation index at: https://songbird.studiocollective.xyz/llms.txt
> Use this file to discover all available pages before exploring further.

# Songbird Headless DAW — Cloud Run Deployment

# Songbird Headless DAW — Cloud Run Deployment

Deploy the Songbird headless DAW engine to Google Cloud Run with GCS Fuse for project storage and spot instances for lower cost.

## Architecture

```
                        ┌─ X-API-Key header ─┐
                        │  or ?api_key= param │
                        └────────┬────────────┘
                                 │
┌────────────────────────────────▼────────────────────────────┐
│  Cloud Run Container (gen2, spot, concurrency=1)            │
│                                                              │
│  ┌───────────┐   ┌──────────────┐   ┌───────────┐          │
│  │   Nginx   │──▶│ Headless DAW │   │  Collab   │          │
│  │   :8080   │   │  WS :9090    │   │  WS :8765 │          │
│  │ (API key  │──▶│  HTTP :9091  │   │           │          │
│  │  + proxy) │   └──────────────┘   └───────────┘          │
│  │           │                                              │
│  │           │──▶┌──────────────┐                           │
│  │           │   │ Clone Svc    │                           │
│  │           │   │  HTTP :9092  │                           │
│  └───────────┘   └──────────────┘                           │
│       │                │                                     │
│       │         ┌──────┴──────┐   ┌──────────────┐          │
│       │         │  GCS Fuse   │   │  GCS Fuse    │          │
│       │         │ /data/proj  │   │ /data/repos  │          │
│       │         └──────┬──────┘   └──────┬───────┘          │
└───────│────────────────│────────────────│────────────────────┘
        │                │                │
        ▼                ▼                ▼
   daw.club         GCS Bucket       GCS Bucket
   (React UI)    (project files)  (gitea bare repos)
```

**Key properties:**

* **API key authentication** — All routes except `/healthz` require a valid `SONGBIRD_API_KEY` via `X-API-Key` header or `?api_key=` query param
* **Spot instances** — \~60-91% cheaper than standard Cloud Run
* **GCS Fuse** — Project files (.bird, audio samples) mounted as a filesystem
* **1 user per instance** — Each DAW session gets a dedicated container (concurrency=1)
* **Scale to zero** — No cost when idle
* **Session affinity** — Sticky WebSocket connections for the duration of a session
* **60-minute timeout** — Long-lived WebSocket sessions supported

## Launch Flow (daw\.club → VM)

When a user clicks **Launch** on a project at daw\.club:

1. daw\.club opens `https://vm.daw.club?repo=<clone_url>&api_key=<key>` in a new tab
2. The React UI loads and detects the `?repo=` query parameter
3. React UI calls `POST /api/clone` with `{ "url": "<clone_url>" }` — the clone service finds the bare repo on the GCS Fuse mount at `/data/repos/<owner>/<repo>.git` and does a fast local checkout to `/tmp/songbird-session/`
4. React UI connects to the headless DAW via WebSocket (`/ws`)
5. React UI calls `projectChoice("openRecent:<cloned_path>")` to load the project

**Why GCS Fuse?** Gitea's bare repos are stored in a GCS bucket (`songbird-gitea-repos`) that is mounted read-only in Cloud Run at `/data/repos`. The clone service reads directly from this mount — no network clone needed.

**Session isolation:** Each user gets their own Cloud Run container (`containerConcurrency: 1`). When the user disconnects, the container eventually idles and is destroyed — all checkout data in `/tmp` is wiped automatically. The next user always gets a fresh container.

### Gitea Setup (one-time)

On the Gitea VM (`git.daw.club`), configure Gitea to store repos on GCS:

```bash theme={null}
# 1. Create the GCS bucket
gcloud storage buckets create gs://songbird-gitea-repos --location=us-central1

# 2. Install GCS Fuse on the Gitea VM
sudo apt-get install gcsfuse

# 3. Mount the bucket
sudo mkdir -p /data/gitea-repos
sudo gcsfuse --implicit-dirs songbird-gitea-repos /data/gitea-repos

# 4. Move existing repos to the mount
sudo rsync -a /var/lib/gitea/data/gitea-repositories/ /data/gitea-repos/

# 5. Update Gitea config (/etc/gitea/app.ini)
#    [repository]
#    ROOT = /data/gitea-repos

# 6. Restart Gitea
sudo systemctl restart gitea

# 7. Grant Cloud Run service account read access
PROJECT_NUMBER=$(gcloud projects describe gen-lang-client-0380481603 --format="value(projectNumber)")
gcloud storage buckets add-iam-policy-binding gs://songbird-gitea-repos \
    --member="serviceAccount:${PROJECT_NUMBER}-compute@developer.gserviceaccount.com" \
    --role="roles/storage.objectViewer"
```

## Current Deployment State

| Item                  | Value                                                                              |
| --------------------- | ---------------------------------------------------------------------------------- |
| GCP Project ID        | `gen-lang-client-0380481603`                                                       |
| GCP Region            | `us-central1`                                                                      |
| Docker Image          | `us-central1-docker.pkg.dev/gen-lang-client-0380481603/songbird/songbird-headless` |
| Cloud Run Service     | `songbird-headless`                                                                |
| Cloud Run URL         | `https://songbird-headless-6kdcepjumq-uc.a.run.app`                                |
| GCS Bucket (projects) | `gs://songbird-projects-gen-lang-client-0380481603`                                |
| GCS Bucket (binary)   | `gs://songbird-daw-club-repos`                                                     |
| Binary path in GCS    | `gs://songbird-daw-club-repos/bin/<version>/SongbirdHeadless-release`              |

### Load Balancer (Public Access)

The GCP org policy blocks `allUsers` IAM binding, so a Global External Application Load Balancer
provides public access. Cloud Run's invoker IAM check is disabled (`--no-invoker-iam-check`),
and nginx handles API key authentication inside the container.

| Resource         | Name                                                             |
| ---------------- | ---------------------------------------------------------------- |
| Static IP        | `34.111.95.214`                                                  |
| NEG              | `songbird-neg`                                                   |
| Backend Service  | `songbird-backend`                                               |
| URL Map          | `songbird-lb`                                                    |
| HTTP Proxy       | `songbird-http-proxy`                                            |
| HTTPS Proxy      | `songbird-https-proxy`                                           |
| SSL Certificate  | `songbird-cert` (Google-managed for `vm.daw.club`)               |
| Forwarding Rules | `songbird-http-rule` (port 80), `songbird-https-rule` (port 443) |

**DNS Setup:** Point `vm.daw.club` A record to `34.111.95.214`.

The SSL certificate will auto-provision once DNS is configured. Until then, HTTP access works
at `http://34.111.95.214`.

**Verify after deploy:**

```bash theme={null}
# Should return 200 (no auth required for healthz)
curl http://34.111.95.214/healthz

# Should return 403 (no API key)
curl http://34.111.95.214/ws

# Should return 200 (API key via header)
curl -H "X-API-Key: $SONGBIRD_API_KEY" http://34.111.95.214/healthz

# Should return 426 (WebSocket upgrade required — means auth passed)
curl -H "X-API-Key: $SONGBIRD_API_KEY" http://34.111.95.214/ws
```

## Prerequisites

1. **GCP project** with billing enabled
2. **gcloud CLI** authenticated (`gcloud auth login`)
3. **APIs enabled:**
   ```bash theme={null}
   gcloud services enable \
       run.googleapis.com \
       cloudbuild.googleapis.com \
       artifactregistry.googleapis.com \
       storage.googleapis.com \
       --project=gen-lang-client-0380481603
   ```
4. **Git submodules initialized** (the Docker build copies the full source tree):
   ```bash theme={null}
   git submodule update --init --recursive
   ```

## Quick Deploy

```bash theme={null}
# From the repo root:
./deploy/cloud-run/deploy.sh
```

This will:

1. Create a GCS bucket for project storage (if it doesn't exist)
2. Grant the Cloud Run service account access to the bucket
3. Build the Docker image via Cloud Build (\~35 min for full C++ build)
4. Deploy to Cloud Run with spot instances + GCS Fuse
5. Set up API key authentication (auto-generates a key if `SONGBIRD_API_KEY` is not set)
6. Attempt `allUsers` IAM binding (gracefully skipped if org policy blocks it)

### Skip-Build Mode (\~2 min instead of \~35 min)

If the C++ binary is already built and uploaded to GCS:

```bash theme={null}
SONGBIRD_API_KEY="your-key" ./deploy/cloud-run/deploy.sh --skip-build --version v1.0.1
```

This downloads the prebuilt binary from GCS and only rebuilds the Docker image with the React UI.

### Configuration

Override defaults via environment variables:

| Variable           | Default                          | Description                                    |
| ------------------ | -------------------------------- | ---------------------------------------------- |
| `GCP_PROJECT_ID`   | `gen-lang-client-0380481603`     | GCP project ID                                 |
| `GCP_REGION`       | `us-central1`                    | Deployment region                              |
| `SERVICE_NAME`     | `songbird-headless`              | Cloud Run service name                         |
| `GCS_BUCKET`       | `songbird-projects-<project-id>` | GCS bucket for project files                   |
| `BINARY_BUCKET`    | `songbird-daw-club-repos`        | GCS bucket for prebuilt binaries               |
| `SONGBIRD_API_KEY` | *(auto-generated)*               | API key for frontend authentication            |
| `MEMORY`           | `2Gi`                            | Container memory limit                         |
| `CPU`              | `2`                              | Container CPU limit                            |
| `MIN_INSTANCES`    | `0`                              | Minimum instances (0 = scale to zero)          |
| `MAX_INSTANCES`    | `10`                             | Maximum instances                              |
| `DRY_RUN`          | `0`                              | Set to `1` to print commands without executing |

Example with custom settings:

```bash theme={null}
MEMORY=4Gi CPU=4 MAX_INSTANCES=20 ./deploy/cloud-run/deploy.sh
```

## API Key Authentication

### Why API keys?

The GCP org policy blocks `allUsers` IAM binding on Cloud Run services. Instead of relaxing the
org policy, we use a shared secret (API key) validated by nginx at the proxy layer.

### How it works

1. **deploy.sh** auto-generates a random key via `python3 -c 'import secrets; print(secrets.token_urlsafe(32))'` if `SONGBIRD_API_KEY` env var is not set
2. The key is passed to the Cloud Run service as an environment variable
3. **entrypoint.sh** validates the key at startup:
   * Refuses to start if `SONGBIRD_API_KEY` is empty (prevents `'' = ''` nginx bypass)
   * Refuses to start if key is the placeholder `REPLACE_WITH_API_KEY`
4. **entrypoint.sh** passes the key to `envsubst` which substitutes it into nginx.conf
5. **nginx.conf** checks every request (except `/healthz`) for a valid key via:
   * `X-API-Key` header
   * `?api_key=` query parameter
6. Returns `403 {"error":"invalid api key"}` on mismatch

### Security considerations

* The API key is stored as a Cloud Run env var (visible in GCP console and `gcloud run services describe`). For higher security, use Secret Manager with `valueFrom.secretKeyRef`.
* Nginx `if` directives don't inherit into nested `location` blocks — each location has its own auth check.
* The `/healthz` endpoint is intentionally unauthenticated (Cloud Run startup/liveness probes need it).
* The `allUsers` IAM binding attempt is still made but gracefully falls back if org policy blocks it.

### Frontend integration

The daw\.club React UI passes the API key when connecting:

```
https://daw.club?ws=songbird-headless-6kdcepjumq-uc.a.run.app/ws&api_key=YOUR_KEY
```

Or via WebSocket header:

```javascript theme={null}
const ws = new WebSocket('wss://songbird-headless-6kdcepjumq-uc.a.run.app/ws', [], {
  headers: { 'X-API-Key': 'YOUR_KEY' }
});
```

## Build Architecture

The Dockerfile uses a multi-stage build:

```
┌─────────────────────────────────────────────────────┐
│ ARG BUILDER_STAGE=full|prebuilt                     │
│                                                      │
│ ┌─────────────────┐   ┌─────────────────────┐      │
│ │ builder-full    │   │ builder-prebuilt     │      │
│ │ (C++ compile    │   │ (COPY from build     │      │
│ │  ~35 min)       │   │  context, ~1 sec)    │      │
│ └────────┬────────┘   └──────────┬──────────┘      │
│          │   selected by ARG     │                   │
│          └──────────┬────────────┘                   │
│                     ▼                                │
│          ┌──────────────────┐                        │
│          │ builder (alias)  │                        │
│          └────────┬─────────┘                        │
│                   │                                  │
│ ┌─────────────────┼─────────────────┐               │
│ │                 │                 │               │
│ ▼                 ▼                 ▼               │
│ react-ui      runtime           runtime             │
│ (vite build)  COPY binary       COPY static         │
│               + nginx + node    + entrypoint        │
└─────────────────────────────────────────────────────┘
```

* **builder-full**: Ubuntu 22.04, installs build-essential + JUCE deps, runs cmake + make with `-j2` (avoids OOM)
* **builder-prebuilt**: Copies binary from `deploy/cloud-run/prebuilt/SongbirdHeadless` (placed there by `deploy.sh --skip-build`)
* **react-ui**: Node 20, runs `npm ci` + `npx vite build` (skips tsc — pre-existing TS errors on headless branch)
* **runtime**: Ubuntu 22.04 with nginx, Node.js 20, Xvfb, GCS Fuse, tini (PID 1)

### Cloud Build

Full builds use `e2-highcpu-8` machine type with 3600s timeout:

```bash theme={null}
gcloud builds submit \
  --tag=us-central1-docker.pkg.dev/gen-lang-client-0380481603/songbird/songbird-headless:v1.0.1 \
  --timeout=3600 --machine-type=e2-highcpu-8 \
  --region=us-central1 .
```

The C++ build uses `-j2` parallelism to stay under memory limits. Total build time is \~35 minutes.

### Binary upload to GCS

After a successful full build, the binary is extracted from the Docker image and uploaded to GCS:

```
gs://songbird-daw-club-repos/bin/v1.0.0/SongbirdHeadless-release
gs://songbird-daw-club-repos/bin/latest/SongbirdHeadless-release
```

This enables `--skip-build` mode for future deploys.

## Manual Deploy

### 1. Build the Docker image

```bash theme={null}
# Option A: Full build with Cloud Build (~35 min)
gcloud builds submit \
  --tag=us-central1-docker.pkg.dev/gen-lang-client-0380481603/songbird/songbird-headless:latest \
  --timeout=3600 --machine-type=e2-highcpu-8 --region=us-central1 .

# Option B: Build locally (needs Docker + 8GB+ RAM)
docker build -t us-central1-docker.pkg.dev/gen-lang-client-0380481603/songbird/songbird-headless .
docker push us-central1-docker.pkg.dev/gen-lang-client-0380481603/songbird/songbird-headless
```

### 2. Deploy to Cloud Run

Edit `deploy/cloud-run/service.yaml`:

* Update the `image:` tag to your version
* Update the `SONGBIRD_API_KEY` value (**do NOT leave it as `REPLACE_WITH_API_KEY`**)
* Update `bucketName` if using a different GCS bucket

```bash theme={null}
gcloud run services replace deploy/cloud-run/service.yaml \
  --project=gen-lang-client-0380481603 --region=us-central1

# Attempt public access (may be blocked by org policy — that's OK, API key handles auth)
gcloud run services add-iam-policy-binding songbird-headless \
  --project=gen-lang-client-0380481603 --region=us-central1 \
  --member="allUsers" --role="roles/run.invoker"
```

### 3. Upload a project

```bash theme={null}
gcloud storage cp myproject.bird gs://songbird-projects-gen-lang-client-0380481603/
```

### 4. Connect from the React UI

```
https://daw.club?ws=songbird-headless-6kdcepjumq-uc.a.run.app/ws&api_key=YOUR_KEY
```

## GCP Authentication Notes

When using short-lived access tokens (e.g. from `gcloud auth print-access-token`):

```bash theme={null}
export CLOUDSDK_AUTH_ACCESS_TOKEN="ya29...."
```

**Important:** `gsutil` does NOT pick up `CLOUDSDK_AUTH_ACCESS_TOKEN`. Always use `gcloud storage`
commands instead:

* `gcloud storage cp` instead of `gsutil cp`
* `gcloud storage ls` instead of `gsutil ls`
* `gcloud storage buckets create` instead of `gsutil mb`

The deploy.sh script already uses `gcloud storage` for this reason.

## Container Ports

| Port | Protocol  | Service                                          |
| ---- | --------- | ------------------------------------------------ |
| 8080 | HTTP      | Nginx reverse proxy (Cloud Run ingress)          |
| 9090 | WebSocket | Headless DAW engine                              |
| 9091 | HTTP      | Headless DAW resource server (audio files, etc.) |
| 8765 | WebSocket | Collaboration server                             |
| 9092 | HTTP      | Clone service (git clone API)                    |

All traffic enters through port 8080 (nginx) and is routed internally:

* `/ws` → headless DAW WebSocket (:9090) — **requires API key**
* `/resources/` → headless DAW HTTP resources (:9091) — **requires API key**
* `/collab` → collaboration server (:8765) — **requires API key**
* `/api/clone` → clone service (:9092) — **requires API key**
* `/healthz` → health check (nginx responds directly) — **no auth**
* `/` → React UI static files — **requires API key**

## Container Startup Sequence

The `entrypoint.sh` script starts services in this order:

1. **Validate API key** — exit if empty or placeholder
2. **GCS Fuse mount** — only for local Docker runs; Cloud Run uses CSI driver
3. **Xvfb** — virtual framebuffer (JUCE/GTK needs a display even headless)
4. **SongbirdHeadless** — the C++ binary (WS :9090, HTTP :9091)
5. **Collab server** — Node.js WebSocket server (:8765)
6. **Clone service** — Node.js HTTP server for git clone API (:9092)
7. **Wait for readiness** — polls headless HTTP port for up to 15 seconds
8. **Nginx** — reverse proxy with API key auth (:8080)

If any child process exits, the container shuts down (Cloud Run will restart it).

## Cost Estimate

With spot instances and scale-to-zero:

| Usage                         | Monthly Cost |
| ----------------------------- | ------------ |
| Idle (0 instances)            | \$0          |
| 1 user, 8 hrs/day             | \~\$5-10     |
| 10 users, 8 hrs/day           | \~\$50-100   |
| Always-on (1 instance, min=1) | \~\$30-50    |

Spot instances are \~60-91% cheaper than standard but may be preempted. Project state is safe on GCS.

## Troubleshooting

### Container won't start

* Check logs: `gcloud run services logs read songbird-headless --region=us-central1`
* If you see `FATAL: SONGBIRD_API_KEY is not set` — the env var is missing from the service config
* If you see `FATAL: SONGBIRD_API_KEY is still set to the placeholder value` — update the key in service.yaml
* The headless binary needs Xvfb (virtual framebuffer) — included in the Docker image
* GTK/WebKit runtime libs must be present — see Dockerfile runtime stage

### 403 on all requests

* Verify `SONGBIRD_API_KEY` is set correctly in the Cloud Run service config
* Check you're passing the key via `X-API-Key` header or `?api_key=` query param
* The `/healthz` endpoint should always return 200 without auth — if it returns 403, the nginx config is wrong

### GCS Fuse mount fails

* Ensure the Cloud Run service account has `storage.objectAdmin` on the bucket
* GCS Fuse requires gen2 execution environment (set in service.yaml)
* Check mount logs in container stderr

### WebSocket connection drops

* Cloud Run has a 60-minute request timeout (configured in service.yaml)
* Session affinity is enabled to maintain sticky connections
* If using a custom domain, ensure the load balancer supports WebSocket upgrade

### Cloud Build times out

* Default timeout is 30 minutes — C++ builds need \~35 minutes
* Always use `--timeout=3600` (60 min) for full builds
* Use `--machine-type=e2-highcpu-8` for faster compilation
* Build uses `-j2` to avoid OOM — don't increase without more memory

### Audio dropouts

* Increase CPU/memory: `MEMORY=4Gi CPU=4 ./deploy/cloud-run/deploy.sh`
* Cloud Run gen2 has better CPU performance than gen1
* Consider `MIN_INSTANCES=1` to avoid cold starts during active use

### Org policy blocks allUsers

* This is expected. The API key approach works without public IAM binding.
* The deploy script gracefully handles this: `|| echo "allUsers binding skipped"`
* Do NOT try to relax the org policy — use the API key instead

## Files Reference

| File                                | Purpose                                                |
| ----------------------------------- | ------------------------------------------------------ |
| `Dockerfile`                        | Multi-stage build: C++ → React UI → runtime with nginx |
| `deploy/cloud-run/deploy.sh`        | One-command deployment script                          |
| `deploy/cloud-run/entrypoint.sh`    | Container entrypoint (process manager)                 |
| `deploy/cloud-run/nginx.conf`       | Reverse proxy with API key auth                        |
| `deploy/cloud-run/clone-service.js` | Git clone HTTP API (clones repos to /tmp)              |
| `deploy/cloud-run/service.yaml`     | Cloud Run Knative service definition (template)        |
| `deploy/cloud-run/prebuilt/.keep`   | Placeholder for legacy Docker builder compat           |
| `.dockerignore`                     | Excludes .git, node\_modules, build artifacts          |