Runbook: zero to a running OpenWA

The gateway is private and never public: operators reach it over the tailnet or an SSH port-forward.

Production is WAFINE on the production zone: the WAFINE web host (web), its api. host (BFF) and its docs. host (docs). Those records do not exist yet — publishing them is a pending owner decision (§7): move the zone's nameservers to Cloudflare, or keep them at the registrar and give the VM a public Caddy entry for the BFF.

The testing playground is a second zone — it already has a Cloudflare zone and tunnel and keeps serving the same roles until the production records exist:

The playground serves the same roles on its own zone (wa. = web, gw. = gateway, api. = BFF, docs. = docs).

0. Operator toolchain (once)Copy Anchor Link

scripts/bootstrap.sh            # oci-cli, opentofu, cloudflared, uv, supabase CLI, Tailscale
                                # detects macOS vs Linux; --check reports without installing

Both are idempotent. Platform differences (BSD vs GNU date, sed -i, seq) live in scripts/lib/os.sh; scripts/tf picks tofu (override with TF_BIN=terraform).

1. Authenticate (you, interactively; nothing here is automated on purpose)Copy Anchor Link

Check Oracle: oci iam region-subscription list --output table must show af-casablanca-1 as home.

2. Tailscale tailnet settings (admin console)Copy Anchor Link

1. DNS → enable MagicDNS and HTTPS certificates (needed for tailscale serve).
2. Access controls → add tag:server (see .claude/skills/edge-network/SKILL.md for the policy).
3. Settings → Keys → generate an auth key: one-off, pre-approved, tags tag:server, expiry 1 day. It goes in terraform/oci/terraform.tfvars.

3. Supabase (free org + project)Copy Anchor Link

1. Dashboard → New organization, plan Free. The MCP connector can't create orgs.
2. Claude creates project openwa-ma in eu-west-3 in that org, applies supabase/migrations/20261007120000_openwa_datastore.sql, then asks you to set the role password.
3. Set the password yourself in the SQL editor (never in git or chat): alter role openwa with login password '<openssl rand -base64 36>';
4. Collect for stack/.env:
   • Session pooler host (Connect → Session pooler)
   • project ref
   • Database → SSL → download the CA as stack/certs/supabase-ca.crt
   • Object storage is Cloudflare R2, not Supabase Storage (decision 2026-10-07) — see docs/02-architecture.md §Decisions 5 and TASKS.md §2 for the R2 buckets and token.

Free-plan limits: 500 MB DB, 1 GB storage, 50 MB per file, project pauses after 7 days of inactivity. The nightly GitHub workflow .github/workflows/nightly-health.yml (02:17 UTC) writes a health snapshot to the database, which keeps the project awake. It needs the secret SUPABASE_DB_URL (see TASKS.md §4). If the project is paused anyway, restore it from the Supabase dashboard.

Migration ledger. supabase db push relies on supabase_migrations.schema_migrations. The two health migrations (20261011120000_ops_health_snapshots, 20261012120000_ops_health_monitor_fixes) were first applied by hand, so the ledger was out of sync. Their two versions were then inserted into that table, after checking that the objects existed. If you apply a migration by hand, record it the same way, or db push will try to apply it again. Check with supabase migration list. Never edit an applied migration file.

4. OCI VMCopy Anchor Link

cd terraform/oci
cp terraform.tfvars.example terraform.tfvars    # tenancy_ocid, compartment_ocid, ssh key, tailscale key
cd ../.. && scripts/oci-a1-retry.sh 120          # shows the plan, asks, retries on "Out of host capacity"

After ~3 minutes the VM wa-ma-1 appears in tailscale status. Check: ssh ubuntu@wa-ma-1 cat /opt/wa/.provisioned.

5. First deployCopy Anchor Link

ssh ubuntu@wa-ma-1 'mkdir -p /opt/wa/stack/certs'
scp stack/certs/supabase-ca.crt ubuntu@wa-ma-1:/opt/wa/stack/certs/
scp stack/.env.example ubuntu@wa-ma-1:/opt/wa/stack/.env
ssh ubuntu@wa-ma-1                                # edit /opt/wa/stack/.env with the Supabase values; leave CLOUDFLARE_TUNNEL_TOKEN empty
scripts/deploy.sh                                 # rsync source → preflight → build openwa-api + bff (≈10–15 min on A1) → up → backup cron → tailscale serve → gateway-key --check
ssh ubuntu@wa-ma-1 /opt/wa/stack/wa apikey        # store the admin key in your password manager

The deploy does not stop if the BFF has no key yet. If the final check prints FAIL gateway rejected the OPENWA_API_KEY, run ./stack/gateway-key.sh && ./stack/wa up -d bff on the VM (details in §8).

Open <host>.<tailnet>.ts.net and log in with the key. Then set BASE_URL, DASHBOARD_URL and CORS_ORIGINS in .env to that URL and run ./stack/wa up -d.

6. Link a WhatsApp numberCopy Anchor Link

Use a dedicated, warmed-up number, never your main business line (OpenWA/README.md, "Before you connect a number"). Dashboard → Sessions → New → Start → scan the QR from WhatsApp → Linked devices.

7. Phase 2: publishing the public hostnamesCopy Anchor Link

Real hostnames, zones and accounts are kept outside the repo in the operator vault; this runbook is written against placeholders.

The testing playground is already on Cloudflare and stays the testing playground; the production zone is where WAFINE launches and its records do not exist yet. ai4x.academy stays on Netlify — don't touch it.

Pending owner decision — how the production web, api and docs hosts get published:

• (a) Move the production zone to Cloudflare. The nameservers sit at the registrar: export the current records, recreate them in a new Cloudflare zone as DNS-only first, switch the registrar's nameservers, and verify with a before/after dig diff. Then add the production web/api/docs hosts exactly like the playground zone. A Cloudflare Tunnel only serves hostnames in its own zone, so this is the only way to keep the api host on the tunnel.
• (b) Keep the nameservers at the registrar and give the VM a public entry point. Vercel works with registrar CNAMEs for the web and docs sites, but the BFF cannot go through the tunnel from another zone, so it needs its own listener: a Caddy reverse proxy on the VM (TLS via Let's Encrypt) plus an A/CNAME for the api host. This reopens an inbound port (443), which the project has avoided so far — it needs a deliberate ingress rule and the captain's approval, and it drops Cloudflare's edge from the BFF path.

The web and docs hosts are registrar-compatible under either option (Vercel DNS-only CNAME + _vercel TXT); only the BFF forces the choice. The steps below apply to whichever zone ends up carrying the records — the playground zone today, the production zone after the decision.

1. The gateway stays private and is never made public (gateway_public = false: no gateway DNS record, no gateway ingress rule, no Access resources). Operators reach it over the tailnet or an SSH port-forward from their machine.
2. Add the web hostname to the Vercel project first, so Vercel names its CNAME target: vercel domains add app.example.com from app/web (playground: its web host), then vercel domains inspect app.example.com. If Vercel asks for a _vercel TXT record (the apex is owned by another Vercel account), add it as DNS-only.
3. cp terraform/cloudflare/terraform.tfvars.example terraform/cloudflare/terraform.tfvars, fill it in (web_cname_target = the CNAME from step 2), then scripts/tf cloudflare init && scripts/tf cloudflare apply
4. Connector first, record second (decision 2026-10-14: publish through the tunnel; the Caddy/proxy ingress plan runs after the E2E passes). On the VM, put scripts/tf cloudflare output -raw tunnel_token into /opt/wa/stack/.env as CLOUDFLARE_TUNNEL_TOKEN, then ./stack/wa up -d. stack/wa enables the public profile by itself and starts openwa-cloudflared. Without a DNS record nothing is reachable yet, so this step is safe before sign-up merges.
5. Confirm the connector: ssh ubuntu@wa-ma-1 'docker logs --tail=50 openwa-cloudflared' shows registered connections, or the Cloudflare API GET /accounts/<account_id>/cfd_tunnel/$(scripts/tf cloudflare output -raw tunnel_id) reports status: healthy with connections > 0.
6. Publish gate (passed 2026-10-14 on the playground). api_public = true is set and applied, which creates one DNS record (cloudflare_dns_record.api[0]). Setting it back to false removes that record and takes the BFF off the internet.
7. Verify from outside: curl -s -o /dev/null -w '%{http_code}\n' https://api.example.com/health is 200 (playground: the API host); an unauthenticated GET /api/workflows is 401; /mcp is 404; dig +short on the playground gateway host stays empty.
8. Wire the web app: BFF_URL=https://api.example.com as a server-only Vercel variable (read only by the /api/bff route handler, §8b; never in the client bundle) and the repo variable HEALTH_URL_API so nightly-health probes it. Leave BFF_CORS_ORIGINS empty on the VM: browsers reach the BFF only through the web app's same-origin proxy, so CORS stays default-deny.
9. Cut over at launch: point the Supabase Auth Site URL at https://app.example.com (and the redirect allowlist), add the docs host to Vercel, and keep the playground records for testing. Then run the §7 verification against production.

8. BFF on the VM (next to the engine)Copy Anchor Link

The BFF (app/bff, Bun/Hono, port 4000) and its Valkey run in the same compose project as OpenWA, so the BFF reaches the engine as http://openwa-api:2785 on the compose network. The BFF publishes only 127.0.0.1:4000 on the VM. Its public path is the tunnel hostname api.<zone> (§7), not a published port.

scripts/deploy.sh ships app/bff/, builds bff (arm64, on the VM) and starts it with the rest of the stack. It needs a working gateway key in /opt/wa/stack/.env as OPENWA_API_KEY. The key is never in git; set it on the VM.

Which key (checked against OpenWA v0.24.0 source, README "Session-scoped" and session.controller.ts): an unscoped OPERATOR key, minted in the dashboard (Settings → API keys, role operator, no sessions ticked).

• Scoped keys exist (allowedSessions, allowedChats) but cannot work here: POST /sessions is closed to a session-scoped key (the new session is outside its allowlist), and the BFF creates a session per onboarded user.
• OPERATOR is enough for everything the BFF does: create/start/stop/delete sessions, QR, send, and sessions/:id/webhooks CRUD. ADMIN is not needed (it only adds key management and per-session proxy config), so the engine's bootstrap admin key is a fallback, not the target.
• Until an operator key is minted, the existing vault key (pass accounts/openwa/ai4xedu/api-key, the bootstrap admin key) works: Option A/B below install either one.
• Mint the operator key once the VM is up, install it with Option B, then confirm with --check. If a BFF call returns 403 on a route not listed above, that route needs a higher role: raise it deliberately, don't fall back to admin silently.

# Option A, on the VM: copy the engine's bootstrap ADMIN key into stack/.env (tested first, never printed): fallback only
ssh ubuntu@wa-ma-1 'cd /opt/wa && ./stack/gateway-key.sh'
# Option B, from the workstation: pipe the key (operator key, or the vault key as fallback); it never touches argv or history
pass show accounts/openwa/ai4xedu/api-key | ssh ubuntu@wa-ma-1 'cd /opt/wa && ./stack/gateway-key.sh --stdin'
# Then recreate the BFF so it re-reads .env, and verify
ssh ubuntu@wa-ma-1 'cd /opt/wa && ./stack/wa up -d bff && ./stack/gateway-key.sh --check'

Why a check script: OpenWA's /api/health is public, so a stale key looks healthy. The BFF's /health/ready now also calls /api/sessions with the key (503 when rejected), and deploy.sh ends with gateway-key.sh --check. PUSH_ENV=1 scripts/deploy.sh overwrites the VM .env with the workstation copy, whose key may be stale: re-run gateway-key.sh after it.

Verify (on the VM, in /opt/wa):

./stack/wa ps                                                  # bff + redis healthy (image HEALTHCHECK uses wget; the base image has no curl)
curl -s http://127.0.0.1:4000/health                           # {"ok":true,...}
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:4000/health/ready   # 200 = gateway up AND key accepted

CI / local checks (no VM needed): scripts/bff-smoke (real image + Valkey + mock gateway, good and stale key), OPENWA_DIR=<upstream checkout> scripts/stack-check (renders the real stack, asserts the BFF wiring).

8b. Web app → BFF (Vercel server proxy)Copy Anchor Link

The browser never talks to the BFF. app/web serves a same-origin route handler at /api/bff/* (app/web/src/app/api/bff/[...path]/route.ts, logic in app/web/src/lib/bff-proxy.ts); the browser calls relative paths and the Vercel server forwards them to BFF_URL. BFF_URL is server-only, never NEXT_PUBLIC_, and the CSP is connect-src 'self'. The proxy forwards only the method, a small header allowlist and the session cookie, drops any client-supplied tenant header, caps bodies at 1 MiB, times out after 15 s, and maps upstream failure to 502/503/504.

The api hostname. Vercel functions run on the public internet and cannot join the tailnet, and the VM has no inbound ports, so the Vercel server reaches the BFF through the Cloudflare Tunnel on https://api.<zone> (§7 steps 4-7, api_public). Decision 2026-10-14: no Cloudflare Access / Zero Trust in front of api. The gate is the BFF itself: every /api/* route needs a Supabase session (tenant = verified user id), /auth/* is rate-limited per client IP, and /webhook/* is HMAC-only. The proxy forwards only /api/* and /auth/*; /health and /webhook are not reachable through it. It still sends CF-Access-Client-Id / CF-Access-Client-Secret if those env vars are ever set (unused today).

Known limit: through the proxy every request reaches the BFF from Vercel's egress IPs, so the /auth/* limit (10/min per IP) is effectively per Vercel IP, not per end user. Fine for the invite-only PoC; revisit before opening sign-up.

Live steps (operator):

1. BFF deployed on the VM (§8) and the api. hostname published (§7 steps 4-7).
2. Vercel project web, Production and Preview, server-only: BFF_URL=https://api.example.com (playground: its API host); vercel env add BFF_URL production from app/web. Never NEXT_PUBLIC_.
3. Redeploy the web app. Verify: https://app.example.com/api/bff/api/me → 401 without a session (proxied to the BFF), https://app.example.com/api/bff/health → 404 (not proxied), then sign up / log in at https://app.example.com (playground: its web host). Without BFF_URL, /api/bff/* answers 503 (bff_not_configured).

8c. Docs site (https://docs.example.com, password protected)Copy Anchor Link

The Fumadocs site in app/docs runs on the Vercel project docs (team omnizya); the playground keeps serving it on its docs host until the DNS move (§7). Every path, including robots.txt, needs HTTP Basic Auth (app/docs/proxy.ts): DOCS_USER / DOCS_PASSWORD live only in the Vercel project; in production a missing value returns 503 (fails closed). Responses carry X-Robots-Tag: noindex, nofollow.

Deploy (prebuilt, because a remote build can't see the repo's docs/ that scripts/sync-docs.ts copies):

cd app/docs
vercel pull --yes --environment=production --scope omnizya
vercel build --prod --scope omnizya
vercel deploy --prebuilt --prod --scope omnizya

Rotate the password: new value in pass web/<zone>/docs-site, then vercel env rm DOCS_PASSWORD production and vercel env add DOCS_PASSWORD production (and preview), then redeploy. Check: curl -s -o /dev/null -w '%{http_code}' https://docs.example.com/docs is 401, and 200 with -u <user>:<password>.

8d. End-to-end journey tests (Playwright)Copy Anchor Link

app/web/e2e/journey.e2e.ts walks the real product path against a deployed stack: J1 landing loads, J2 sign up / sign in (invite-only), J3 request a WhatsApp session and show the QR, J3b scan the QR with a phone (human), J4 workflows page, J5 mint a scoped API key and call one REST endpoint, J6 MCP OAuth discovery.

Run it with the Bun script; BASE_URL is the only required knob (default http://localhost:3000, where the config starts next dev for you):

cd app/web
BASE_URL=https://<pr>.vercel.app bun run test:e2e          # against a preview/prod URL
bun run test:e2e                                          # against a local next dev
bunx playwright install --with-deps chromium              # first run only

Credentials come from the environment, never the repo: E2E_EMAIL / E2E_PASSWORD (the invited test account), E2E_WA_NUMBER (its invited E.164 number, for sign-up), E2E_MCP_URL (the BFF /mcp, J6). With none set the whole suite skips — a run is safe on any machine and in any CI.

Two step kinds are marked rather than faked: the QR scan is test.fixme unless E2E_LIVE_QR=1 (a human must scan it), and steps the product cannot satisfy yet are test.fail with the blocking task named in the description. Today J3/J5 are test.fail on owa-session-issuance / owa-session-qr (TASKS.md §3): when those land the mark must be removed — an unexpected pass fails the job on purpose.

CI: the e2e job in .github/workflows/app.yml runs on every PR against the branch's Vercel preview URL, discovered from the GitHub Deployment Vercel creates (set the repo variable E2E_BASE_URL to override). The preview must have BFF_URL set in the Vercel project's Preview environment (§8b). No preview URL or no E2E_* secrets → the job skips, so it never blocks a PR. The HTML report is uploaded as the playwright-report artifact.

Day-2Copy Anchor Link

Workstation disk spaceCopy Anchor Link

/home on the operator workstation once hit 100% and broke the validation pipeline (git, bun and builds fail with confusing errors). Check free space before a deploy or a validation run, then clear only caches that are rebuilt on demand:

df -h /home                                          # free space first
du -sh ~/.cache/* 2>/dev/null | sort -h | tail      # the big ones
bun pm cache rm                                      # bun package cache
npm cache clean --force                              # npm cache
rm -rf ~/.cache/infra-tofu/plugin-cache              # OpenTofu provider cache; run `scripts/tf <oci|aws|cloudflare> init` afterwards
docker system prune -f                               # dangling images and stopped containers (only if Docker is installed)

Do not delete ~/.config/infra/ (secrets), ~/.oci, stack/.env, *.tfvars, or OpenWA/.