Skip to main content

Operations Runbooks

These runbooks turn the deployment architecture into repeatable procedures. They deliberately separate repository state from hosted state: a migration, function, flag, or secret in source is not evidence that the corresponding remote control plane has been changed.

The production Supabase project reference used by the repository is:

bxmwvqnilignrxkitaae

Confirm the target account, project, environment, and release authority before every remote command. The examples below show the required shape; they do not authorize a production change.

Non-Negotiable Rules​

  1. Edit Edge Function source in supabase/functions/<slug>/index.ts before deployment.
  2. Deploy every Edge Function through the Supabase CLI with both the explicit project reference and --no-verify-jwt.
  3. Never deploy a dashboard-only function variant or use a deployment path that cannot pass the required flag.
  4. Prefer forward corrective migrations; never reset a remote database.
  5. Keep the service-role key out of the browser, repository, logs, screenshots, and documentation.
  6. Derive email links from the server-side APP_URL, never a request field.
  7. Run HawkScan only against local or deliberately isolated synthetic data, never production.
  8. Record the exact source commit and rollback target before a release.

Runbook: Prepare a Release​

1. Establish Scope​

Record:

  • commit and branch
  • target environment and Supabase project reference
  • migrations added
  • Edge Function directories changed
  • frontend, docs, agent, and provider changes
  • secrets or feature flags required
  • user-visible and data-compatibility impact
  • previous known-good versions

Review the diff for accidental .env, build, scan-output, or generated credential files.

2. Validate Locally​

Frontend:

cd frontend
npm run lint
npm test
npm run i18n:audit
npm run build

Run the relevant Playwright projects after checking their target and test tenant. See Testing and Accessibility.

Documentation:

cd docs
npm run typecheck
npm run build

Agent, when changed:

cd agent
pnpm install
pnpm build

Database, when changed:

npx supabase start
npx supabase db reset

A local supabase/config.toml may define ports, Auth hooks, and function settings, but that file is ignored by Git in this repository. Do not treat one developer's local config as a production declaration. Verify hosted Auth and function settings separately.

3. Apply Security Validation​

For application code changes, run the local HawkScan remediation loop in LOCAL_DAST_RUNBOOK.md:

bash scripts/seed-local-scan.sh
bash scripts/scan-local.sh

The scanner targets the local Supabase API at http://localhost:54421 by default and uses synthetic tenant data. Review stackhawk.yml and the target environment immediately before the run. Exit code 42 means findings exceed the configured threshold; do not misreport it as a scanner crash.

The local runbook documents a temporary ?no-dts import workaround. Reverse only that exact mechanical change afterward; do not discard unrelated working tree changes.

Documentation-only changes do not require DAST.

Runbook: Apply Database Migrations​

Preconditions​

  • every schema change is a new file under supabase/migrations/
  • the full local reset succeeds against PostgreSQL 17
  • RLS, grants, triggers, and SECURITY DEFINER search paths were reviewed
  • destructive/data-rewriting statements have an owner-approved backup and recovery plan
  • old and new frontend/function versions remain compatible for the rollout window

Compare Remote State​

npx supabase link --project-ref bxmwvqnilignrxkitaae
npx supabase migration list

Linking does not apply changes. Compare local and remote lists and investigate unexpected remote-only or missing versions before continuing.

Apply​

npx supabase db push

Read the pending list and confirmation carefully. Do not use db reset or an equivalent destructive command against a hosted project.

Verify​

At minimum:

  1. inspect migration status again
  2. exercise an ordinary member and an admin/consultant path
  3. perform a wrong-tenant negative request
  4. verify affected RLS policies and RPC execute grants
  5. verify relevant triggers and Realtime publication
  6. test the prior application version if the rollout is not atomic
  7. check database and Auth logs for new denials or exceptions

For Auth-hook migrations, also verify the hosted Before User Created setting:

pg-functions://postgres/public/hook_before_user_created

The SQL function existing does not prove that the hosted Auth control plane invokes it.

Correct a Migration​

If an applied migration is wrong:

  1. stop dependent application rollout
  2. preserve evidence and assess data already changed
  3. write a new forward migration
  4. validate from a clean local reset and an upgrade-shaped database
  5. apply with the same review

Do not edit or delete an already shared migration to make history look clean.

Runbook: Deploy an Edge Function​

Preconditions​

  • repository source contains the intended implementation
  • the directory actually exists; local config may contain future placeholders
  • methods, body limits, CORS, auth, role, and tenant checks were tested
  • AI spend records both token_usage and record_token_usage
  • required feature flag, rate limit, budget, and idempotency controls exist
  • secrets are available in the target environment

Deploy​

npx supabase functions deploy <name> \
  --project-ref bxmwvqnilignrxkitaae \
  --no-verify-jwt

The --no-verify-jwt flag is required for every function. Navaid validates bearer tokens inside functions with supabase.auth.getUser(token) because the gateway verifier rejects the project's ES256 tokens.

Verify​

Run a bounded smoke matrix:

RequestExpected result
OPTIONSsuccess with canonical CORS headers
wrong method405 where the endpoint is method-specific
no/invalid bearergeneric 401 on protected endpoints
wrong tenant/rolegeneric 403 or 404
malformed/oversized inputcontrolled 400/413
valid requestexpected data/provider path
rate/budget denialcontrolled 429/402-style application response as implemented

Confirm Access-Control-Allow-Headers includes:

authorization, x-client-info, apikey, content-type, baggage, sentry-trace

For Gemini functions, compare per-request and monthly usage after the smoke turn. For link-bearing email functions, unset/missing APP_URL must decline to send. For WebSockets, test upgrade auth, normal close, abrupt disconnect, reconnect, transcript persistence, and usage metering.

Function Rollback​

Checkout or otherwise stage the prior known-good repository source, then redeploy that directory with the same project reference and --no-verify-jwt. Verify the version by behaviour and logs. Never reconstruct a lost production function from a dashboard editor.

Runbook: Configure Function Secrets​

Inventory by capability:

CapabilitySecrets
GeminiGEMINI_API_KEY
Live model/voice overrideGEMINI_LIVE_MODEL, GEMINI_LIVE_VOICE
TTS model overrideGEMINI_TTS_MODEL
Mail linksAPP_URL
ResendRESEND_API_KEY
LiveKitLIVEKIT_URL, LIVEKIT_API_KEY, LIVEKIT_API_SECRET
R2R2_ACCOUNT_ID, R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY, R2_BUCKET

Supabase injects its URL and service-role secret into hosted functions. Never mirror the service role into a VITE_ variable.

Set values through the approved secret-management path. If using the CLI:

npx supabase secrets set NAME=<value> \
  --project-ref bxmwvqnilignrxkitaae

Avoid literal values in shared shell history. Record secret name, owner, rotation date, consuming workloads, and validation result without recording the value.

After rotation, restart or redeploy every long-running consumer and verify both success and old-key rejection.

Runbook: Release the Frontend​

Build Contract​

Cloudflare Pages uses:

SettingValue
Rootfrontend
Node20 or later
Installnpm 10 npm ci
Buildnpm run build
Outputfrontend/dist

When dependencies changed:

cd frontend
npx -y npm@10 install --package-lock-only --no-audit --no-fund
npx -y npm@10 ci --dry-run --no-audit --no-fund

Confirm package-lock.json contains node_modules/@emnapi/core and commit package.json and the lock together.

Pre-Publish Inspection​

  • required browser variables use only public anon/DSN values
  • no service-role or provider secret is in dist
  • _headers and _redirects are present
  • no .map files will be published
  • version.json exists and is not assigned immutable caching
  • direct client-route navigation falls back to the SPA

Post-Publish Smoke​

  1. load marketing and login from a fresh browser
  2. navigate directly to an authenticated lazy route
  3. sign in through the expected invite-gated flow
  4. confirm Supabase REST, function, Realtime, and LiveKit origins are allowed by CSP only as needed
  5. perform a controlled AI turn and verify accounting
  6. open an old tab and verify the update prompt or one-shot chunk recovery
  7. send sanitized Sentry test telemetry and confirm its release
  8. verify no source map URL is public

Rollback​

Promote the previous known-good Cloudflare Pages build. Additive schema changes should keep that client functional. If not, decide on a forward compatibility fix rather than attempting to remove live data blindly.

Runbook: Release the Documentation​

cd docs
npm ci
npm run typecheck
npm run build

Publish docs/build through the documentation host configured outside this repository. This repository currently has no checked-in GitHub Actions deployment workflow, so do not assume a merge performs the publish.

Verify:

  • developer and user navigation
  • internal links and code blocks
  • Navaid logo, favicon, typography, and palette
  • light/dark contrast and mobile menu
  • canonical URL/base URL
  • no application or provider secret appears in generated HTML

Documentation branding is static and does not read tenant theme_config.

Runbook: Release the LiveKit Agent​

The agent/ process is a persistent Node worker, not an Edge Function.

cd agent
pnpm install
pnpm build
pnpm start

The host needs:

LIVEKIT_URL
LIVEKIT_API_KEY
LIVEKIT_API_SECRET
GOOGLE_API_KEY or GEMINI_API_KEY
SUPABASE_URL
SUPABASE_SERVICE_ROLE_KEY

Optional GEMINI_LIVE_MODEL and GEMINI_LIVE_VOICE override the current defaults.

Acceptance:

  1. two authorized users and the agent join one test room
  2. the worker receives the intended session-* context
  3. each transcript is attributed to the correct app user
  4. nonparticipant or removed-user speech is refused
  5. transcript and maturity side effects persist
  6. usage is recorded periodically and at close
  7. leave/remove disconnects the media participant

The worker's tighter named dispatch to only session-* rooms remains future hardening. Do not mark the agent production-ready based on a successful build alone; its README explicitly requires live validation.

Runbook: Enable Optional Providers​

Cloudflare R2 Documents​

  1. create a private bucket
  2. issue credentials restricted to that bucket
  3. set all four R2 secrets; partial configuration remains unavailable
  4. allow only required app origins and PUT, GET, HEAD in bucket CORS
  5. enable storage for an isolated test company
  6. test sign, direct upload, HEAD confirmation, quota, download, expiry, cross-tenant denial, and deletion

Never make the bucket public. Signed URLs are short-lived capabilities.

  1. verify sender domain and identity
  2. set RESEND_API_KEY
  3. set APP_URL to the exact trusted application origin
  4. test each locale with a controlled recipient
  5. confirm the request body cannot substitute a host
  6. confirm missing APP_URL sends no link-bearing email

Auth OTP SMTP is a separate hosted Auth configuration even if it uses the same provider account.

MCP OAuth​

Local supabase/config.toml, when present, has OAuth server functionality disabled by default. For a hosted environment:

  1. enable the Supabase OAuth server deliberately
  2. set the consent path to /oauth/consent
  3. decide whether dynamic registration is allowed
  4. confirm redirect allowlists
  5. enable mcp_enabled
  6. exercise discovery, challenge, consent, token issuance, tenant selection, read tools, and authorized writes
  7. deny a token without the required client_id/scope context

See MCP and OAuth for the protocol and permission model.

Runbook: Change or Roll Back a Gemini Model​

Current defaults are:

Text: gemini-3.1-flash-lite
Live: gemini-3.1-flash-live-preview
TTS:  gemini-3.1-flash-tts-preview

Text functions import a source constant. Changing the text model requires a reviewed source change and redeployment of every affected function. Live and TTS paths support environment overrides:

GEMINI_LIVE_MODEL
GEMINI_TTS_MODEL

GEMINI_LIVE_VOICE changes the voice, not the model.

Before changing a model:

  • verify official availability and region/project access
  • compare request/response and usage metadata contracts
  • test JSON extraction, tools, audio modality, and context limits as relevant
  • run deterministic tests plus one isolated provider smoke
  • confirm pricing/budget assumptions
  • record the old value for rollback

Rollback an override to the prior known-good value and reconnect/restart live consumers. Roll back a source constant by redeploying the prior repository version with the mandatory function flags. Never hide a model outage by disabling token accounting or tenant authorization.

Runbook: Feature-Flag Incident​

  1. identify whether the flag is enforced in the browser, server, or both
  2. use the System Admin path to change only an allowlisted key
  3. verify the persisted system_config shape
  4. test an already-open client and a fresh client
  5. verify direct API calls are denied if the feature requires server enforcement
  6. record the change and restoration condition

voice_enabled is currently a browser control; chat-live does not read it. Do not rely on that flag as an emergency server kill switch. Server-enforced examples include aida_enabled, mcp_enabled, and lead-hunt/budget controls as described in Administration and Feature Flags.

daily_token_budget and alert_config are stored by the Admin UI but have no current runtime daily enforcer or alert sender. Monthly budget controls remain the effective AI spend gate.

Runbook: Investigate an AI Spend Anomaly​

  1. stop or disable the affected server-enforced feature where a genuine kill switch exists
  2. identify model, function, user, company, and time bucket without exporting confidential prompts
  3. compare token_usage with token_usage_daily and token_usage_monthly
  4. inspect record_token_usage failures in function/agent logs
  5. check retry, WebSocket close, and partial-success paths
  6. verify user and company monthly limits
  7. reconcile provider-side usage
  8. fix every missing accounting path and backfill only through a reviewed, auditable procedure

An insert into token_usage alone does not update the monthly gate. See AI and Token Accounting.

Runbook: Incident Triage and Recovery​

First 15 Minutes​

  1. name an incident owner and recorder
  2. identify affected capability, tenant scope, start time, and current release
  3. preserve logs and status without copying sensitive payloads
  4. determine whether confidentiality, integrity, availability, or spend is at risk
  5. apply the smallest reversible mitigation
  6. identify and protect the rollback target

Use Observability and Reliability to locate browser, function, database, agent, and provider signals.

Recovery Principles​

  • frontend: promote a previous build
  • functions: redeploy previous repository source with --no-verify-jwt
  • database: use a forward corrective migration
  • agent: deploy/restart a previous worker image/version
  • provider override: restore a known-good value, then restart consumers
  • compromised secret: rotate; never restore the compromised value
  • R2 or Realtime drift: reconcile durable database state before deleting data

After recovery, verify authorization, tenant isolation, accounting, audit/ history side effects, and the original user path. Monitor long enough to cover delayed jobs, cached clients, expiring tokens, and reconnects.

Runbook: Suspected Secret Exposure​

  1. classify the value and all workloads that consume it
  2. revoke/rotate it in the provider control plane
  3. update every function, worker, build environment, and local operator store
  4. redeploy/restart consumers
  5. prove the old value is rejected
  6. review access logs for the exposure window
  7. remove it from current files and coordinate any history rewrite
  8. document impact and prevention without reproducing the secret

For the known historical Supabase service-role exposure, follow docs/SECURITY_KEY_ROTATION_RUNBOOK.md. Rotation is the load-bearing response; history rewriting is coordinated cleanup and force-pushing it is destructive.

Release Record Template​

Release:
Commit:
Environment/project:
Owner:
Approved scope:
Migrations:
Functions and mandatory flags:
Frontend/docs/agent versions:
Provider or secret changes:
Feature-flag changes:
Validation performed:
Known limitations:
Rollback targets:
Post-release result:

Keep this record in the approved operational system, not in a public document when it contains internal environment details.

Final Checklist​

  • exact source commit and target verified
  • local validation matched the changed boundaries
  • migration list reviewed and forward recovery prepared
  • each changed function deployed from repository source with --no-verify-jwt
  • hosted Auth hook/OAuth settings verified where relevant
  • browser build contains no secrets or public source maps
  • one authorized and one wrong-tenant request tested
  • AI spend and aggregates reconcile
  • optional LiveKit, R2, Resend, or MCP path validated if changed
  • Sentry/log signals are present and sanitized
  • old clients and rollback targets remain usable
  • repository and operational records agree

For component-specific details, continue with Deployment, Security and Privacy, and Database Schema.

Last updated on