Skip to main content

Realtime, Multiplayer, and Voice

Navaid has three distinct real-time paths:

  1. single-user voice through the chat-live Edge Function,
  2. multiplayer text through Postgres, Realtime, and chat-multi,
  3. multiplayer voice through LiveKit and the persistent agent/ worker.

They share session/message/maturity data but have different connection, authorization, ordering, and failure semantics.

Single-user Voice​

Components​

SessionSingle / useLiveVoice
├─ microphone capture and PCM conversion
├─ chat-live WebSocket
├─ playback queue / worklet
├─ reconnect and Gemini resume handle
└─ transcript/maturity UI

chat-live
├─ query-token Auth validation
├─ session/company authorization
├─ budget gate and realtime usage recorder
├─ Gemini Live relay
├─ transcript persistence
└─ text-model maturity extraction

Connection​

The browser calls buildLiveVoiceWsUrl(sessionId,resumeHandle?). Because the WebSocket API cannot set Authorization headers, the URL includes the current access token.

Before upgrading, the function:

  1. validates token with auth.getUser,
  2. loads the session,
  3. verifies active company access,
  4. checks budget,
  5. loads prompt/history/context.

Never log or send the complete URL to Sentry.

Audio​

Browser audio is converted to the PCM format expected by Gemini Live. Returned audio is decoded/queued for playback. Audio worklets are served from frontend/public/, so CSP permits blob: workers/media where needed.

Resume​

Gemini can provide a resumption handle. The hook stores it and reconnects after transient failure. Reconnect must:

  • acquire a current Auth token,
  • avoid overlapping sockets,
  • stop old capture/playback resources,
  • preserve the session ID,
  • fall back to a fresh upstream session when the handle is rejected.

Maturity extraction​

Spoken replies do not carry the same reliable structured blocks as text chat. After an assistant turn, chat-live uses the text model to extract maturity signals and applies them with the shared optimistic persistence helper.

Both the Live call and extraction spend must be accounted.

Feature flag​

voice_enabled controls the client affordance. chat-live does not currently read the flag. It should not be described as a server kill switch.

Multiplayer Text​

Data model​

  • sessions.is_multiplayer
  • session_participants
  • attributed messages.user_id
  • multiplayer message kind/processing fields
  • turn-claim state/RPC

Creation and join​

createMultiplayerSession creates a group session and owner participant. Invited users receive participant rows/notifications. can_join_session and session_add_participant constrain self-join paths.

Every participant must also retain active membership of the session company.

Realtime subscription​

useMultiplayerSession subscribes to relevant:

  • messages
  • participants
  • presence/status as implemented

On subscription or network recovery, reload durable rows and reconcile by ID. Realtime delivery can be duplicate, delayed, or out of order; it is not a transaction log exposed directly to UI.

AI turn claim​

Several connected clients can observe the same user turn. chat-multi calls claim_session_turn() so only one request generates the next assistant turn.

new user messages
├─ client A requests AI
└─ client B requests AI
└─ claim_session_turn
├─ winner drains pending turns and calls Gemini
└─ loser receives {busy:true}

The winner composes a speaker-aware prompt, persists one assistant reply, updates maturity/tasks/accounting, and releases/advances turn state.

Attribution controls​

Authenticated client message inserts are restricted to role='user'. messages_pin_author_user_id() forces user_id = auth.uid(). The service-role assistant pipeline remains exempt.

This prevents one participant from forging another speaker or an assistant message.

Feature flag​

multiplayer_enabled is currently a client access flag. Backend operations still authorize participants and membership but do not use it as a universal server kill switch.

Multiplayer Voice​

Why a separate worker​

LiveKit Agents holds a room connection and a Gemini Live socket for the duration of a session. Supabase Edge Functions are request scoped and unsuitable for that lifecycle.

agent/src/agent.ts is a Node worker using:

  • @livekit/agents
  • Google Live plugin
  • Silero VAD
  • Supabase JS service client

Room naming​

livekit-token accepts:

  • controlled lab-* rooms for the hidden prototype
  • session-<uuid> for real group sessions

Arbitrary names are rejected.

Token minting​

For a real session room, livekit-token verifies:

  • signed-in user
  • LiveKit configuration
  • group_voice_enabled unless system admin
  • existing multiplayer session/company
  • active company membership
  • owner or participant row
  • budget

It returns a short-lived room token, URL, identity, display name, and canonical room.

Agent session authorization​

For session-* rooms the agent:

  1. parses the session ID,
  2. loads session/company/profile/prompt context through the service role,
  3. refuses session audio when the owner/session authorization context is invalid,
  4. rechecks a speaker's participant/membership relationship before linking and before persisting a turn.

Service-role access makes these checks mandatory.

Speaker attribution​

Participant identity and metadata are minted by the token function. The worker:

  • tracks active human speakers,
  • links agent input to the active participant,
  • binds a transcript segment to the speaker when it opens,
  • falls back carefully for final-only transcripts,
  • drops turns without a trustworthy speaker,
  • revalidates the speaker before message insert.

Transcript persistence​

Human and assistant turns are inserted into messages with:

  • session ID
  • correct role
  • attributed user where applicable
  • voice source metadata
  • interruption marker for interrupted assistant output

Messages then flow to connected clients through Realtime.

Maturity context​

The worker composes instructions from:

  • active v2 advisor prompt
  • company facts and maturity summary
  • UK-readiness scope guardrail
  • session identity

It persists speech; group maturity processing remains coordinated with application session flows rather than allowing arbitrary model writes.

Usage accounting​

The worker estimates Live usage conservatively, inserts audit rows, and calls record_token_usage. It rechecks authorization and budget during the session so a room token is not indefinite authority.

Current dispatch caveat​

The worker README notes that scoping dispatch exclusively through named/explicit session-* dispatch is future hardening. Do not claim it is already complete.

Leave and Remove​

livekit-session-control is the only authoritative leave/remove path.

ActionAllowed caller
Leave selfcurrent participant
Remove othersession owner or company admin/consultant/system admin

The function:

  1. verifies session/company/action,
  2. deletes participant state with service privilege,
  3. calls LiveKit to remove all matching connected identities,
  4. reports whether media-plane removal occurred.

A direct table delete would leave an already-connected participant in the room.

Notifications​

notify-session-add sends branded participant mail only after confirming:

  • caller identity,
  • session existence,
  • caller's owner/admin authority,
  • target is owner or has a participant row,
  • trusted APP_URL.

This prevents using Navaid mail as an arbitrary-recipient spam relay.

Failure and Recovery​

Realtime disconnect​

  • show connection state,
  • resubscribe once,
  • reload durable rows,
  • dedupe by ID,
  • do not generate AI solely from an unverified duplicate event.

Gemini Live disconnect​

  • stop/replace the upstream socket,
  • attempt one bounded resume path,
  • preserve already-persisted transcripts,
  • flush usage,
  • avoid replaying already-committed turns.

LiveKit agent absent​

Humans may connect without an AI participant. UI should show agent readiness rather than pretending the assistant is listening.

Membership revoked mid-room​

Short room-token TTL limits initial authority. Agent/function checks and authoritative session control must remove/refuse the participant promptly.

Duplicate assistant replies​

Inspect claim_session_turn use and whether multiple clients bypassed the request helper.

Local Development​

Single voice:

npx supabase functions serve --no-verify-jwt
cd frontend
npm run dev

Group voice additionally:

cd agent
pnpm dev

Required LiveKit/Gemini/Supabase values are documented in Getting Started.

Use synthetic companies/transcripts for live acceptance tests.

Security Checklist​

  • no access token in logs/analytics
  • current auth.getUser before WebSocket upgrade
  • active company membership
  • participant/owner binding
  • strict room-name validation
  • short token TTL
  • speaker identity not taken from transcript text
  • membership recheck before persistence
  • leave/remove affects database and media plane
  • prompt scope/tenant context validated
  • usage recorder single-instance and flushed
  • canonical CORS/CSP origins

Extension Guidance​

When adding a real-time event:

  1. define the durable row/state first,
  2. make event handling idempotent,
  3. tolerate duplicate/out-of-order delivery,
  4. establish one-winner semantics for side effects,
  5. bind actor and tenant server-side,
  6. specify reconnect/replay behaviour,
  7. account provider spend,
  8. test two tabs and two users concurrently.