Observability and Reliability

Navaid spans a static browser application, Supabase database and Edge Functions, Gemini APIs, LiveKit, Cloudflare R2, Resend, and an optional voice agent. Reliability therefore depends on being able to distinguish browser, authorization, persistence, provider, and accounting failures without exposing the confidential content being processed.

Current Telemetry Surfaces

Surface	Current signal	Important limitation
Browser	Sentry errors, traces, masked replays	sampled; DSN/build credentials must be configured
Frontend release	version.json polling and stale-chunk recovery	detects a new bundle, not backend compatibility
Edge Functions	Supabase function logs and HTTP status	console logging is uneven; no shared correlation ID
Database	token aggregates, profile history, admin audit, status rows	business records are not a replacement for metrics
LiveKit agent	prefixed console warnings/errors	no repository-defined central exporter
Gemini/LiveKit/R2/Resend	response status and provider dashboards	access and retention live in separate control planes

This source tree does not prove that remote alerts, dashboards, log drains, or provider monitors are configured. Verify each environment before relying on it during an incident.

Browser Sentry

frontend/src/instrument.ts initializes Sentry before the React application. The current privacy and sampling contract is:

sendDefaultPii: false
tracesSampleRate: import.meta.env.PROD ? 0.2 : 1.0
replaysSessionSampleRate: 0.1
replaysOnErrorSampleRate: 1.0

Session Replay uses both maskAllText: true and blockAllMedia: true. These settings protect transcripts, assessment content, uploaded-document text, and company strategy from appearing in replay recordings. Do not weaken them to make a debugging session more convenient.

Browser tracing propagates toward localhost and the configured Supabase project. Every Edge Function's CORS response must continue to allow baggage and sentry-trace. That enables the browser request; it does not by itself create server spans. The checked-in functions currently rely on Supabase logs rather than shared Sentry server instrumentation.

React 19's onUncaughtError, onCaughtError, and onRecoverableError hooks forward errors to Sentry. A top-level Sentry.ErrorBoundary gives users a safe fallback while retaining the real error for diagnostics. Handled failures should go through frontend/src/utils/logger.ts so they are not visible only in the developer console.

Source Maps

Production builds create hidden source maps. When SENTRY_AUTH_TOKEN, SENTRY_ORG, and SENTRY_PROJECT are present, the Vite plugin uploads them and deletes them from dist. Without a token, a fallback plugin strips .map files. A release check should prove that no source maps are in the published asset set.

The Sentry DSN is a browser ingestion identifier and uses the VITE_ prefix. The source-map auth token is a build secret and must never use that prefix.

Frontend Release Reliability

Every Vite build writes a unique dist/version.json. UpdatePrompt polls it without cache every 60 seconds and offers a refresh when the version changes. This reduces the time users remain on an old application shell.

Lazy routes can still request a hashed chunk removed by a new deployment. vite:preloadError triggers a guarded single reload. A second failure falls through to the error boundary rather than entering a reload loop. Keep the unit coverage in chunkReload.test.ts when changing this mechanism.

During release verification:

1. open the current application before deployment
2. deploy a new bundle
3. confirm version.json changes and is not cached
4. navigate through a lazy route from the old tab
5. verify one recovery refresh at most
6. verify the error boundary remains usable for unrelated failures

Edge Function Logging

Functions use prefixed console.log, console.warn, and console.error messages in Supabase's runtime logs. Prefer stable event names and metadata:

console.error('[generate-summary] profile_sync_failed', {
  sessionId,
  companyId,
  errorCode,
})

Log identifiers only when operationally necessary. Never log:

• bearer, OAuth, invitation, session-share, R2, or LiveKit tokens
• Gemini, Supabase, Resend, R2, Sentry, or LiveKit secrets
• raw prompts, transcripts, document text, email bodies, or model responses
• database connection strings or provider response headers
• full internal errors returned to the caller

Treat URLs as sensitive too: query strings may contain capability tokens and competitor URLs may disclose a customer's strategy. Parse and allowlist fields before logging.

The current functions do not share a generated request/correlation ID. When adding one, accept a safe inbound identifier only if its shape and length are bounded; otherwise generate it server-side. Return it in a non-sensitive error envelope and include it in every downstream log. Do not use a user-controlled ID as authorization evidence.

Durable Operational Records

Several tables are useful when reconstructing business-side effects:

Token Usage

• token_usage is the per-request audit trail.
• token_usage_daily and token_usage_monthly are aggregate budget views.
• record_token_usage updates the aggregate path used by enforcement.

If an AI request succeeded but monthly totals did not change, treat it as an accounting defect. Compare the per-row insert, RPC result, model identifier, user/company attribution, and calendar bucket. See AI and Token Accounting.

Company Knowledge History

company_profile_history versions fact, readiness, learning, and maturity changes from sessions, admin edits, and MCP writes. It supports provenance and reconciliation after a partial profile update. It is not an unrestricted application log: inserts and change-source values are controlled.

Administration Audit

admin_audit_log records system-administration actions through the log_admin_action SECURITY DEFINER RPC, which stamps the authenticated actor and server time. Direct browser inserts are denied. The current frontend helper logs audit failures but does not fail the mutation, so absence of an audit row can indicate a partial operation and should be monitored rather than assumed impossible.

Workflow State

Session status, summary_claimed_at, task status, invitation consumption, R2 document state, and participant rows provide recovery evidence. Query them with the least privileged operational role available. Avoid “fixing” a status by editing production rows until the invariants and retry semantics are understood.

Reliability Patterns

Atomic Claims and Idempotency

Expensive or destructive operations require one winner:

• summary generation uses claim_session_summary; stale claims are reclaimable after ten minutes and hard failures release the claim best-effort
• multiplayer turns use a server-side claim/lock path
• invite-code consumption is idempotent per code and email
• MCP fact updates avoid version bumps when a retry changes nothing
• demo teardown is designed to tolerate repeated cleanup

When adding a retryable endpoint, make the idempotency decision in the database transaction, not with a read-then-write check in TypeScript.

Optimistic Concurrency

Parallel text, voice, summary, admin, and MCP updates can touch the same company profile. Shared persistence performs version-aware retries, with a documented last-writer-wins fallback after bounded conflicts. Preserve provenance and test simultaneous updates; do not silently replace this with a blind overwrite.

Bounded Failure

• rate limits bound expensive request bursts
• monthly user/company budgets stop supported AI paths before provider spend
• Promise.allSettled lets lead discovery tolerate a failed search angle
• provider responses are parsed and validated before persistence
• non-critical task generation may fail without losing the primary chat turn
• Realtime consumers refetch durable rows rather than treating a broadcast as the source of truth

State explicitly whether a downstream failure is fatal, retryable, or best-effort. A swallowed error must still produce a signal if it can create drift.

Streaming and Disconnects

Live functions meter elapsed model connection time and stop the usage recorder on normal close, client error, upstream error, and exception paths. Transcript buffers attempt a final persistence on close. Test abrupt browser disconnect, provider disconnect, repeated close callbacks, and accounting failure.

For multiplayer voice, the agent rechecks participant authorization rather than assuming a room token remains valid forever. Dropping an unowned transcript is safer than attaching speech to the wrong user.

Health and Alerting Gaps

The Admin UI stores daily_token_budget and alert_config in system_config, and a budget_alerts table exists. Current main has no runtime daily-budget enforcer or alert sender consuming those settings. Monthly budget enforcement is the active control. Do not write a runbook that assumes daily alerts will fire.

Other gaps that should be treated explicitly:

• no repository-defined synthetic health check spanning browser to database
• no shared Edge Function correlation middleware
• no checked-in centralized log retention policy
• no code-defined SLOs or paging thresholds
• Sentry covers the browser, not every server/provider hop

Adding a dashboard without an owner and response threshold does not close an operational gap.

Suggested Service Indicators

Define targets in the deployment environment, then measure at least:

Capability	Indicators
Sign-in	OTP request success, verification success, invite-preflight denials
Text chat	latency, 2xx/4xx/5xx, Gemini error rate, accounting success
Summary	claim conflicts, completion latency, profile/history sync failures
Live voice	connection setup, disconnect reason, transcript persistence, metering
Multiplayer	room join, participant authorization denial, agent dispatch, turn lock
Documents	sign/confirm latency, R2 status transitions, quota denials
Email	Resend acceptance and function failure by template
MCP	OAuth challenge, token validation denial, tool latency/error by name
Frontend	error rate by release, Web Vitals/traces, stale-chunk recovery

Use counts and bounded identifiers, not sensitive payloads, as dimensions. High-cardinality user, session, URL, and document labels increase cost and can create a privacy problem.

Incident Triage

Work from the outside in:

1. record the environment, release/version, time window, capability, role, and tenant without copying confidential content into the incident channel
2. check browser network status and Sentry release/error
3. identify the Edge Function and inspect its bounded logs
4. verify authorization, rate, and budget decisions
5. inspect durable workflow rows and claims
6. check the relevant provider control plane
7. compare token accounting and audit/history side effects
8. reproduce in an isolated environment
9. mitigate or roll back using Operations Runbooks

Common signatures:

Symptom	First checks
Browser CORS failure	OPTIONS, canonical headers, baggage, sentry-trace
Hosted function 401	--no-verify-jwt deployment and function-level auth
Chat 429	rate bucket, monthly user/company usage, concurrent turns
Summary remains active	claim time, provider error, profile-history write
Voice closes immediately	browser token, Gemini key/model, WebSocket close reason
Participant heard but not saved	participant mapping, authorization recheck, agent log
Upload stuck	document storage status, signed URL age, R2 HEAD/confirm
New UI breaks after release	version polling, hashed chunk, error boundary, source map

Adding Telemetry

For a new capability:

1. define the user-visible success and durable completion
2. choose a stable operation/event name
3. log authorization denials separately from provider failures
4. propagate or generate a safe correlation ID
5. measure latency, outcome, retry, and partial side effects
6. add a privacy review for every field
7. add alert ownership and a tested response runbook
8. verify telemetry failure cannot expose data or break the primary operation

Keep Security and Privacy and Testing and Accessibility aligned whenever a new observability provider or replay mechanism is introduced.