Gemini Live voice app with React and FastAPI

This is a small runnable example of a browser voice UI that talks to Gemini Live through your own backend instead of wiring Gemini directly into React.

The shape is intentionally simple:

React voice UI
  -> your own WebSocket protocol
  -> FastAPI
  -> Gemini Live

The important part is the boundary. React records microphone audio, sends product-shaped messages to FastAPI, and plays product-shaped events back. FastAPI owns the Gemini Live session, the API key, the model name, the system instruction and the SDK response shape.

So the frontend never sees your Gemini API key, never chooses the model, never sends the prompt, and never needs to know what a Gemini response object looks like.

What you get

• A FastAPI backend with WS /api/voice
• A React/Vite frontend with a hold-to-talk button
• Browser microphone capture as PCM16 audio
• Scheduled PCM playback for Gemini's audio response
• A tiny app-owned WebSocket protocol between React and FastAPI
• Basic demo validation for message size, base64 input, sample rate and PCM alignment

This is not a production voice app. It has no authentication, no rate limiting, no persistence, no billing protection and no mock provider for tests. It is a runnable starting point for the provider boundary.

Project shape

.
├── .env.example
├── .gitignore
├── LICENSE
├── README.md
├── backend
│   ├── main.py
│   ├── pyproject.toml
│   └── uv.lock
├── docs
│   └── screenshot.png
└── frontend
    ├── index.html
    ├── package-lock.json
    ├── package.json
    ├── tsconfig.json
    ├── vite.config.ts
    └── src
        ├── App.tsx
        ├── audio.ts
        ├── main.tsx
        ├── styles.css
        └── vite-env.d.ts

Requirements

• Python 3.11+
• uv
• Node.js 20+
• a Gemini API key with Live API access
• a browser that supports getUserMedia and Web Audio APIs

Chrome is the easiest browser to start with for this demo. Browser audio APIs have enough weird edges that I would not start cross-browser polishing before the basic loop works.

Setup

From the repo root:

cp .env.example .env

Edit .env and set GEMINI_API_KEY:

GEMINI_API_KEY=your-key-here
GEMINI_LIVE_MODEL=gemini-3.1-flash-live-preview

Do not put the Gemini API key anywhere in frontend/. The backend reads it from .env and uses it server-side only. The demo system instruction lives in backend/main.py, because prompts are application code in this example, not local runtime secrets.

Run it

Start the backend in one terminal:

uv run --project backend uvicorn main:app --app-dir backend --reload --host 127.0.0.1 --port 8000

If you do not use uv, create a normal virtualenv instead:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e backend
python -m uvicorn main:app --app-dir backend --reload --host 127.0.0.1 --port 8000

The backend exposes:

• GET /api/health
• WS /api/voice

Start the frontend in another terminal:

npm --prefix frontend install
npm --prefix frontend run dev

Open http://127.0.0.1:5173, hold the button, talk, then release it.

Vite proxies /api/* and WebSocket upgrades to FastAPI on port 8000, so the browser still connects to a relative /api/voice path. That is deliberate, because it keeps the frontend shaped like a same-origin app instead of hardcoding provider or backend origins everywhere.

How the demo works

1. React opens your WebSocket, not Gemini's

frontend/src/App.tsx opens:

ws://127.0.0.1:5173/api/voice

In development, Vite forwards that to FastAPI:

// frontend/vite.config.ts
proxy: {
  '/api': {
    target: 'http://127.0.0.1:8000',
    changeOrigin: true,
    ws: true,
  },
}

The browser never opens a socket to Google.

2. The browser sends product messages

The browser sends audio chunks and an explicit turn-end message:

type ClientMessage =
  | { type: 'audio_chunk'; audio_base64: string; sample_rate_hz: 16000 }
  | { type: 'audio_turn_end' }

Those names describe what the user is doing in the app. They are not Gemini event names.

3. FastAPI translates product messages to Gemini input

backend/main.py accepts the browser WebSocket, opens a Gemini Live session and forwards validated PCM16 chunks through session.send_realtime_input(...).

The backend is also where the system instruction and model name live:

config = types.LiveConnectConfig(
    response_modalities=[types.Modality.AUDIO],
    system_instruction=types.Content(parts=[types.Part(text=SYSTEM_INSTRUCTION)]),
    output_audio_transcription=types.AudioTranscriptionConfig(),
)

That is the main point of this example. Product prompt text and provider configuration stay server-side.

4. FastAPI translates Gemini output back to product events

The browser receives these events:

type ServerEvent =
  | { type: 'tutor_audio_chunk'; audio_base64: string; sample_rate_hz: number }
  | { type: 'tutor_transcript'; text: string }
  | { type: 'turn_completed' }
  | { type: 'error'; code: string; detail: string }

React only needs to know how to play tutor_audio_chunk, append tutor_transcript, and react to turn_completed. If Gemini moves a field around in the SDK response, the fix belongs in backend/main.py, not in your React components.

Audio notes

Gemini Live expects raw PCM16 mono audio at 16kHz for input, and returns raw PCM16 audio at 24kHz for output.

This demo uses ScriptProcessorNode to keep the audio capture code easy to read. For production, use AudioWorkletNode.

The demo records microphone audio with an AudioContext and ScriptProcessorNode, then resamples to 16kHz before sending.

Playback uses a small scheduled PCM player in frontend/src/audio.ts. Each returned audio chunk is scheduled where the previous one ended, otherwise the tutor voice tends to stutter between chunks.

Troubleshooting

missing_gemini_api_key

Copy .env.example to .env and set GEMINI_API_KEY. The backend intentionally refuses to start a voice session with the placeholder value.

WebSocket failed

Make sure the backend is running on 127.0.0.1:8000 and the frontend is running through Vite on 127.0.0.1:5173. The frontend expects Vite's proxy to forward /api/voice to FastAPI.

The browser asks for microphone permission after a quick tap

The UI is push-to-talk, but browsers still control the permission prompt. The app cancels stale recording starts if you release early, but the browser may still show the permission prompt once capture has been requested.

No audio plays

Make sure you start the interaction from the button. Browsers can keep an AudioContext suspended until a user gesture resumes it, so playback is resumed from the pointer interaction.

The backend rejects the sample rate

This demo sends 16kHz PCM16 audio. The frontend resamples before sending and includes sample_rate_hz: 16000; the backend rejects unexpected rates so the failure is obvious during local debugging.

What is deliberately missing

This demo has a little validation because otherwise broken browser messages are annoying to debug. That is not the same thing as a production security boundary.

Before using this pattern for real users, add:

• authentication and ownership checks
• WebSocket Origin validation
• per-user usage limits and billing protection
• edge/request-size limits aligned with backend limits
• durable session and conversation state
• reconnection behavior
• barge-in, so users can interrupt the tutor mid-answer
• safe logging that does not leak prompts, raw audio, transcripts or provider payloads
• a mock provider so CI does not spend tokens

If you copy anything from this repo, it should be the boundary: browser speaks your protocol, backend speaks Gemini.