DeskBot is an AI-powered desktop voice assistant built using an ESP32-S3, INMP441 I2S microphone, Flask, Faster-Whisper, and a Large Language Model (LLM). The system continuously monitors audio, detects user speech, listens for a configurable wake word, and intelligently manages conversation state through automatic activation and sleep modes.
The project combines lightweight embedded hardware with GPU-accelerated speech recognition and modern language models to create a responsive, modular, and extensible voice assistant suitable for desktop automation, robotics, IoT, and embedded AI research.
- Continuous audio capture using the ESP32-S3
- Voice Activity Detection (VAD) to detect speech
- Wake word detection ("DeskBot")
- Automatic transition to conversation mode after wake word detection
- Automatic sleep mode after prolonged silence
- Low-latency HTTP communication over Wi-Fi
- GPU-accelerated Faster-Whisper speech recognition
- LLM-powered conversational responses
- Modular client-server architecture
- Easily extendable to home automation and robotics applications
+-----------------------+
| User Speech |
+-----------+-----------+
|
v
Continuous Audio Sampling
|
v
+-----------------------------+
| ESP32-S3 Client |
| INMP441 Microphone |
+-------------+---------------+
|
Voice Activity Detection
|
Speech Detected?
Yes No
| |
| Continue
| Listening
|
v
Buffer Audio Clip
|
v
HTTP Audio Upload (~80 ms)
|
v
+-----------------------+
| Flask Server |
+-----------+-----------+
|
v
Faster-Whisper Speech Recognition
|
v
Wake Word Verification
|
+------------+-------------+
| |
Wake Word Ignore Audio
Detected
|
v
Conversation Mode
|
v
Send Prompt to LLM
|
v
Generate Text Response
|
v
Wait for Additional Speech
|
Silence Timeout?
Yes No
| |
| Continue
| Conversation
|
v
Return to Passive Listening
| Component | Description |
|---|---|
| ESP32-S3 Development Board | Main embedded controller |
| INMP441 | I2S MEMS microphone |
| USB-C | Programming and power |
| Wi-Fi | Communication with local server |
- Arduino IDE
- ESP32 Arduino Core
- WiFi Library
- HTTPClient
- I2S Driver
- Python
- Flask
- Faster-Whisper
- PyTorch
- CUDA
- NumPy
- Faster-Whisper
- Gemini
- Ollama
- OpenAI-compatible LLMs
DeskBot/
│
├── ESP32/
│ ├── DeskBot.ino
│ └── audio.cpp
│
├── server/
│ ├── app.py
│ ├── pipeline.py
│ ├── whisper_utils.py
│ ├── wakeword.py
│ └── requirements.txt
│
├── audio/
│
├── docs/
│
├── README.md
│
└── LICENSE
- The ESP32 continuously samples audio from the INMP441 microphone.
- Voice Activity Detection monitors incoming audio for speech.
- Only speech segments are buffered, reducing unnecessary network traffic.
- Buffered audio is transmitted to the Flask server over HTTP.
- Faster-Whisper converts the speech into text.
- The server checks whether the wake word is present.
- Once activated, user requests are forwarded to the configured LLM.
- The LLM generates a natural language response.
- DeskBot remains in conversation mode until a configurable silence timeout expires.
- After inactivity, the assistant automatically returns to passive listening mode.
+----------------------+
| Passive Listening |
+----------+-----------+
|
Speech Detected
|
v
Upload Audio Clip
|
v
Wake Word Detected?
No Yes
| |
| v
| Conversation Mode
| |
| Additional Speech?
| Yes No
| | |
| | Silence Timer
| | |
| +------------+
| |
+---------------+
|
v
Passive Listening
git clone https://github.com/<username>/DeskBot.git
cd DeskBotpip install -r requirements.txtExample packages:
flask
faster-whisper
torch
numpy
soundfile
Open
ESP32/DeskBot.ino
Configure your Wi-Fi credentials and server address.
const char* ssid = "...";
const char* password = "...";
const char* serverURL = "http://YOUR_PC_IP:5000/upload";Upload the firmware using the Arduino IDE.
python app.pyThe server will begin listening for incoming audio requests.
Listening...
Wake word detected.
User:
DeskBot, what's the weather today?
Transcribing...
Text:
What's the weather today?
Generating response...
DeskBot:
Today's forecast is sunny with a high of 30°C.
DeskBot currently communicates between the ESP32 and the Flask server using HTTP over a local Wi-Fi network.
Measured communication performance:
| Operation | Typical Latency |
|---|---|
| HTTP Request Setup | 10–20 ms |
| Audio Upload | 50–80 ms |
| Total Communication | ~80 ms |
Since the overall speech interaction typically requires 2–5 seconds, the communication overhead introduced by HTTP is negligible. This makes HTTP a practical and straightforward solution for the current architecture while simplifying debugging and deployment.
Future versions may adopt persistent WebSocket connections for continuous audio streaming and lower protocol overhead.
Measured on a local Wi-Fi network with GPU-accelerated Faster-Whisper.
| Stage | Typical Time |
|---|---|
| Audio Upload | ~80 ms |
| Speech Recognition | 0.5–0.9 s |
| Wake Word Detection | <20 ms |
| LLM Response | 1–3 s |
| End-to-End Response | ~2–5 s |
The majority of latency originates from speech recognition and language model inference rather than network communication.
- Desktop AI assistant
- Voice-controlled robotics
- Home automation
- Accessibility tools
- Smart IoT interfaces
- Educational AI assistant
- Embedded AI research platform
- Streaming speech recognition
- Streaming LLM responses directly - until pipeline completes
- Persistent conversation memory
- Long-term user personalization
- WebSocket-based continuous audio streaming
- Context-aware multi-turn conversations
- Fully edge-deployed AI (no compute in Node ) pipeline
- ESP32-S3
- Arduino
- INMP441
- Flask
- Python
- HTTP
- Wi-Fi
- Faster-Whisper
- PyTorch
- CUDA
- Gemini
- Ollama
- Large Language Models
This project is released under the MIT License.