A fully-local, voice-driven AI desktop assistant and study companion. Speech-to-text, text-to-speech, reasoning, speaker identification, and memory all run on-device — no cloud dependency for core operation. Claude is an optional quality tier for complex reasoning.
Aria's minimal Samaritan-inspired interface — one accent colour, real state, nothing else.
Aria proposes. Chan decides.
- Fully local stack — faster-whisper
large-v2on CUDA for STT, Kokoro ONNX for TTS, local Ollama (mistral) for live reasoning,resemblyzerfor speaker ID,fastembedfor semantic routing. Optional: Claude for complex/format-heavy answers, Gemini for screen vision. - Capability-plugin architecture — skills self-register at the edge (
capabilities/); the core router stays domain-agnostic. Routing is keyword claims → semantic match → generalist fallback. No central intent map to maintain. - Voice permission tiers — Aria identifies who is speaking from a local voiceprint and gates every action accordingly: the owner gets full control; any other voice gets question-answering only, with a polite refusal for anything private or state-changing.
- Study capture — during a study session Aria transcribes locally, distils teaching content into tagged notes (local LLM — free, private, unlimited), then quizzes you on them and reports where your gaps were.
- Live companion mode — in a meeting or call, Aria hears both the mic and the system loopback (what you hear), buffers everything for context, and answers questions on screen or aloud — with speaker ID deciding what each voice may ask.
- Session memory — when a live session ends, it's auto-distilled into categorised, decaying memory (knowledge / facts / meetings / commitments). Crash-resilient: checkpoints during the session, emergency save on any exit, recovery on next start. Ask "what did I agree to?" the next day and it knows.
- Passive developmental observation — a second memory layer models growth over time: recurring knowledge gaps, topics you return to, gaps that later close. Patterns require code-verified recurrence across multiple sessions before the model may even phrase them, and every claim is traceable to its evidence ("why do you think that?").
- Transcription glossary — teaches Whisper your proper nouns (names, companies, tools) via prompt seeding plus conservative post-correction, so "Orca" doesn't become "Oracle". The real glossary is local and gitignored.
- Minimal terminal UI — state, three mode indicators, the response (in full — expands, then scrolls), and a command line. Diagnostics live in
aria.logand an auto-generated per-session digest, not on screen.
Aria listens for her name ("Aria, …") — or hears everything in conversation/live mode. Typed input goes through the identical router. Phrasings below are the real registered triggers (representative, not exhaustive).
| Say | Does |
|---|---|
| "live mode on / off" | Live companion: fast local answers, loopback listening, session memory capture |
| "conversation mode on / off" | Hands-free conversation without repeating her name |
| "analysis mode on / off" | Passive screen analysis + study capture session |
| "speak answers" / "screen answers" | How live answers are delivered (aloud vs silently on the dashboard) |
| "what time is it" / "what's today's date" | The basics (open to any voice) |
| "show my reminders" / "my schedule" | Calendar & reminders |
| "what's the weather" | Local weather (open to any voice) |
| "search for …" / "look up …" | Web search (open to any voice) |
| "what's on my screen?" | Screen-assist: captures the active window and explains it (Ctrl+Alt+A hotkey does the same with a spoken question) |
| "explain this code" / "walk me through this" | Teaching-mode code explanation, saved to a personal study library |
| "what did you notice?" | Reads queued proactive insights |
| "goodbye" / "shut down" | Clean exit (session memory saves first) |
Anything that matches nothing specific is answered as a general question — locally in live mode (~1s), or by Claude otherwise. Asking for depth or format ("in 3 bullet points", "go deeper") escalates to the quality tier.
| Say | Does |
|---|---|
| "I'm studying [topic]" | Names the topic for the study session's notes |
| "quiz me" / "quiz me on [topic]" | Interactive quiz generated from your own study notes |
| "where were my gaps?" / "what did I get wrong?" | Gap analysis from the session's Q&A |
type authorise capture (or say "authorise meeting capture") |
Affirm, once per session, that you are authorised to record — no transcript is written and no meeting recorder starts before this |
| "start / stop transcribing" | Marks a meeting segment in the live transcript (or runs the legacy recorder outside live mode) |
| "summarise the last meeting" | Meeting → study notes |
| "what did I agree to?" / "what was discussed?" | Recall from stored session memory |
| "what sessions do you have?" | Inspect stored sessions (with expiry) |
| "forget that session" | Delete a stored session (always logged, never silent) |
| "what patterns have you noticed?" / "what am I struggling with?" / "where have I improved?" | Developmental insights — only ever claims patterns with real multi-session evidence |
| "why do you think that?" | The actual evidence behind the last insight |
| "drop that pattern" | Retire an insight permanently |
| "add [name] to the glossary" | Teach Whisper a new proper noun |
python main.py --enrol-voiceRecords five short clips — each prompt varies pace, distance and register on purpose — and stores a local voiceprint (data/speaker_profile.json — gitignored, never leaves the machine). This is what powers the permission tiers below and the "Chan" label in transcripts. Re-enrol with more clips (--samples 8) if real sessions show many "uncertain" scores. Without enrolment, every voice is treated as the owner (no lockout).
Every spoken turn is speaker-identified (~25ms, CPU) and classified owner or other before any capability runs:
| Open to any voice | Owner-only | |
|---|---|---|
| Examples | time, date, weather, web search, current-event follow-ups, general questions | mode toggles, calendar & notifications, screen/vision, code explanation, meeting control & summaries, study capture/quiz/gaps, session-memory recall & delete, developmental queries, glossary edits |
| Principle | general knowledge, no state, no private data | anything that reads the owner's private data or changes state |
A guest asking "what's the difference between TCP and UDP?" gets a real answer. The same guest saying "analysis mode off" or "what did Chan agree to?" gets a polite refusal. New capabilities default to owner-only — open is opt-in per intent.
mic ─┐ ┌─ Kokoro TTS (CUDA) → speakers
├─ VAD gate → Whisper large-v2 (CUDA) ──┤
loopback ─┘ │ └─ minimal terminal UI
▼
speaker ID (local voiceprint, CPU)
▼
permission tier (owner / other)
▼
router: keyword claims → semantic match (CPU embeddings) → fallback
▼
capability plugins: assistant · study · quiz · meeting · vision ·
code-explanation · web · session-memory · developmental · reasoning
▼
local Ollama (live mode, ~1s) · Claude (optional quality tier)
Cross-cutting: session memory with checkpointed crash recovery, a per-session diagnostic digest generated at shutdown, structured self-review events, and aria.log as the raw diagnostic stream. Runtime data (memory, notes, voiceprint, glossary, captures) lives under data/ — gitignored, on-device only.
Requirements: Windows 11 · Python 3.13 · NVIDIA GPU strongly recommended (Whisper + Kokoro + Ollama share it; ~12GB VRAM is comfortable) · Ollama with mistral pulled · optional Anthropic/Gemini API keys for the cloud tiers.
git clone https://github.com/chansg/aria.git
cd aria
python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -r requirements.txt
Copy-Item config.example.py config.py # then edit locally — never committed
Copy-Item glossary.example.json data\glossary.json # optional: your vocabulary
python main.py # starts straight into listening on the default mic
python main.py --pick-mic # one-off interactive device pick
python main.py --enrol-voice # one-time voiceprint enrolmentconfig.example.py documents every setting (models, thresholds, retention windows, live-mode behaviour) with the reasoning behind each default.
GPU note:
fastembedpulls the CPUonnxruntime, which can shadowonnxruntime-gpuand drop Kokoro off CUDA. Fix:pip uninstall -y onnxruntime; pip install --force-reinstall --no-deps onnxruntime-gpu.
python -m tools.export_session --latest # newest session -> data/transcripts/exports/<id>.md
python -m tools.export_session --session 20260817-130002 # by session id (or a date, or a session-memory id)
python -m tools.export_session --session 2026-08-17 --format json
python -m tools.export_session --find CVE-2024-1234 # which sessions mention this entity
python -m tools.loopback_probe # prove call audio can be captured on this machineOne self-contained file: participants (labels, never identities), matched entities, the distilled summary with the turns each item came from, actions, meeting segments, and the verbatim transcript. No embeddings. Entities are matched by ENTITY_PATTERNS in your local config (the shipped default is only the generic CVE pattern — add your own issue-key patterns there, never in the repository) and indexed in data/transcripts/entity_index.jsonl so sessions stay searchable across weeks. A run with several meetings is split into separate session-memory records after SESSION_IDLE_SPLIT_MINUTES of silence.
python -m pytest -q # 700+ unit, routing, capability, and resilience testsThe suite includes real process-kill tests for session-memory recovery and an adversarial corpus proving the glossary never corrects ordinary English.
Everything that matters stays on the machine: transcription, reasoning (live mode), speaker identification, all memory. Raw audio is never persisted. The voiceprint, glossary, notes, transcripts and memory live in gitignored data/. The optional cloud tiers (Claude, Gemini) see individual queries only — never the microphone stream, never stored memory beyond the context of the question asked.
Meeting capture is authorised per session, and evidenced. In live mode every heard turn — your mic and call audio — is kept as a verbatim, timestamped, speaker-labelled transcript (data/transcripts/<session-id>.jsonl, one line per turn, flushed as it is heard); distilled session memory is a derived view of it. No transcript is written and no meeting recorder starts until you affirm, once per session, that you are authorised to record (type authorise capture; the disclaimer is shown alongside the prompt); the affirmation — time, method, and the version of the recording disclaimer in force — is written into the transcript header. Decline or forget, and Aria runs normally with no transcript persisted; there is no partial transcript. (Distilled session memory is unchanged by this and keeps its own privacy notice and decay.) Meeting segments open automatically from call activity, so no spoken trigger is ever needed. Speakers are labelled, never identified: the owner is "Chan"; every other voice is "Speaker 01", "Speaker 02"… separated by clustering utterance voiceprints within the session only — those voiceprints are discarded at exit, nothing is stored for anyone but the owner, and names heard in the conversation are never used to label a speaker (so "Speaker 01" today may be "Speaker 02" tomorrow, by design). Turns the voiceprint cannot verify are "Unknown", not the owner. Whisper garble on non-speech is kept verbatim but flagged and excluded from distillation.
- Accurate observability over silent failure — every fault is visible in
aria.logand the session digest. - Measure, then tune: thresholds (speaker match, recall relevance, glossary correction) are set from logged real data, not guesses.
- A false pattern is worse than no pattern — evidence-gated claims everywhere the system models its user.
- Small reviewed changes; a test for every bug found in real use.
- The core stays domain-agnostic; skills are plugins at the edge.