Skip to content
BranchesTags

Repository files navigation

OpenSQZ Glass

Wearable sensing. Nearby local intelligence.

An open-source research platform for local-first visual assistance, connecting lightweight first-person sensing with multimodal inference on a nearby laptop or edge host.

中文 · Quick Start · Hardware Guide · ACL 2026 Paper · Safety & Privacy · Roadmap

Status ACL 2026 Sensing Default runtime Inference

OpenSQZ Glass 3D-printed prototype viewed from the front

OpenSQZ Glass 3D-printed frame and sensing hardware.

Important: The large model does not run on the ESP32 glasses. OpenSQZ Glass uses a sensing-computing split: the wearable captures first-person image and audio streams, while a nearby user-controlled computer performs local inference and speech generation.

Research boundary: OpenSQZ Glass is a research prototype. It is not a certified navigation aid, medical device, safety-critical system, or production-ready skill platform.

News

Overview

OpenSQZ Glass brings several related research directions into one repository without treating them as one finished product. The common idea is simple:

OpenSQZ Glass system overview adapted from the UbiComp/ISWC Figure 1

System overview for the UbiComp/ISWC hardware direction, adapted from Figure 1. Network credentials are replaced with a public-safe local-configuration prompt.

On the glasses On the nearby host In this repository
ESP32-S3 camera and PDM microphone capture first-person context A laptop or edge host runs ASR/VLM/TTS or MiniCPM-o locally Firmware, host bridges, evaluation tools, experimental runtime, session replay, and hardware documentation

The sensing device, model backend, and publication track are independent axes. An ESP32 is a device, MiniCPM-V and MiniCPM-o are model paths, and ACL or UbiComp/ISWC identifies a research snapshot rather than a separate product fork.

flowchart LR
  subgraph D["Wearable sensing"]
    ESP["ESP32-S3 glasses\ncamera + microphone"]
    ROKID["Rokid\nplanned public adapter"]
    RAYNEO["RayNeo\nplanned adapter"]
  end

  ESP --> BRIDGE["OpenSQZ host bridge"]
  ROKID -.-> BRIDGE
  RAYNEO -.-> BRIDGE

  subgraph H["Nearby laptop / edge host"]
    BRIDGE --> CORE["Core path\nMiniCPM-V 4.5\nmodular ASR / VLM / TTS"]
    BRIDGE --> OMNI["Omni path\nMiniCPM-o 4.5\nllama.cpp-omni"]
    CORE --> OUT["Local speech output"]
    OMNI --> OUT
    BRIDGE --> SESSION["Local logs and replay"]
  end
Loading

Solid arrows represent code or artifacts present in the public repository. Dashed device links are planned and must not be interpreted as released support.

Tracks and Maturity

Research direction Device Model/backend Scope Maturity
ACL 2026 / OpenGlass-Core ESP32-S3 glasses MiniCPM-V 4.5, modular ASR/VLM/TTS Sensing-computing split, local visual assistance, evaluation and latency artifacts Published research baseline; reproduction artifacts available
UbiComp/ISWC Hardware OpenSQZ 3D-printed ESP32 frame Backend-independent CAD, print plate, BOM, module placement, assembly and validation documentation Public draft; several hardware facts still require verification
OmniRuntime / ESP32 ESP32-S3 glasses MiniCPM-o 4.5 + llama.cpp-omni Control panel, live multimodal bridge, prompt switching, recording and replay Experimental; observed on the maintainer setup, clean-machine validation pending
OmniRuntime / Rokid Rokid glasses MiniCPM-o 4.5 APK-to-host bridge and shared runtime Planned public integration; required bridge/APK source is not in the current public tree
Future device adapters RayNeo and other glasses To be selected Additional device-specific transport adapters Planned; no public implementation yet

Current development default: MiniCPM-o 4.5 is the default experimental runtime path. The ACL 2026 system is a distinct, reproducible research snapshot built around MiniCPM-V 4.5; it is one part of OpenSQZ Glass, not the identity of the whole project.

Device Status

Device family Public artifacts Current status
ESP32-S3 prototype Camera/PDM firmware, device registry format, host bridge, evaluation scripts, CAD/BOM/docs Primary public sensing path; local Wi-Fi and DHCP configuration required
Rokid Launcher profile and documentation references Not runnable from a clean public clone because the required bridge source and APK are not currently published
RayNeo No adapter source yet Reserved as a future device boundary; not currently supported

Quick Start

The control panel is intended to make repeated runs one-click after a one-time setup. It does not download model weights, clone upstream repositories, compile llama.cpp-omni, or flash the ESP32 for you.

Current clean-clone status: The panel UI starts from the repository root, but the current launcher still reads machine-specific paths from runtime/openglass_omni/panel.py. The included runtime.local.json loader is not yet connected to that panel. Follow the effective configuration locations below; a code update is still required before this can be called a portable one-click installation.

1. Prerequisites

The currently exercised maintainer path uses:

  • Windows 11, Python 3.10, and an activated Conda environment.
  • An NVIDIA GPU and CUDA-capable llama.cpp-omni build.
  • Visual Studio 2022 C++ Build Tools and CMake.
  • Arduino IDE with ESP32 board support for firmware flashing.
  • MiniCPM-o 4.5 GGUF weights stored outside this repository.
  • A local Wi-Fi network shared by the host and ESP32 glasses.

Model weights are not distributed by this repository.

2. Clone the V2 upstream master branches

Keep all three repositories independent. Do not copy OpenSQZ Glass files into MiniCPM-o-Demo.

The public launcher follows the current V2 process chain: llama-omni-server -> worker -> gateway -> demo. It targets the maintained master branches of both upstream projects. The three-process V1 chain used for the WAIC demonstration (worker launching llama-server itself) is a historical setup, not the default installation path documented here.

git clone --branch master https://github.com/tc-mb/llama.cpp-omni.git
cd llama.cpp-omni
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON -DLLAMA_CURL=OFF
cmake --build build --config Release --target llama-omni-server -j
cd ..

git clone --branch master https://github.com/OpenBMB/MiniCPM-o-Demo.git
cd MiniCPM-o-Demo
python -m pip install -r requirements.txt
cd ..

git clone https://github.com/OpenSQZ/OpenGlass.git
cd OpenGlass
python -m pip install -r runtime/openglass_omni/requirements.txt

Because upstream master branches can change, record the exact commit SHAs used for every validated OpenSQZ Glass runtime release. Upstream changes may alter ports, arguments, protocols, TTS behavior, or process ownership.

3. Prepare the model files

Place the MiniCPM-o 4.5 GGUF modules in one external directory. The current launcher expects the main model path passed with -m; the vision, audio, TTS, and Token2Wav files must follow the layout required by your checked-out llama.cpp-omni revision.

MiniCPM-o-4_5-gguf/
├── MiniCPM-o-4_5-Q4_K_M.gguf
├── vision/
├── audio/
├── tts/
└── token2wav-gguf/

Follow the upstream llama.cpp-omni prerequisites for exact filenames and downloads.

4. Register your ESP32 glasses

Create the ignored local device registry:

Copy-Item examples/configs/devices.example.json runtime/openglass_omni/devices.json

Edit runtime/openglass_omni/devices.json:

{
  "devices": [
    {
      "name": "My-Glasses",
      "esp32_host": "YOUR_ESP32_IP",
      "esp32_port": 80,
      "rotate": 0
    }
  ]
}
  • name is the ID shown in the panel dropdown.
  • esp32_host is the DHCP address printed by the ESP32 serial monitor after boot.
  • rotate is clockwise camera rotation: 0, 90, 180, or 270.
  • The local registry is ignored by Git. Never commit private device addresses.

5. Configure the current launcher

At present, these are the effective settings:

What to configure Effective location now Value
MiniCPM-o-Demo checkout panel.py CONFIG["minicpm_demo_dir"] Absolute path containing upstream worker.py and gateway.py
llama-omni-server binary panel.py CONFIG["procs"]["llama"] Compiled executable under llama.cpp-omni/build
Main GGUF model Same llama command after -m Absolute path to the main MiniCPM-o 4.5 GGUF
Glasses name/IP/rotation runtime/openglass_omni/devices.json One entry per ESP32 glasses prototype
Prompt presets panel.py CONFIG["presets"] Interaction prompts shown by the current panel

runtime.example.json and prompts.json document the intended local configuration boundary, but the current panel does not consume either file. Copying the runtime example to runtime.local.json does not yet replace the hardcoded panel paths or prompt presets. This is a known integration issue, not a user configuration mistake.

6. Configure and flash Wi-Fi firmware

Open CameraWebServer_PDM_Audio/CameraWebServer_PDM_Audio.ino, set YOUR_WIFI_NAME and YOUR_WIFI_PASSWORD, select the correct ESP32-S3 board, and upload the firmware. Open Serial Monitor at 115200 baud and copy the assigned IP into your local devices.json.

The tracked firmware currently contains placeholders. examples/configs/esp32_wifi.example.h is documentation-only and is not yet included by the firmware; a later code change will move credentials into an ignored local header.

7. Launch

Activate the same Python environment used for MiniCPM-o-Demo, then run from the OpenGlass repository root:

python glasses_panel.py

Select ESP32 Glasses, choose the device name, and click Start. The current panel attempts to start:

llama-omni-server :22500
        -> worker :22400
        -> gateway :8006
        -> ESP32 bridge / local view :8080

The chain is ready only when all four process indicators are green and the first-person view is updating. A successful UI launch alone does not prove the model, audio, image, and response path is complete.

Panel lifecycle

  • Stop stops only the active device bridge and keeps the shared backend stages available.
  • Restart restarts the active bridge with the selected prompt.
  • Stop All asks the bridge to finish session recording, then stops gateway, worker, and backend in reverse order.
  • Closing the panel normally calls synchronous cleanup before the panel process exits.
  • Force-killing the panel, closing the terminal abruptly, or using processes started outside the current panel may leave services running. Check listening ports before restarting.
Get-NetTCPConnection -State Listen -ErrorAction SilentlyContinue |
  Where-Object LocalPort -in 22500,22400,8006,8080,18080

Hardware

The hardware track publishes the physical design separately from any model backend:

Available artifacts include an editable STEP file, a 3MF print plate, a sanitized BOM, and approved project images. STL exports, a public wiring diagram, pin map, soldering guide, complete validation results, and the assembly video are not part of the current public release.

Repository Map

OpenGlass/
├── glasses_panel.py                 # Root entry point for the experimental panel
├── runtime/openglass_omni/          # Panel, ESP32 bridge, recording and replay
├── CameraWebServer_PDM_Audio/       # ESP32-S3 camera + PDM microphone firmware
├── eval_benchmark/                  # ACL/Core evaluation and latency scripts
├── hardware/                        # CAD, BOM, images and bilingual build report
├── papers/acl2026.md                # ACL/Core publication page
├── docs/                            # Architecture, quickstart, safety and roadmap
├── examples/configs/                # Sanitized local-configuration templates
└── assets/                          # Prototype photos, figures and logos

Upstream model projects and model weights remain external dependencies and are not vendored here.

Known Limitations

  • A clean-machine end-to-end Omni run has not yet been verified from the current public tree.
  • The current panel still contains machine-specific runtime paths instead of consuming runtime.local.json.
  • Prompt presets are still embedded in panel.py; the standalone prompts.json file is not yet connected.
  • ESP32 Wi-Fi credentials still require editing the tracked .ino; the local header template is not wired in yet.
  • Normal panel close performs cleanup, but abnormal termination can leave child or externally started processes running.
  • The public repository does not currently contain the Rokid bridge source or APK.
  • RayNeo support is planned but not implemented.
  • Long-running Omni sessions, robust barge-in, session restart, and skill injection are active experiments, not solved platform features.
  • Hardware battery life, comfort, charging/debug behavior, autofocus behavior, wiring, and final print settings require further verification.
  • Model responses can be wrong or delayed. Do not rely on the system for certified navigation or safety-critical decisions.

See the roadmap and release checklist for the remaining work.

Publication

The ACL 2026 paper documents the OpenGlass-Core research snapshot. It does not define the full scope of OpenSQZ Glass or the later MiniCPM-o runtime and hardware tracks.

  • Title: OpenGlass: A Sensing-Computing Split Architecture for Local MLLM-Driven Real-Time Visual Assistance
  • Authors: Mengzhang Li and Yuan Yao
  • Venue: ACL 2026 System Demonstrations, pages 829-839

[ACL Anthology] [PDF] [DOI]

@inproceedings{li2026openglass,
  title={OpenGlass: A Sensing-Computing Split Architecture for Local MLLM-Driven Real-Time Visual Assistance},
  author={Li, Mengzhang and Yao, Yuan},
  booktitle={Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)},
  pages={829--839},
  year={2026}
}

License and Contributing

OpenSQZ Glass is distributed under the Apache License 2.0.

Issues and focused pull requests are welcome at the OpenSQZ/OpenGlass repository. Before contributing:

  • Do not commit Wi-Fi credentials, private IP registries, model weights, personal data, raw private sessions, or absolute local paths.
  • Keep MiniCPM-o-Demo and llama.cpp-omni as independent upstream checkouts rather than copied source trees.
  • Mark experimental device/backend combinations honestly and avoid production-readiness or certified-safety claims.
  • Document the exact upstream branch and commit used for runtime changes.

About

An open research platform for wearable sensing, nearby-device local multimodal inference, and local-first visual assistance.

Topics

ai-glasseshardwaremultimodelomnipersonassmartglasseswearablewearable-devices

Resources

Readme

Code of conduct

Code of conduct

Contributing

Contributing

Security policy

Security policy
Activity
Custom properties

Stars

17 stars

Watchers

1 watching

Forks

2 forks
Report repository

Releases

Packages

Used by

Contributors

Languages