notes_whisper.cpp

For AI4deafblin we use a tool to benchmark
(-1) install uv - curl -LsSf https://astral.sh/uv/install.sh | sh

Based on https://thomasthelliez.com/blog/run-whisper-cpp-with-cuda-on-jetson-orin-nano-super/. There are many mistakes. For example --gpu is not a correct option. Refer back to the github for whisper.cpp for actual run options.

Setup
(0) sudo apt update; sudo apt upgrade
sudo apt install cuda locate
(0.1) sudo apt install -y python3 python3-pip portaudio19-dev ffmpeg
(0.2) pip3 install --user sounddevice numpy requests 
(0.3) sudo apt install build-essential git libcurl4-openssl-dev libsdl2-dev

Whisper.cpp
(1) clone whisper.cpp by "git clone https://github.com/ggerganov/whisper.cpp.git"
(1.1) cd whisper.cpp
(1.2) Configure the build CUDA "cmake -B build -DCMAKE_BUILD_TYPE=Release -DCMAKE_BUILD_TYPE=Release"
(1.3) Compile for default CPU by "cmake --build build -- -j2"
The above instruction is incorrect. Use the following:
(1.4A) modify stream.cpp

./CMakeLists.txt:34:# sllu add Braille library
./examples/stream/stream.cpp:16:// sllu mod - starts here
./examples/stream/stream.cpp:22:// sllu mod - ends here
./examples/stream/stream.cpp:378:			// sllu - print English braille grade 1  :: starts here
./examples/stream/stream.cpp:396:			// sllu - print English braille grade 1  :: ends here

(1.4) compile for streaming & louis (braille)
cmake -B build -DWHISPER_SDL2=ON -DCMAKE_CXX_FLAGS="-DUSE_LIBLOUIS"
cmake -B build -DWHISPER_SDL2=ON -DCMAKE_EXE_LINKER_FLAGS="-llouis"
cmake --build build --config Release
It will create a binary program whisper-stream
use "whisper-stream -c 0" to run
-c <HW device>

cmake -B build -DGGML_CUDA=1 -DWHISPER_CUBLAS=0
make --build build -j --config Release
test run by "./build/bin/whisper-cli -m models/ggml-base.en.bin -f samples/jfk.wav
use watch -n 1 nvidia-smi to see if GPU is active (jetson does not support nvidia-smi since it is not PCIe based but integrated)
use jtop but first "sudo -H pip install jetson-stats" reboot

(1.4) Download a model if it has not been done 
(1.5) test run with CPU by "./build/bin/whisper-cli -m models/ggml-base.en.bin -f samples/jfk.wav -oved CPU"
(1.6) observe the output
(1.7) Compile (build) with CUDA support "cmake -B build -DGGML_CUDA=ON"
(1.8) cmake --build build --config Release
(1.9) run with GPU? ./build/bin/whisper-cli -m models/ggml-base.en.bin -f samples/jfk.wav -oved GPU

Whisper-server
(2) ./build/bin/whisper-server -m models/ggml-base.en.bin --gpu --port 8080 

Capture Voice Input & Detect Wakeup Word
Based on https://www.seeedstudio.com/blog/2025/07/21/building-an-ai-voice-assistant-on-nvidia-jetson-voice-activation-speech-to-text/?srsltid=AfmBOopHILwpT0QXEQaumD6245NPMLlHHFx0ywLkx6T2kCdh_eVU_a7Z
(3) sudo apt install nlohmann-json3-dev libcurl4-openssl-dev mpg123
(3.1) git clone https://github.com/jjjadand/record-activate.git
(3.2) sudo apt install pavucontrol
(3.3) test : pactl list short sources
(3.3.1) test: arecord -l

Camera App
Based on https://developer.nvidia.com/embedded/learn/tutorials/first-picture-csi-usb-camera#CameraGuide-SupportedCameras
(4) nvgstcapture-1.0 --camsrc=0 --cap-dev-node=0 -m 2 --file-name=/tmp/test
-m 2 (video)
use command 1 and 0 to start and stop.
use command q to quit
-file-name is the directory and file_prefix. It will creeate multiple captures with unique ID each time

Audio recording
(5) arecord -D hw:2,0 -f S16_LE -r 44100 -d 5 -t wav test_audio2.wav
-d will record for 5 secods
-t is type
the file name for the recording is the last paramenter

Library that converts text to braille
(6) sudo apt install liblouis-dev
(6.1) test it with simple C program - braille_char that prints "Hello World" in Braille (Letter-for_Letter)
(6.1.1) compile it with "gcc braille_char.c -o braille_char -llouis"
(6.2) modify src code in whisper.cpp/example/stream to output Braille
(6.3) modify CMakelist.txt to add liblouis
    target_link_libraries(${TARGET} PRIVATE common common-sdl louis whisper ${CMAKE_THREAD_LIBS_INIT})

------------------------------------------------
How to build key control whisper

run this at ~/whisper.cpp/examples/stream

# 1. Clear out old cache files to map the new executable path 

rm -rf build

RUN this at ~/whisper.cpp

# 2. Configure with your exact Braille macro and linker properties
cmake -B build -DWHISPER_SDL2=ON -DCMAKE_CXX_FLAGS='-DUSE_LIBLOUIS' -DCMAKE_EXE_LINKER_FLAGS='-llouis'

# 3. Compile the new standalone binary target
cmake --build build --config Release

cmake --build build --config Release