Ridgeline

A local research assistant built on a hardware aware inference engine, targeting Intel Lunar Lake. All inference happens on device.

Status: complete through Phase 14 of 14. The engine measures the machine, predicts what it will do with a workload, places work accordingly and says how much to trust the prediction. The application ingests documents, embeds them on the NPU, answers questions with checked citations, serves it over HTTP and opens in a browser. Fourteen phases of measurements are in docs/findings/, and the ones that refuted the design are in there too. See docs/state.md for what exists today and what is broken.

What it is

Two layers, deliberately separated.

The engine (src/ridgeline/) places inference work across CPU, integrated GPU and NPU from measured hardware characteristics rather than guesswork. It measures the machine, derives a roofline model, and decides placement from it.

The application (src/reader/) is a research assistant. Documents go in, questions come out with citations, nothing leaves the machine.

The application depends on interfaces in ridgeline/ports.py, never on an engine implementation, so the engine stays portable and both halves are testable in isolation. A fake engine backs the whole application test suite, which is how 838 tests run in CI on a Linux box with no Intel device in it.

What the measurements refuted

This section is first on purpose. Every phase of this project ended in a measurement, and the interesting half of those measurements contradicted something the project had assumed. A README that led with the wins and buried these would be describing a different piece of software.

More context makes answers worse, and prefill being cheap does not save it. The premise of the design is that reading is nearly free, so the machine should read whole documents rather than fragments. Reading is indeed nearly free. It does not help.

Documents supplied	Expected document reached the model	Answer cited the right one	Never answered at all	Median
1	0.676	0.432	0.07	8.8 s
2	0.838	0.446	0.01	14.8 s
4	0.865	0.216	0.14	35.5 s
8	1.000	0.108	0.11	53.1 s
16	1.000	0.027	0.41	64.7 s
30	1.000	0.054	0.32	71.3 s

By eight documents an expected document is in front of the model on every single question and the answer cites the right one eight times less often. Fourteen times the prompt costs six times the wall clock. The binding constraint is the model's attention and its output budget, not the machine's bandwidth. The default is two documents. See the findings.

Speculative decoding loses on this machine by a factor of four, and the roofline said it would win. Draft on the CPU, target verifying on the integrated GPU: proposals are accepted 42% to 49% of the time up to k=3, which is a healthy acceptance rate, and every draft length is slower than not doing it at all.

k	decode tok/s	accepted	against the control	what a proposal costs
control	47.41		1.00x
1	12.63	42.1%	0.27x	1.40x a verification
3	8.44	49.0%	0.18x	1.55x a verification
8	3.93	27.0%	0.08x	1.50x a verification

The draft model costs more per proposal than the target costs per verification, so it has nothing to sell at any acceptance rate. The prediction that said otherwise was wrong by a factor of five, and it is the first time a bad prediction in this project produced a bad design decision rather than a bad number. Verification itself is output lossless, which was checked. See the findings.

The cost model needs a second term and the second term is not there. Effective decode bandwidth is not a property of a device: on the integrated GPU it rises from 47 GB/s at Qwen3-0.6B to 75 at Qwen2.5-1.5B and falls back to 52 at Qwen3-4B. A fixed cost per token explains the rise and cannot explain the fall. Fitted to all four models the intercept goes negative and the guards refuse it. Held out one model at a time, the two term fit scores 94.6% mean absolute error against 49.8% for the single figure it was meant to replace.

So predictions come from one effective bandwidth per engine whose held out error is 49.8%, and every prediction says which models it rested on and how far outside them it reached. It is good enough to rank engines and not good enough to quote. See ADR 0024 and the findings.

The one change that makes the system four times faster costs quality, so it is off by default. Generation is 98.5% of a request and the model's reasoning is 87.3% of the generation. Telling the model not to think aloud takes the median question from 13.64 s to 3.33 s and 401 generated tokens to 57. It also raises citation compliance from 0.486 to 0.541, which is the opposite of what was predicted. And it costs 0.176 of phrase recall and one question of correct citation, which is a regression by the definition this project wrote in Phase 9 precisely so that Phase 13 could not argue its way past it. It ships as --suppress-reasoning and it is not the default. See ADR 0028.

Why it is built this way

Every design decision follows from measurement on the target machine, an Intel Core Ultra 5 238V with an Arc 130V integrated GPU and an AI Boost NPU.

Observation	Measured	Design consequence
Prefill throughput (integrated GPU)	7,638 tok/s	Reading is cheap. Favour large context.
Decode throughput	~55 tok/s	Generation is expensive. Keep answers short.
Host memory bandwidth	82.8 GB/s of 136.5 theoretical	Bytes read per token sets speed, not parameter count.
Bandwidth achieved during decode	46.7 to 75.0 GB/s on the integrated GPU, by model	Predict from what the engine achieves, not from one figure per device.
Decode arithmetic intensity	271x below the compute roof	Compute is nearly free. Spend it to save bandwidth.
Two engines on one workload	Slower than one	Give engines different jobs, never the same job.
Python per token loop	33% slower than native	Never iterate per token in Python.
Generation, as a share of a request	98.5%	Nothing on the host side is worth optimising.

The raw measurements are in docs/findings/ and the scripts from the prior investigation that produced the first of them are in bench/.

Requirements

• Windows on Intel Lunar Lake, for the full engine
• Python 3.12
• OpenVINO 2026.3 and OpenVINO GenAI 2026.3, for anything touching a device

The engine core and its unit suite install and run without any of the above, which is how CI works on Linux.

Install

pip install "ridgeline[reader,engine] @ git+https://github.com/Protonicwave/ridgeline"

Not on PyPI. It is one machine's worth of evidence and a version number of 0.1.0, and claiming a package name is a promise this cannot keep yet.

The extras are separate because they fail differently. The base install is numpy and nothing else, so the engine's logic and its tests run anywhere. [reader] adds document parsing and the HTTP server, 24 packages. [engine] adds OpenVINO, which needs an Intel device to be useful. Asking for a command whose extra is absent gets one line naming what to install, not an import traceback.

From a checkout:

git clone https://github.com/Protonicwave/ridgeline
cd ridgeline
python -m venv .venv
.venv/Scripts/python.exe -m pip install -e ".[dev,reader,engine]"

Run the checks:

.venv/Scripts/python.exe -m ruff check .
.venv/Scripts/python.exe -m mypy
.venv/Scripts/python.exe -m pytest -m "not integration"   # 838 tests, no hardware, 10 s
.venv/Scripts/python.exe -m pytest -m integration         # 55 tests, requires an Intel device

The engine

Describe the machine:

ridgeline probe                    # devices, adapters, supply state
ridgeline probe --idle-seconds 10  # and the idle floor

The report is written to docs/findings/results/machine.json. It names every OpenVINO device with its reported peak compute and memory, the performance counter adapters discovered on the machine, and which adapter serves which device. On a machine with no Intel device the command says what is missing rather than failing obscurely.

Model the machine:

ridgeline roofline --theoretical-gbs 136.5              # measure and predict
ridgeline roofline --machine docs/findings/results/machine.json

This measures memory bandwidth with a threaded STREAM benchmark, calibrates it against decode rates measured on the machine, and reports the ridge point and achievable throughput per engine. With --machine it rebuilds the roofline from a saved report, which works on a machine with no Intel device present.

Two bandwidth figures are kept apart and never blended, because predicting the accelerators from the host figure was 65% to 114% optimistic. See ADR 0005.

List the models and what the roofline expects of each pairing, before loading any of them:

ridgeline models              # geometry, cache cost, predictions
ridgeline models --verify     # and hash each weight blob

Generate, benchmark, and let placement choose an engine:

ridgeline generate --model qwen3-0.6b-int4-ov --prompt "..."
ridgeline generate --device NPU --prompt "..."                    # or name one
ridgeline benchmark --prompt-tokens 128,512,1024 --max-new 32     # predict, measure, record
ridgeline speculative                                             # sweep the draft length

Each pairing is predicted before the model is loaded, then measured, and one record per model is written to docs/findings/results/. Those records are what the calibration reads, so the loop between measurement and prediction is closed and the ordering is enforced by the code rather than by discipline.

Select a configuration for a machine, without compiling anything:

ridgeline autotune              # model, engine and context budget from measurement alone
ridgeline autotune --validate   # score every candidate predictor, held out by model

Every pairing is priced, including the ones that cannot run, and each carries its reason. An engine with no calibration is refused rather than ranked against one that has.

Model weights are never committed. They are fetched on demand and the registry records identifiers and checksums.

The application

reader ingest ~/Documents/papers --corpus corpus.db
reader index --corpus corpus.db                        # embed, in batches
reader search "what limits decode?" --corpus corpus.db
reader ask "what limits decode?" --corpus corpus.db
reader status --corpus corpus.db

PDF, Markdown and plain text are parsed with their page and section structure kept, split on real structural boundaries, and stored with the offsets that map any span back to where it came from. Re-running is a no-op for documents that have not changed, and a document is addressed by its content and by the pipeline that processed it, so changing a parser or a chunk window reprocesses the corpus while a preserved modification time cannot hide an edit.

Embedding runs on whichever engine placement chooses, which on this machine is the NPU, leaving the integrated GPU free to serve queries. Over the fixed question set, fusing a dense scan with a term index beats either alone:

Retriever	hit@1	MRR	recall@3	Median latency
Fused	0.80	0.873	0.93	30 ms
Dense only	0.67	0.807	0.90	24 ms
Lexical only	0.60	0.761	0.87	2 ms

Asking a question fills the generator's context budget with whole documents, streams a short answer, and checks every citation and quotation in it against what was actually supplied. A marker naming a document that was never in the context is reported as fabricated; a quoted phrase the context does not contain is reported as not found; an answer that cites nothing is reported as unchecked rather than shown as though it had been verified. What is not checked is whether the sentence follows from the document, which would be entailment, and ADR 0011 says so rather than implying otherwise.

Or serve it:

reader serve --corpus corpus.db     # loopback, port 8000, interface at /
curl -N "http://127.0.0.1:8000/query/stream?question=what+limits+decode"

Both models are compiled at startup and held for the life of the process, because compiling one costs 4.3 s warm and up to four minutes cold, and one request uses them at a time. The HTTP layer itself is free, adding 4.1 ms to a request that takes eight seconds. Answers stream as server sent events with the model's reasoning on a separate event from the answer, because on a real question the first reasoning token arrives at 3.3 s and the first token of the answer only at 8.9 s of a 10.1 s request.

The interface is documented in docs/api.md and published as docs/openapi.json, which is generated from the application and checked by a test, so it cannot drift from what the server serves.

The browser interface

http://127.0.0.1:8000/ is the same process. Plain ES modules and hand written CSS, no build step, nothing fetched from another host, served from the same origin so no cross origin policy exists to get wrong. See ADR 0016.

Measured in Phase 14, in Chrome, against the server on loopback:

	Cold	Warm
Interactive	892 ms	8 ms
First contentful paint	1,088 ms	28 ms
Engines named in the masthead	971 ms	22 ms
Transferred	96.9 KB over 13 requests	21.1 KB

The health poll costs 2.0 ms a call at four second intervals, so an eight hour session is 7,200 calls and 1.26 MB. The document viewer renders 44,485 characters in 22 ms and has no virtualisation, so it is linear in document size and reaches 8.1 s at 16 MB, which is far larger than anything this corpus holds.

Three things it shows that a chat window does not.

The model's reasoning, as reasoning. It fills a block while the model thinks and collapses at the first token of the answer. On a measured question the first reasoning token arrived at 3.75 s and the first token of the answer at 14.13 s of a 15.28 s request, 424 reasoning events against 48. Showing it as the answer would be a lie and hiding it would be nine seconds of blank screen.

Where the request sat on the roofline. Bytes read per token against tokens per second, log on both axes, each calibrated engine's roof as a line, and this request as a point beneath the roof of the engine that served it. The predicted rate is drawn beside the measured one, because the gap is the number the project is trying to close. An engine with no calibration gets no roof and no prediction rather than a guess.

What every claim rests on. Each [n] opens the document the answer was checked against, with any verified quotation highlighted in place. A marker naming a document nobody supplied is drawn as a fabrication and is not clickable.

Evaluation

Quality comes before speed, which is why the harness was built before any of the optimisation phases.

reader pin --ref 30c2c7c --into .eval-corpus    # the corpus, at a commit
reader evaluate --corpus eval.db                # quality and cost
reader compare BEFORE.json AFTER.json           # what moved, and whether it may be compared
reader study --corpus eval.db                   # sweep one condition on purpose
reader profile --corpus eval.db                 # where the milliseconds went

Thirty seven questions over this repository's own documentation, pinned to a commit so that writing documentation cannot change the corpus underneath a measurement. Retrieval scores hit@1 0.68, hit@10 0.97 and MRR 0.771. Nothing was fabricated in 37 answers. Citation compliance is 0.49, which is the largest quality gap the harness measures and is reported as such.

Three properties make the numbers worth something. Every prompt carries a distinct opening and each run spends two extra generations proving that the opening actually defeated a live cache, because a control that cannot fail is not a control. Host bandwidth is measured either side of the question set and a run on a machine that has drifted refuses itself, which it has done more than once. And compare refuses two runs whose conditions differ, except for the one intervention under test, which it reports at the top instead.

Two runs of one commit produce byte identical answers on all 37 questions, so the harness has no noise of its own. What does move a result is the cache defeating salt each prompt carries, by up to four questions, and that is the floor a sweep is judged against.

Layout

src/ridgeline/    The engine
  hardware/       Device discovery and capability probing
  telemetry/      Power and per engine utilisation sampling
  roofline/       Bandwidth measurement, ridge model, cost fit, prediction
  runtime/        Registry, pipelines, budget, placement, compatibility
  autotune.py     Selecting a configuration from measurement alone
  ports.py        The interfaces the application depends on
src/reader/       The application
  ingest/         Folder watching, parsing, chunking, the corpus database
  index/ retrieve/ answer/
  server/         The HTTP API, and the second composition root
  web/            The browser interface: assets, no build step
  evaluate/       The question set, the conditions a run must prove, the profiler
tests/
  unit/           838 tests, no hardware required
  integration/    55 tests, requires OpenVINO and a device
bench/            Measurement harnesses from the prior investigation, unchanged
docs/             Architecture, state, 29 decisions, 12 findings

Documentation

• CLAUDE.md, conventions and orientation, read first
• docs/architecture.md, design and module boundaries
• docs/state.md, current status and every open issue
• docs/findings/, the measurement writeups
• docs/decisions/, architecture decision records
• docs/api.md, the HTTP interface and its error model
• docs/implementation-plan.md, the full build plan

Limitations

Stated plainly, and there are more of them than there are features.

The engine has run on exactly one machine. That is the central limitation and it is the claim the autotuner exists to support. Everything needed is in place: no hardware figure anywhere in src/ is a constant, the roofline can be rebuilt from a saved report on a machine with no Intel device, and a suite of tests drives the whole selection chain over synthetic machines with no NPU, no integrated GPU, one engine, or no calibration at all, checking that each degrades to a named constraint rather than a crash. What has never happened is a model loading on other silicon. See ADR 0029.

Prediction carries 49.8% held out mean absolute error. Good enough to rank engines, not good enough to quote as a number, and it says so on every prediction.

Citation compliance is 0.49. The model omits its citation marker on a third to a half of answers depending on how much context it is given. It is reported as unchecked rather than papered over, and nothing found so far raises it except suppressing the reasoning, which costs more elsewhere than it gains here.

A grounded verdict means the references are real, not that the answer is right. Asked for the capital of Peru the model answered The capital of Peru is Lima [1], cited a document about HTTP endpoints, and passed every check, exactly as designed. The badge says "References check out" for that reason.

Two of the five models cannot be used the way the plan assumed. The NPU refuses both Qwen3 exports in the VPU compiler, and the 30B mixture of experts gets no integrated GPU at all, since 15.6 GiB of weights against 17.7 GB exposed leaves nothing for the cache. Placement now declines an engine known to refuse a model rather than discovering it at load.

CPU decode of a model above roughly 2 GB is not reproducible on this machine. The same cell measured 9.46 and then 2.05 tok/s in two runs of one command. Those cells are flagged in the reports rather than dropped, and they are the worst cells in the error distribution, so the headline prediction error is pessimistic by an amount this machine cannot quantify.

A machine warmed by generation takes longer to recover than anyone guesses. Twenty five minutes of idle was not enough after two hours of sweeping. Quality is immune, because generation is deterministic and thermal state changes the clock rather than the text, but any cost comparison has to budget for it and nothing schedules that wait.

Watching is a walk rather than a watch. The server re-walks its folders on an interval; nothing subscribes to filesystem notifications, and an ingest run cannot be cancelled once started.

The server serves one request at a time and has no authentication. Both are deliberate: decode is bandwidth bound so concurrent requests would queue in the memory system anyway, and it binds to loopback, so an auth scheme would be a claim about a threat model nobody has stated. The queue is visible on /health.

Nothing bounds request size or rate. Same non decision as the line above, and cheap to add the moment there is a reason to.

The browser interface has no compiler checking it against the schema. That is the price of having no build step: a renamed response field reaches the page as an undefined rather than as a failed build.

Telemetry is Windows specific. Per engine utilisation comes from the PDH API and power from the battery interface, which only reports while unplugged, and has never been exercised under discharge because every run so far has been on mains.

Embedding and generation overlap, but not for free. Generation retains 69.7% of its rate while the NPU embeds, so indexing belongs at ingest and not alongside an answer.

Licence

Apache-2.0. See LICENSE.