Follow-on to #16. The open kernels are close enough on the 35B
(Qwen3.6-35B-A3B) now that the remaining gap is two specific pieces of work
rather than a general slowness. Here is where it stands.
Measured on a quiet machine, all 40 layers, our open-kernel path against stock
FLM 1.0.2:
|
open kernels |
closed engine |
| reading a 1,000-token prompt |
23 ms per token |
15 |
| reading a 2,600-token prompt |
25 |
8.5 |
| writing tokens, with 1,000 of context |
6.8 per second |
12 |
Reading a prompt used to be three to five times slower than the closed engine.
The expert kernel that just landed - it puts eight tokens through each expert
in one go instead of one - brought that to 1.6x and 2.9x. Writing has not been
worked on at all yet.
Reading: the biggest piece left is still on the CPU. Prompts are read 256
tokens at a time. Everything in that work is on the NPU except one step -
attention, where each token is weighed against all the tokens before it - which
still runs on the processor. It is also the only part that grows as the prompt
gets longer: at 2,600 tokens it is 2.5 seconds of the 3.7 the CPU spends on
each batch of 256. The arithmetic itself is small enough that the NPU should
finish it in well under a tenth of a second. It costs twice over, because
pushing that much data through the processor also slows down the NPU work that
comes after it. With it moved I would expect reading to settle near 9.4 ms per
token and stop growing with prompt length, against the closed engine's 8.5.
Writing: one step is running at a third of the speed everything else gets.
Producing a token takes 128 ms and almost all of it is the NPU; there is no
meaningful CPU time to remove. Of that, 75 ms is a single step repeated across
30 of the 40 layers - the first half of each of the model's linear-attention
layers, which reads that layer's weights and updates its running state. It
reads about 25 MB and takes 2.1 ms, which is 12 GB per second. The same
hardware reads at 26 to 33 GB per second everywhere else in this model, so this
one step is the outlier, and it is the step itself rather than anything around
it: it costs the same run in isolation as it does in a real pass, and the same
at one token of context as at a thousand. If it read at the rate the rest do,
writing would be about 12.8 tokens a second, which is past the closed engine.
I have a guess at the cause - it hands its results back in very small pieces,
where the reading side works in pieces forty times larger - but that is
reasoning on paper. The first version of the expert kernel looked like the same
kind of problem and turned out to be limited by something else entirely, so the
first job is to measure it rather than act on the guess.
Two caveats on the table: the closed-engine numbers were measured on 11
September and not since, and the 2,600-token prompt is the same length as the
one the closed engine read but not the same text.
Follow-on to #16. The open kernels are close enough on the 35B
(Qwen3.6-35B-A3B) now that the remaining gap is two specific pieces of work
rather than a general slowness. Here is where it stands.
Measured on a quiet machine, all 40 layers, our open-kernel path against stock
FLM 1.0.2:
Reading a prompt used to be three to five times slower than the closed engine.
The expert kernel that just landed - it puts eight tokens through each expert
in one go instead of one - brought that to 1.6x and 2.9x. Writing has not been
worked on at all yet.
Reading: the biggest piece left is still on the CPU. Prompts are read 256
tokens at a time. Everything in that work is on the NPU except one step -
attention, where each token is weighed against all the tokens before it - which
still runs on the processor. It is also the only part that grows as the prompt
gets longer: at 2,600 tokens it is 2.5 seconds of the 3.7 the CPU spends on
each batch of 256. The arithmetic itself is small enough that the NPU should
finish it in well under a tenth of a second. It costs twice over, because
pushing that much data through the processor also slows down the NPU work that
comes after it. With it moved I would expect reading to settle near 9.4 ms per
token and stop growing with prompt length, against the closed engine's 8.5.
Writing: one step is running at a third of the speed everything else gets.
Producing a token takes 128 ms and almost all of it is the NPU; there is no
meaningful CPU time to remove. Of that, 75 ms is a single step repeated across
30 of the 40 layers - the first half of each of the model's linear-attention
layers, which reads that layer's weights and updates its running state. It
reads about 25 MB and takes 2.1 ms, which is 12 GB per second. The same
hardware reads at 26 to 33 GB per second everywhere else in this model, so this
one step is the outlier, and it is the step itself rather than anything around
it: it costs the same run in isolation as it does in a real pass, and the same
at one token of context as at a thousand. If it read at the rate the rest do,
writing would be about 12.8 tokens a second, which is past the closed engine.
I have a guess at the cause - it hands its results back in very small pieces,
where the reading side works in pieces forty times larger - but that is
reasoning on paper. The first version of the expert kernel looked like the same
kind of problem and turned out to be limited by something else entirely, so the
first job is to measure it rather than act on the guess.
Two caveats on the table: the closed-engine numbers were measured on 11
September and not since, and the 2,600-token prompt is the same length as the
one the closed engine read but not the same text.