Skip to content

Correct the keccak gate and prove bound necessity - #950

Draft
jotabulacios wants to merge 11 commits into
docs-keccak-formal-verification-baselinefrom
fix/keccak-gate-bound-necessity
Draft

Correct the keccak gate and prove bound necessity#950
jotabulacios wants to merge 11 commits into
docs-keccak-formal-verification-baselinefrom
fix/keccak-gate-bound-necessity

Conversation

@jotabulacios

@jotabulacios jotabulacios commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Motivation

The gate's line citations are its only audit trail into the Rust. The README states
that faithfulness of the model to the Rust is a human obligation, and the citations
are how someone discharges it. All of them are wrong. rs:539-588 ("theta: Cxz XOR
chain") lands on the KeccakRc sender, rs:796-870 ("chi: AND then XOR") lands on
Iota, and rs:593-631/rs:723-766 name BusInteraction::sender(BusId::Hwsl, …)
blocks that #889 deleted outright; the file now contains zero. test_ref.py cites
execution.rs:646-680 for KECCAK_RC/KECCAK_RHO, which sit at 782 and 810. They
were moved by #876, an unrelated hint-ecall PR that grew the file by 136 lines, so
the cited range is now ArithOp::Div and the citation was stale three days before
this branch opened.

The README also declares a scope gap as its first follow-up: QF-BV cannot test
whether the AreBytes/IS_BIT bounds are sufficient mod p. It cannot be closed
in the same model. The bounds are carried as the width of the bitvectors, and mod
2^n the factor 2^16 is a zero divisor, so even a widened model would report the
decomposition pinned. Mod the Goldilocks prime 2^16 is invertible.

Separately, the AIR's count tests (1031 interactions, 1480 columns, 140 constraints)
catch anything that adds or removes one of those, but not a rewiring that keeps them.
Changing the first BusValue of the ρ ARE_BYTES pair from cols::rot_left to
cols::rot_right leaves all three counts untouched, is satisfied by every honest
trace, and makes ρ forgeable.

Description

  • Cites each modeled equation by the construct it names — the
    // --- <Group>: … --- banner title, or the cols:: / KeccakRndConstraints
    symbol — instead of a line number. Names survive the churn that broke these from
    two unrelated directions. No executable change in those three files: their ASTs are
    identical modulo docstrings, and the board is unchanged.

  • Drops the README section that catalogued where the stale numbers really pointed,
    since with the citations fixed it no longer describes anything.

  • Adds field_model.py, the companion integer-mod-p model of the inline θ/ρ
    identities, with a switch per range check. in · 2^rnc = right · 2^16 + left pins
    nothing on its own over a field, since for any left there is exactly one right.
    What the range checks do is rule out the deviation (L, R) → (L − 2^16·d, R + d),
    and they do not do it equally:

    dropped θ (Cxz_left/Cxz_right) ρ (rot_left/rot_right)
    the left range check sound, implied forgeable
    the right range check sound, implied sound, implied
    both forgeable forgeable, output entirely free

    left and right enter with weights 1 and 2^16. Bounding left kills the
    deviation outright; bounding right only narrows it. In θ the residual window is
    closed by the parity of left, the shift being by one, and by the carry being a
    single bit. In ρ, right is a halfword and d = ±1 fits exactly at saturation,
    which theta = 0xFFFFFFFFFFFFFFFF reaches on all 25 lanes.

  • How a cell is decided. Dropping a check does not free its column: the ByteAlu
    operand that reads it still confines it, because the BITWISE table holds byte rows
    only, so checked_byte + this ∈ [0,255] forces this ∈ [−255,255]. Only with that
    window is the deviation an integer at all — over the field an unbounded right
    admits a full-size solution for every left, and no argument about small d would
    mean anything. operand_summand_window derives the window,
    difference_form_is_exact checks the magnitude step (every term under 2^33 ≪ p,
    so the field identity is the integer identity and Euclidean division is unique),
    and surviving_deviation then sweeps all 2^16 input halfwords against one
    interval per column, returning either "pinned" or the first survivor. It also
    asserts the honest pair lies inside the modelled intervals, the failure that would
    make a "pinned" verdict meaningless. The "both dropped" row is the one
    configuration with no per-column window — the operand bounds only the sum of two
    unchecked columns — so it is decided by an explicit witness instead.

  • Adds combinatorics.py, the premises the sweeps rest on, none of which had been
    checked: π is a bijection on the lanes, all 400 rot_left/rot_right byte columns
    are read exactly once by a pi operand, the pi offsets are even, and the four θ
    carries are a permutation of the four rotated_C low bytes. Read-once is the
    load-bearing one — a column read twice would need the intersection of two windows.
    It is exposed as premises() and imported by both necessity scripts, so it is not
    a file you are asked to remember to run.

  • Adds witness_fullchip.py, which builds the ρ forgery as a whole KECCAK_RND row
    from a reachable message state instead of a lane in isolation: all 140 constraints
    and every ByteAlu operand satisfied, output lanes differing from FIPS-202. Honest
    and forged rows come from one builder and the honest row is required to be exactly
    the reference, because "the output differs" is also what a bug in the script itself
    produces. Every saturated lane is forged in turn — 11 of them, 10 with a non-zero
    rotation — and the wrong output lanes must be a subset of the lanes that can move.

  • Replaces the README's "both are load-bearing" with the table above, states that the
    24/24 UNSAT verdict is conditional on the range checks existing, and records what
    the two implied halves are worth, since they are not the same kind of saving: ρ's is
    100 AreBytes sends, θ's is 20 degree-3 polynomial constraints — the reason
    KeccakRndConstraints declares max_degree() = 3. Neither is proposed here as an
    optimization; the result bounds the ceiling.

  • Adds make verify-keccak and a workflow that runs it on every PR touching
    formal_verification/, on any base branch. Nothing in the repo ran anything under
    that directory before: the solver-free half takes about three seconds, and leaving
    it manual meant the only automated guard was a digest test whose documented remedy
    is satisfied by pasting a constant. The QF-BV gate stays manual — z3 bindings, about
    three minutes — as the README documents. It runs on this PR as
    Keccak round gate (solver-free), green in 6 s. The trigger filters by path without
    restricting the base branch, which is why it runs on a stacked PR at all —
    Spec structure test, which restricts to main, does not run here.

  • Adds test_keccak_rnd_air_structure_is_pinned, which digests the bus wiring (ids,
    multiplicities, column indices, linear coefficients) and the captured constraint IR
    (ops, dimensions, field constants, roots). FNV-1a is written out because std's
    hasher is not stable across toolchains. When it fails the wiring changed, and the
    directory has to be re-run before the digests are updated.

#889 replaced the HWSL lookups with these identities and dropped 120 sends per row.
Under the lookup, left and right were pinned individually and dropping either
check was harmless; under the identity, dropping rot_left's is forgeable. The 100
saved ρ sends were paid for with a range check that changed status, and nothing
outside this directory records it.

Tests

  • make verify-keccak is green: test_ref.py, test_dataflow.py,
    combinatorics.py, necessity_theta.py, necessity_rho.py and
    witness_fullchip.py, about three seconds. The sound results need no solver — they
    are exact integer-bounding arguments over complete enumerations of all 2^16 input
    halfwords.
  • The existing board's verdicts are unchanged: z3_parallel.py (24/24 UNSAT, 5/5
    negative controls SAT, positive control PASS) and tamper_test.py (clean UNSAT,
    4/4 tampers SAT). test_ref.py, model_dataflow.py and z3_verify.py are touched
    only in their citations, and their ASTs are identical modulo docstrings.
  • Each new check was confirmed to fail when it should. Widening the left interval
    to a halfword and a bit, or dropping rot_left's check, produce survivors in the
    sweep. Making Dxz read column (x+1) instead of (x+4), or dropping χ's NOT,
    fail the full-chip positive control — both of those used to pass. Ignoring the
    target in configuration D fails its target check, which is how the off-by-one in
    that recurrence was found: it indexed c[(j+1)%4] while the closed form for r0
    above it solves for c[j], so it hit the intended pi halfwords on 0 of 25 lanes.
  • test_keccak_rnd_air_structure_is_pinned was checked against the tamper it exists
    for. With cols::rot_leftcols::rot_right applied to the ρ ARE_BYTES pair,
    test_keccak_bus_interaction_counts, test_keccak_column_counts,
    test_keccak_constraint_counts, test_pi_virtual_matches_rotate and
    test_keccak_core_round_state_consistency all pass, and only the digest fails.
  • cargo fmt --check and cargo clippy --all-targets -D warnings clean.

Configuration D of the rho necessity analysis claims the lane's output is
completely free: with neither range check present, the cyclic system in
`right` solves for an arbitrary target. The check backing that claim only
asked whether the shift identity holds for the constructed pair - and `L[j]`
is derived FROM the identity, so it held by construction and could not fail.
Nothing verified the target was reached.

With the target now compared, it turns out it was not. The recurrence indexed
`c[(j+1)%4]` while the closed form for `r0` directly above it solves
`R[j-1] - 2**16*R[j] = c[j]`, so the two disagreed by one and the
construction hit its intended pi halfwords on 0 of 25 lanes. Indexed `c[j]`,
which is what `r0` was derived for, the lanes hit their target 25 of 25.

The verdict for configuration D is unchanged - the system is solvable, the
output is free - but the exhibited construction now demonstrates it.
The board's three "sound - implied" cells were backed by
`check(2**16 > MASK16)`: a comparison of two constants, true whatever the
chip does. And the argument it stood for does not cover the two
configurations where the column left unchecked is `right`, because there the
deviation is not a small integer at all - over the field, any `left` in
[0, 2**16) admits `right = (in*2**rnc - left)*inv(2**16)`, a full-size field
element.

What closes those configurations is the ByteAlu operand that reads the
unchecked column. The BITWISE table holds byte rows only, so
`checked_byte + this` in [0, 255] confines `this` to [-255, 255]; every term
of the identity then stays under 2**33, the field identity IS the integer
identity, and Euclidean division is unique. That chain is now in the model:
`operand_summand_window` derives the window, `difference_form_is_exact`
checks the magnitude step the difference form had silently assumed, and
`surviving_deviation` sweeps all 2**16 input halfwords against one interval
per column. It replaces both the constant comparisons and the four
hand-picked inputs that stood in for theta's configuration B, and it asserts
the honest pair lies inside the modelled intervals - the failure that would
make a "pinned" verdict meaningless.

The windows rest on each column being read by exactly ONE operand byte, so
`combinatorics.py` grows the theta analogue of that premise: the four carries
are a permutation of the four rotated_C low bytes. It is exposed as
`premises()` and imported by both necessity scripts, rather than being a file
whose docstring asks you to run it first.

Verdicts are unchanged, but each is now the output of a sweep that fails when
its inputs do: widening the left interval to a halfword and a bit, or
dropping rot_left's check, both produce survivors.
`witness_fullchip.py` asserted that at least one output lane differs from the
reference - which is also what a bug in its own hand transcription of the
round produces. Both failure modes were reachable: making Dxz read column
(x+1) instead of (x+4), or dropping chi's NOT, left the script printing
FULL-CHIP WITNESS VERIFIED while reporting 12 wrong lanes instead of 2. That
is the fail-open direction discipline 1 of the README calls the only
dangerous one.

Honest and forged rows now come out of one `build_row(tamper=...)`, and the
honest row must be EXACTLY the reference - cross-checked against keccak_ref
and against model_dataflow's mirror, the one test_dataflow.py validates -
before any claim about the forged row is made. Both mutations above now fail
that control.

The forgery is also no longer demonstrated on a lane picked by list order,
which happened to be (0,0): the one lane with RHO = 0, no rotation at all.
All 11 saturated lanes are forged in turn, 10 of them with a non-zero
rotation, and the wrong output lanes must be a subset of the lanes that can
move - the single pi lane reading the forged source, itself read by three chi
lanes - which pins where the forgery leaks instead of only counting lanes.
Nothing in the repo ran anything under formal_verification/: no Makefile
target, no workflow. The gate's only automated guard was the Rust digest
test, and its documented remedy - re-run the directory, then update the
digests - is satisfied by pasting the new constant without running anything.
The daily LOC report meanwhile counts the directory as its own "formal
verification" section, so those lines are reported under that heading with
nothing executing them.

`make verify-keccak` runs the half that needs no solver: the FIPS-202
reference anchors, the concrete mirror, the combinatorial premises, both
necessity boards and the full-chip witness. About three seconds. A new
workflow runs it on every pull request that touches the directory, on any
base branch, so stacked PRs are covered too.

The QF-BV gate itself stays manual - it needs z3's Python bindings and about
three minutes - exactly as the directory README documents.
The README promoted the necessity table to the headline result without saying
how a cell is decided, which left the two "implied" rows reading as bare
assertions. It now names the mechanism - the operand window, the integrality
step, the complete sweep - and says which single configuration has no
per-column window and is therefore decided by a witness instead.

It also records what the two implied halves are worth, because they are not
the same kind of saving: rho's is 100 AreBytes sends, theta's is 20 degree-3
polynomial constraints, which are the reason this AIR declares max_degree 3.
Neither is proposed as an optimization here.

The file list and the run instructions follow the code: `combinatorics.py`
carries the theta premise and is imported rather than run by hand, the
necessity scripts sweep rather than sample, the witness has a positive
control, and `make verify-keccak` is the one command CI runs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant