Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions crypto/math-cuda/src/fri.rs
Original file line number Diff line number Diff line change
Expand Up @@ -205,6 +205,10 @@ impl FriCommitState {
num_leaves_u64,
&mut leaves_view,
)?,
DeviceHash::Rpo256 | DeviceHash::Rpx256 | DeviceHash::Poseidon => unimplemented!(
"{:?} device commit not yet ported (FRI layer ext3 leaves)",
self.hash
),
}
}
match self.hash {
Expand All @@ -220,6 +224,10 @@ impl FriCommitState {
&mut nodes_dev,
num_leaves,
)?,
DeviceHash::Rpo256 | DeviceHash::Rpx256 | DeviceHash::Poseidon => unimplemented!(
"{:?} device commit not yet ported (FRI layer inner tree levels)",
self.hash
),
}

// Update inv_twiddles for the next layer: `new[j] = old[2j]^2` for
Expand Down
15 changes: 15 additions & 0 deletions crypto/math-cuda/src/lde.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1054,6 +1054,9 @@ fn build_inner_tree_levels_for(
DeviceHash::Blake3 => {
crate::blake3::build_inner_tree_levels(stream, be, nodes_dev, leaves_len)
}
DeviceHash::Rpo256 | DeviceHash::Rpx256 | DeviceHash::Poseidon => {
unimplemented!("{hash:?} device commit not yet ported (inner tree levels)")
}
}
}

Expand Down Expand Up @@ -1138,6 +1141,9 @@ fn coset_lde_row_major_inner(
log_lde,
&mut leaves_view,
)?,
DeviceHash::Rpo256 | DeviceHash::Rpx256 | DeviceHash::Poseidon => {
unimplemented!("{hash:?} device commit not yet ported (row-major row-pair leaves)")
}
}
}
build_inner_tree_levels_for(hash, stream.as_ref(), be, &mut nodes_dev, num_leaves)?;
Expand Down Expand Up @@ -1333,6 +1339,9 @@ pub fn coset_lde_row_major_split_trees(
log_lde,
&mut leaves_view,
)?,
DeviceHash::Rpo256 | DeviceHash::Rpx256 | DeviceHash::Poseidon => unimplemented!(
"{hash:?} device commit not yet ported (row-major row-pair leaves, column range)"
),
}
}
build_inner_tree_levels_for(hash, stream.as_ref(), be, &mut nodes_dev, num_leaves)?;
Expand Down Expand Up @@ -2145,6 +2154,9 @@ fn coset_lde_batch_base_into_with_merkle_tree_inner(
lde_u64,
&mut leaves_view,
)?,
(DeviceHash::Rpo256 | DeviceHash::Rpx256 | DeviceHash::Poseidon, _) => {
unimplemented!("{hash:?} device commit not yet ported (column-major base leaves)")
}
}
}

Expand Down Expand Up @@ -2387,6 +2399,9 @@ fn evaluate_poly_coset_batch_ext3_into_inner(
log_num_rows,
&mut leaves_view,
)?,
DeviceHash::Rpo256 | DeviceHash::Rpx256 | DeviceHash::Poseidon => {
unimplemented!("{hash:?} device commit not yet ported (comp-poly ext3 leaves)")
}
}
}
build_inner_tree_levels_for(hash, stream.as_ref(), be, &mut nodes_dev, num_leaves)?;
Expand Down
22 changes: 20 additions & 2 deletions crypto/math-cuda/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -36,17 +36,35 @@ pub type Result<T> = std::result::Result<T, cudarc::driver::DriverError>;
///
/// The fused LDE+commit pipelines ([`lde`]), the composition-poly tree
/// builders ([`merkle`] / [`blake3`]) and the FRI layer commits ([`fri`])
/// each exist kernel-for-kernel in both families; this enum is the dispatch
/// key callers pass down. It deliberately carries no round counts or
/// each exist kernel-for-kernel in both BYTE families; this enum is the
/// dispatch key callers pass down. It deliberately carries no round counts or
/// parameters: within one build each family is a single concrete hash
/// (keccak-256, or `Blake3Chain` at the compiled round count), exactly as on
/// the host.
///
/// ★ The three ALGEBRAIC keys name hashes whose device kernels are not yet
/// ported. Every dispatch site in this crate carries an arm for them that
/// aborts with `unimplemented!` naming the hash — never an arm that launches a
/// byte-hash kernel in its place. The keys exist ahead of their kernels so the
/// host side (`stark::config::DeviceTreeBackend`) can name every commitment
/// hash under `cuda`: a tree labelled RPO is then built by RPO kernels or not
/// built at all. Porting a family means replacing those arms with launches,
/// and the set of arms is the checklist.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum DeviceHash {
/// Keccak-256 leaves and parents.
Keccak256,
/// `Blake3Chain` leaves and parents at the compiled round count.
Blake3,
/// RPO256 leaves and parents. No device kernels yet: every dispatch site
/// aborts loudly on this key.
Rpo256,
/// RPX256 (XHash12) leaves and parents. No device kernels yet: every
/// dispatch site aborts loudly on this key.
Rpx256,
/// ⚠ Poseidon-original — UNSHIPPABLE on the host side too; present so the
/// key set mirrors `CommitmentHash` one-to-one. No device kernels.
Poseidon,
}

/// Toolchain sanity: plain wrapping u64 vector add. Not a field op.
Expand Down
47 changes: 32 additions & 15 deletions crypto/stark/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -68,13 +68,20 @@ pub type FriLayerMerkleTree<F> = MerkleTree<FriLayerMerkleTreeBackend<F>>;
/// 32-byte-node backend would compile there and hand back trees wearing a
/// name whose hash the kernels never computed.
///
/// This trait closes that hole from both ends: `COMMITMENT_HASH` is the
/// dispatch key `gpu_lde` hands to `math-cuda` (selecting the keccak or the
/// BLAKE3 kernel family at every leaf, level and FRI-layer launch), and
/// implementing the trait is the reviewable claim that device kernels
/// producing exactly this backend's hash exist. A backend over some other
/// hash has no true constant to supply, so writing the impl is a deliberate
/// false statement rather than an omission nobody had to make.
/// This trait closes that hole: `COMMITMENT_HASH` is the dispatch key
/// `gpu_lde` hands to `math-cuda`, which selects the kernel family at every
/// leaf, level and FRI-layer launch — or, for a hash whose kernels are not yet
/// ported (the algebraic three), aborts at that launch with `unimplemented!`
/// naming the hash. Either way a tree labelled `Self` was hashed by `Self`'s
/// kernels or was not built at all; what the trait rules out is the third
/// outcome, a tree built by another hash's kernels and labelled `Self`.
///
/// Implementing it is therefore the reviewable statement of WHICH hash the
/// device must compute for this backend, not a claim that it already can. A
/// backend whose hash no [`CommitmentHash`] variant names has no true constant
/// to supply, so writing the impl is a deliberate false statement rather than
/// an omission nobody had to make. The algebraic impls live beside their
/// backends in `prover::lfm::algebraic_commit`.
pub trait DeviceTreeBackend: IsMerkleTreeBackend<Node = Commitment> {
/// The hash the device kernels must compute for trees labelled `Self`.
const COMMITMENT_HASH: CommitmentHash;
Expand Down Expand Up @@ -114,6 +121,12 @@ where
/// artifacts name a hash. Every such match is a place that has to be revisited
/// before this crate commits under a second hash; adding [`Self::Blake3`] broke
/// them, which is what that mechanism is for.
///
/// ★ Under `cuda` every variant also needs a `math_cuda::DeviceHash` twin.
/// `gpu_lde`'s bridge is total in both directions and asserts the pairing at
/// compile time, so a variant added here without one is a build error naming
/// the gap, and the bridge cannot cross-pair two hashes. A twin whose kernels
/// are not yet ported is legal: its dispatch arms abort loudly.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum CommitmentHash {
/// Keccak-256 at both the leaf and the parent layer.
Expand Down Expand Up @@ -382,15 +395,19 @@ const _: fn() = || {
/// [`DeviceTreeBackend`] impls above true statements rather than decoration —
/// `gpu_lde` dispatches device kernels on that constant, so a mismatch here
/// would be a GPU run hashing under a name the roots do not deserve.
const _: fn() = || {
fn assert_device_hash<B: DeviceTreeBackend>(expect: CommitmentHash) {
assert!(matches!(
(B::COMMITMENT_HASH, expect),
(CommitmentHash::Keccak256, CommitmentHash::Keccak256)
| (CommitmentHash::Blake3, CommitmentHash::Blake3)
));
///
/// A `const` block rather than a never-called closure, so the value comparison
/// is evaluated at compile time and not merely type-checked. Discriminants are
/// compared because `PartialEq` is not `const`. The algebraic configurations
/// carry the same check beside their impls in `prover::lfm::algebraic_commit`.
const _: () = {
const fn assert_device_hash<B: DeviceTreeBackend>(expect: CommitmentHash) {
assert!(
B::COMMITMENT_HASH as u8 == expect as u8,
"a configuration's member must name the configuration's own hash as its device key"
);
}
fn assert_same<T>(_: core::marker::PhantomData<(T, T)>) {}
const fn assert_same<T>(_: core::marker::PhantomData<(T, T)>) {}

assert_device_hash::<<KeccakStarkHash as StarkHash>::Batched<GoldilocksField>>(
CommitmentHash::Keccak256,
Expand Down
90 changes: 74 additions & 16 deletions crypto/stark/src/gpu_lde.rs
Original file line number Diff line number Diff line change
Expand Up @@ -40,27 +40,73 @@ use rayon::prelude::{IndexedParallelIterator, ParallelIterator, ParallelSliceMut
use crate::config::{Commitment, CommitmentHash, DeviceTreeBackend};
use crate::domain::Domain;
use crate::fri::fri_commitment::FriLayer;
use crate::fri::fri_decommit::FriDecommitment;
use crate::trace::LDETraceTable;

/// The `math_cuda` dispatch key for `B`'s hash.
fn device_hash_of<B: DeviceTreeBackend>() -> math_cuda::DeviceHash {
match B::COMMITMENT_HASH {
/// The `math_cuda` dispatch key for a commitment hash.
///
/// Total over [`CommitmentHash`], so a variant added there without a
/// [`math_cuda::DeviceHash`] twin fails to compile HERE, naming the gap, rather
/// than at whatever `match` first meets it. Every tree entry point below
/// dispatches on the result, and `math-cuda` either has kernels for the key or
/// aborts loudly at the first launch — a build must never quietly commit under
/// a hash the configuration did not name (HASH-PINNING.md), so there is no
/// byte-hash fallback anywhere on this path.
const fn device_hash_for(hash: CommitmentHash) -> math_cuda::DeviceHash {
match hash {
CommitmentHash::Keccak256 => math_cuda::DeviceHash::Keccak256,
CommitmentHash::Blake3 => math_cuda::DeviceHash::Blake3,
// The algebraic hashes have no device kernels yet. Loud by design: a
// build must never quietly commit under a hash the configuration did
// not name (HASH-PINNING.md), so there is no byte-hash fallback here.
// The per-table redo's dispatch lane replaces this arm with real
// `DeviceHash` variants once the kernels exist.
CommitmentHash::Rpo256 | CommitmentHash::Rpx256 | CommitmentHash::Poseidon => {
unimplemented!(
"{:?}: device Merkle commits are keccak/BLAKE3-only until the algebraic kernels land",
B::COMMITMENT_HASH
)
}
CommitmentHash::Rpo256 => math_cuda::DeviceHash::Rpo256,
CommitmentHash::Rpx256 => math_cuda::DeviceHash::Rpx256,
CommitmentHash::Poseidon => math_cuda::DeviceHash::Poseidon,
}
}

/// The inverse of [`device_hash_for`] — total over [`math_cuda::DeviceHash`],
/// so a device key no commitment hash names fails to compile here too.
const fn commitment_hash_for(hash: math_cuda::DeviceHash) -> CommitmentHash {
match hash {
math_cuda::DeviceHash::Keccak256 => CommitmentHash::Keccak256,
math_cuda::DeviceHash::Blake3 => CommitmentHash::Blake3,
math_cuda::DeviceHash::Rpo256 => CommitmentHash::Rpo256,
math_cuda::DeviceHash::Rpx256 => CommitmentHash::Rpx256,
math_cuda::DeviceHash::Poseidon => CommitmentHash::Poseidon,
}
}
use crate::fri::fri_decommit::FriDecommitment;
use crate::trace::LDETraceTable;

/// ★ Every [`CommitmentHash`] variant has exactly one [`math_cuda::DeviceHash`]
/// twin, and the pairing is not crossed.
///
/// The two `match`es above being total already proves each side maps
/// somewhere; this proves the two maps are inverse to each other, which is what
/// rules out a mis-paired arm (`Rpx256 => DeviceHash::Rpo256`) — the one
/// editing error that would hand a tree a name its kernels do not deserve once
/// the algebraic kernels exist. A sixth variant fails `device_hash_for` first
/// and is added to this list with its arm. Discriminants are compared because
/// `PartialEq` is not `const`.
const _: () = {
const ALL: [CommitmentHash; 5] = [
CommitmentHash::Keccak256,
CommitmentHash::Blake3,
CommitmentHash::Rpo256,
CommitmentHash::Rpx256,
CommitmentHash::Poseidon,
];
let mut i = 0;
while i < ALL.len() {
let back = commitment_hash_for(device_hash_for(ALL[i]));
assert!(
back as u8 == ALL[i] as u8,
"a CommitmentHash must round-trip through its DeviceHash twin"
);
i += 1;
}
};

/// The `math_cuda` dispatch key for `B`'s hash.
fn device_hash_of<B: DeviceTreeBackend>() -> math_cuda::DeviceHash {
device_hash_for(B::COMMITMENT_HASH)
}

/// Break-even LDE size. For LDE sizes smaller than this, the CPU
/// `coset_lde_full_expand` completes in a few hundred microseconds and the
Expand Down Expand Up @@ -1376,6 +1422,12 @@ where
math_cuda::DeviceHash::Blake3 => {
math_cuda::blake3::build_comp_poly_tree_from_evals_ext3_keep(&raw_parts)
}
math_cuda::DeviceHash::Rpo256
| math_cuda::DeviceHash::Rpx256
| math_cuda::DeviceHash::Poseidon => unimplemented!(
"{:?} device commit not yet ported (comp-poly tree from ext3 evals)",
B::COMMITMENT_HASH
),
} {
Ok(t) => t,
Err(_) => return None,
Expand Down Expand Up @@ -1419,6 +1471,12 @@ where
handle.m,
handle.lde_size,
),
math_cuda::DeviceHash::Rpo256
| math_cuda::DeviceHash::Rpx256
| math_cuda::DeviceHash::Poseidon => unimplemented!(
"{:?} device commit not yet ported (comp-poly tree from resident slabs)",
B::COMMITMENT_HASH
),
}
.ok()?;
GPU_COMP_POLY_TREE_CALLS.fetch_add(1, Ordering::Relaxed);
Expand Down
12 changes: 12 additions & 0 deletions prover/src/hash_pin.rs
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,18 @@
//! `IsStreamingLeafBackend` import in `proof_arena` were found. None of those
//! three shows up on a build that only ever pins BLAKE3.
//!
//! # `cuda` on an algebraic pin
//!
//! Compiles, and still cannot prove under the wrong hash. The algebraic
//! backends are `DeviceTreeBackend`s carrying their own `CommitmentHash` as the
//! device dispatch key, and `math-cuda` has no kernels for those permutations
//! yet, so a GPU run under an algebraic pin aborts at its first device commit
//! with `unimplemented!` naming the hash. ⛔ Neither a `compile_error!` nor a
//! byte-hash fallback belongs here: the first hides the cuda lint arm from the
//! branch, the second is exactly the silent wrong-hash build this module exists
//! to make impossible. Proving a block under an algebraic pin on GPU means
//! landing the kernels, and nothing less.
//!
//! # ⚠ TWO regenerations, not one
//!
//! A pin change is **not** complete until every root blessed under the old hash
Expand Down
Loading
Loading