Skip to content

fix(ks/auto-bench): clone reference inputs for v1 instead of re-seeding - #167

Open
Jiwangreal wants to merge 3 commits into
DeepLink-org:mainfrom
Jiwangreal:fix/ks-auto-bench-model-device
Open

fix(ks/auto-bench): clone reference inputs for v1 instead of re-seeding#167
Jiwangreal wants to merge 3 commits into
DeepLink-org:mainfrom
Jiwangreal:fix/ks-auto-bench-model-device

Conversation

@Jiwangreal

@Jiwangreal Jiwangreal commented Jul 23, 2026

Copy link
Copy Markdown

Summary

v0/reference get_inputs is now the canonical input source. v1 receives a deep copy of the same tensors via clone_value() instead of generating its own inputs under the same seed, eliminating false correctness failures from CPU vs CUDA RNG differences.

compare_case later moves both copies to the same accelerator before executing either implementation, so device placement is unaffected.

Changes

  • benchmarks/ks/auto_bench.py: In build_case, v1 inputs are cloned from v0 canonical inputs instead of independently generated via set_seed + v1_get_inputs. Removes the argument-count check (canonical data guarantees matching lengths).
  • tests/benchmarks/test_ks_auto_bench.py: Adds test_build_case_clones_reference_inputs_for_v1 verifying both value equality and independent storage (distinct data_ptr).

Verification

  • python3.10 -m unittest tests.benchmarks.test_ks_auto_bench -v passed 4/4
  • Task 1 benchmark passes accuracy check at 1.069x speedup

Closes the device-RNG mismatch that caused false negatives when v0 ran on CPU and v1 on CUDA (or vice versa).

wangji added 3 commits July 23, 2026 16:38
…-sdpa flag

- Reseed RNG before constructing both v0 and v1 models in build_case
  so parameterized models start from equivalent weights.
- Add --math-sdpa flag to force PyTorch SDPA to math backend,
  bypassing incompatible FlashAttention plugins on ROCm/DCU.
- Add unit test verifying reseeded models produce identical weights.
v0/reference get_inputs is now the canonical input source.  v1
receives a deep copy of the same tensors via clone_value() instead
of generating its own inputs under the same seed, eliminating
false correctness failures from CPU vs CUDA RNG differences.

compare_case later moves both copies to the same accelerator before
executing, so device placement is unaffected.

Adds test_build_case_clones_reference_inputs_for_v1 to verify both
value equality and independent storage (distinct data_ptr).
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.


wangji seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

@Jiwangreal Jiwangreal changed the title fix(ks): move benchmark models to accelerator + reseed for deterministic results fix(ks/auto-bench): clone reference inputs for v1 instead of re-seeding Jul 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants