English | 简体中文(从零编写)
This guide is for an agent that already has a GPU kernel implementation —
with its own code and its own tests — and wants to rewrite it into a
kernel_package that the kernel_zoo platform can benchmark. If instead you
are creating a package from a blank slate, read the step-by-step authoring
guide at kernel_package_guide.md
(简体中文) first; this document assumes you have code to port and explains the
same contracts through that lens.
By the end of this guide you will know:
- the exact on-disk layout of a kernel_package and why it is shaped that way;
- what each file is responsible for, why the design forces that shape, and how the server/runner consume each file;
- a concrete before/after recipe for moving your implementation and your test suite into the kernel_zoo protocol.
Normative references (read these when you need the exact field lists):
| Document | What it pins down |
|---|---|
kernel_package_format.md |
on-disk format + every schema table |
schemas/ |
the JSON Schemas (source of truth) |
api_reference.md |
the HTTP API |
examples/ |
a complete, validating example package |
kernel_zoo is a git-repository-as-database leaderboard. There is no external
database: the kernel repo is the database, and every kernel_package
directory in it is one operator implementation. Understanding the lifecycle is
the fastest way to understand why every file contract exists.
┌──────────────┐ POST /api/submissions ┌────────────────────────────────┐
│ You (agent) │ ───────────────────────▶ │ server │
└──────────────┘ multipart tar.gz │ │
│ validate → enqueue │
│ pending/<uuid>/ │
│ │
┌──────────────┐ GET /api/runner/claim │ │
│ runner │ ◀─────────────────────── │ │
│ (container) │ │ │
│ │ GET /api/runner/package/│ downloads your tarball │
│ │ <uuid> ────────────▶│ un-tars into a fresh workdir │
│ │ │ │
│ │ python build_test_env.py│ │
│ │ <workdir> │ │
│ │ python benchmark.py │ │
│ │ <workdir> │ │
│ │ ── stdout = JSON ──────▶│ │
│ │ POST .../{uuid}/result │ accept? → two-phase git commit│
└──────────────┘ │ → index.json rebuild │
└────────────────────────────────┘
The runner drives your package through exactly two entry points, in order:
python build_test_env.py <workdir>— construct the experiment.python benchmark.py <workdir>— run it, judge it, report a single JSON on stdout.
If either exits non-zero, or benchmark.py's JSON violates its schema, the
result is recorded as rejected. There is no other way your code runs on the
platform.
A kernel_package is a directory inside the kernel repo. Its relative path is
its kernel_id — that string appears in the URL, in desc.yaml's implicit
location, in bench_result.yaml, and in the benchmark output, and every copy
must match exactly.
<arch>/<op_class>/<sub_class>/<quant>/<your_name>/ ← this path = kernel_id
├── .kernel_package/
│ ├── desc.yaml ← operator metadata (you write)
│ ├── bench_result.yaml ← scoreboard (seed only; server owns it after first accept)
│ ├── build_test_env.py ← generates inputs + expected outputs + build artifacts
│ └── benchmark.py ← runs your kernel, checks correctness, measures, decides accept
└── src/ ← your implementation, however you like to organize it
├── my_kernel.py ← (example) your ported kernel code
└── reference.py ← (example) a naive reference your tests compare against
Hard rules
- Every path segment matches
^[A-Za-z0-9._-]+$— no spaces, no slashes, no.., no Unicode. .kernel_package/contains all four files. A package missing any one is markedinvalid(visible in listings but not claimable).desc.yaml.reference_sourceslists relative file paths that must actually exist inside the package — the server checks this before accepting any submission.- Only the server writes
bench_result.yaml. A submission tarball that containsbench_result.yamlis treated as scoreboard tampering and rejected with aVALIDATION_ERROR.
The split between src/ and .kernel_package/ is the core design decision:
src/ is your code (freely shaped, fully overwritten on every accept),
while .kernel_package/ is the contract area — metadata that belongs to
the platform. Your implementation never touches the scoreboard, and the
platform never merges your code; it replaces src/ wholesale.
Most GPU kernel projects contain the pieces below. Do this mapping first — it tells you exactly which files to create.
| Your existing artifact | Maps to | How |
|---|---|---|
the kernel source (kernel.cu, kernel.py, Makefile, …) |
src/… |
copy/adapt it; reference it from desc.yaml |
| a reference / naive implementation used for validation | src/… |
keep it; build_test_env.py imports it to produce expected outputs |
| test data generation (random tensors, fixtures, golden files) | build_test_env.py |
writes workdir/inputs/ + workdir/expected/ |
correctness assertions (assert_allclose, pytest.approx, …) |
benchmark.py |
re-expressed as a correctness check against workdir/expected/ |
| a latency / throughput micro-benchmark | benchmark.py |
re-expressed as metrics |
| per-case test configurations (shapes, dtypes, batch sizes) | KERNEL_ZOO_TEST_CASE_METAS + desc.yaml.io_signature |
the platform's notion of "test cases" |
| the build/install step | build_test_env.py |
compile into workdir/build/ so the benchmark can load it |
| the "does the suite pass?" summary | overall.accept + accept_reason |
your pass/fail policy becomes the accept policy |
The key mental shift: you are not shipping your test runner to kernel_zoo.
You are translating its intent into two files that the platform's own runner
executes. Your assertions do not run under pytest — they run inside
benchmark.py as the correctness gate. Your timings do not print a report —
they become JSON metrics that drive the leaderboard.
The first four path segments follow the classification convention and should
mirror desc.yaml.category:
<arch>/<op_class>/<sub_class>/<quant>/<your_name>/
gfx928/Attention/MHA/fp16/my_flash_attn/
Decide this up front because it is replicated in four places that must agree:
the directory, desc.yaml.category, bench_result.yaml.kernel_id, and every
benchmark output's kernel_id.
Then decide your test cases. A test case is one (parameterized) run of the
operator — a particular shape, batch, dtype configuration. Pick a small set
that captures the operator's behaviour and its performance profile (e.g. a
small shape for correctness sanity and a large shape for throughput). These
become KERNEL_ZOO_TEST_CASE_METAS and the params are free-form but should
be self-describing.
Copy your kernel code into src/ (or any sibling directory you prefer —
include/, Makefile, CMakeLists.txt, .cu, .hip, .cl are all fine;
the platform does not care). The only constraint on src/ is that every path
listed in desc.yaml.reference_sources exists.
Keep your reference/naive implementation in src/ too — build_test_env.py
will import it to generate expected outputs. The example package does exactly
this: build_test_env.py does from ref_attention import reference_attention.
To make your code importable from the package root, both build_test_env.py
and benchmark.py prepend src/ to sys.path:
PKG_ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(PKG_ROOT / "src"))If your kernel needs a compiler step (e.g. hipcc/nvcc), do that in
build_test_env.py and emit artifacts into workdir/build/.
desc.yaml is your operator's "datasheet". It is read by the WebUI, by
index.json, and above all by future optimizing agents that have never
seen your code — so the description field is the most load-bearing piece of
the whole package.
schema_version: "1.0.0"
name: MHA FP16 Reference
category: # must mirror the first 4 path segments
arch: gfx928
op_class: Attention
sub_class: MHA
quant: fp16
summary: Pure-Python reference multi-head attention used to exercise the contract layer.
description: |
## 数学定义
S[b,h,i,:] = (Q[b,h,i,:] @ K[b,h,j,:].T) / sqrt(d_k)
P = softmax(S); O = P @ V
## 输入布局
Q, K, V: [batch, heads, seq_len, head_dim], fp16.
## 性能提示
- Use MFMA instructions for the batched GEMMs.
- Use online safe-softmax.
## 已知陷阱
- Layout of K/V (BHD vs BSH) changes KV-cache performance.
io_signature:
- name: Q
kind: in # in | out | inout
dtype: fp16 # generic dtype name, not vendor-specific
shape: [batch, heads, seq_len, head_dim]
shape_desc: "[B, H, S, D]"
semantics: Query tensor.
- name: O
kind: out
dtype: fp16
shape: [batch, heads, seq_len, head_dim]
shape_desc: "[B, H, S, D]"
semantics: softmax(Q K^T / sqrt(d)) V.
quantization:
scheme: none # or fp8_e4m3, int8, …; plus optional granularity/block_size/group_size
reference_sources: # relative file paths; must exist at accept time
- src/ref_attention.py
status: normal # normal | frozen | deprecated (frozen blocks claiming)
tags: [attention, fp16] # optional, for WebUI filters
owner: alice # optionalField-by-field (required unless marked optional):
| Field | Purpose & why |
|---|---|
schema_version |
semver for future schema migration gating. |
name |
human-readable display name for the WebUI. |
category |
the four classification axes. Why: it duplicates the directory prefix so a single file can be classified without walking paths — but index.json reads this field, not the path, so the two must agree or the WebUI mis-classifies. |
summary |
≤300 chars; shown in /api/kernels listings. |
description |
long markdown, the contract for agents. Why: an optimizer agent must be able to write a competitive implementation without reading your src/. Structure it as math definition / input layout / perf hints / known traps. |
io_signature |
ordered I/O contract. kind is in/out/inout; dtype uses generic names (fp16, bf16, fp8_e4m3, int32) not vendor naming. shape entries may be integers or symbolic names. |
quantization |
scheme required; granularity ∈ per_tensor/per_block/per_channel; sizes are optional. Use scheme: none for unquantized. |
reference_sources |
array of file paths relative to the package. Why: the server verifies these exist before accepting a result — this is what stops a submission from silently referencing files that were never shipped. Note: it is not URLs/DOIs; put paper references in description. |
status |
normal/frozen/deprecated. frozen prevents claiming (used to stop old packages being re-benchmarked). |
min_arch_feature / tags / owner |
optional: runner arch-feature selection, WebUI filters, attribution. |
Anti-patterns (all rejected by the schema, which is closed —
additionalProperties: false):
acceptance.*fields — the accept policy lives inbenchmark.py, not here.reference_sourcescontaining URLs or DOIs — only relative file paths.io_signatureitems missingname/kind/dtype/shape, or using a vendor dtype likefloat16instead offp16.
This file constructs the experiment: it generates every input tensor and
every expected output tensor (and optionally compiles the kernel) so that
benchmark.py only has to run and measure.
Contract
- Must define
build_env(workdir: Path) -> None. - Must be runnable as a script:
python build_test_env.py <workdir>. - Writes (the runner reads exactly these locations):
workdir/inputs/<test_case_id>_*.npy— input tensorsworkdir/expected/<test_case_id>_*.npy— expected outputs (for correctness comparison)workdir/build/— optional compile artifacts (.so,.cubin, …)workdir/build_manifest.json— generation parameters + file sha256
The hard requirement: determinism. Under the same desc.yaml and the same
workdir, build_env must produce bit-identical outputs every time. Use a
fixed constant seed (np.random.default_rng(SEED)) — never time(),
os.urandom, or random.random(). Why: the whole leaderboard is
comparable only if two runs of the same code face identical inputs and
expected outputs. A non-deterministic generator creates false regressions and
destroys the ranking's meaning.
Why separate construction from evaluation? Building inputs/expected offline lets you reuse them across repeated benchmark runs without regenerating (random inputs would break correctness comparison), and keeps bulky, platform-specific compile artifacts out of git.
Minimal shape (this mirrors the example package):
from __future__ import annotations
import json, sys
from pathlib import Path
import numpy as np
PKG_ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(PKG_ROOT / "src"))
from my_kernel import reference_fn # your reference implementation
TEST_CASE_PARAMS = { # must mirror KERNEL_ZOO_TEST_CASE_METAS in benchmark.py
"small_b1_h1_s16_d16": {"batch": 1, "heads": 1, "seq_len": 16, "head_dim": 16},
"large_b1_h2_s64_d32": {"batch": 1, "heads": 2, "seq_len": 64, "head_dim": 32},
}
SEED = 0xBEEF # fixed; do not change
def build_env(workdir: Path) -> None:
(workdir / "inputs").mkdir(parents=True, exist_ok=True)
(workdir / "expected").mkdir(parents=True, exist_ok=True)
rng = np.random.default_rng(SEED)
for tc_id, params in TEST_CASE_PARAMS.items():
q = rng.standard_normal((params["batch"], params["heads"], params["seq_len"], params["head_dim"])).astype(np.float32)
o = reference_fn(q, q, q) # your reference produces the expected output
np.save(workdir / "inputs" / f"{tc_id}_q.npy", q)
np.save(workdir / "expected" / f"{tc_id}_o.npy", o)
(workdir / "build_manifest.json").write_text(json.dumps({"seed": SEED}))
def main(argv: list[str]) -> int:
if len(argv) != 2:
print("usage: build_test_env.py <workdir>", file=sys.stderr); return 2
build_env(Path(argv[1])); return 0
if __name__ == "__main__":
raise SystemExit(main(sys.argv))System interaction: the runner calls this before the benchmark, on a
fresh workdir. If it exits non-zero, the runner reports build_ok=false and
the server rejects the result outright.
This is the heart of the package and where your existing tests do most of their work. It runs the kernel, checks correctness against the expected outputs, measures metrics, and — critically — decides accept or reject. The server trusts this decision.
Contract
- Defines module-level
KERNEL_ZOO_TEST_CASE_METAS: list[dict]. - Runnable as
python benchmark.py <workdir>. - stdout is exactly one JSON document (schema
benchmark_output.json); every log/error line goes to stderr. The runner parses stdout as JSON — one stray line breaks the result. - Makes the accept/reject decision; the server does not re-judge.
KERNEL_ZOO_TEST_CASE_METAS — one entry per test case:
KERNEL_ZOO_TEST_CASE_METAS = [
{
"test_case_id": "small_b1_h1_s16_d16", # ^[A-Za-z0-9._-]+$, unique
"params": {"batch": 1, "heads": 1, "seq_len": 16, "head_dim": 16}, # free-form
"primary_metric": "primary_latency_ms", # must be one of metrics[].name
"metrics": [
{"name": "primary_latency_ms", "unit": "ms", "direction": "lower_better"},
{"name": "tflops", "unit": "TFlops", "direction": "higher_better"},
],
"tolerance": {"correctness_atol": 1e-4, "correctness_rtol": 1e-4},
},
# ... one per test case ...
]Why direction lives here and not in bench_result.yaml: the best-value
comparison rule is self-declared by the benchmark (per case), so the server
can apply it without re-deriving intent; the scoreboard only stores numbers.
Output JSON (core fields; see ../schemas/benchmark_output.json):
{
"schema_version": "1.0.0",
"kernel_id": "gfx928/Attention/MHA/fp16/my_flash_attn",
"ran_at": "2026-08-03T11:00:00Z",
"runner_id": "runner-sh01",
"arch": "gfx928",
"overall": {"accept": true, "accept_reason": "all cases passed; within 5% of best", "build_ok": true},
"cases": [
{
"test_case_id": "small_b1_h1_s16_d16",
"correctness": "passed",
"correctness_detail": {"max_abs_err": 1.2e-4, "max_rel_err": 8e-5},
"accept": true,
"accept_reason": "primary_latency 1.20ms vs current 1.23ms (-2.4%)",
"metrics": [{"name": "primary_latency_ms", "value": 1.20}, {"name": "tflops", "value": 318.7}],
"error": null
}
]
}Server-side hard constraints (violating any of them rejects the result):
- The JSON passes the
benchmark_output.jsonschema. cases[].test_case_idis exactly the set inKERNEL_ZOO_TEST_CASE_METAS— no fewer, no more.- If
overall.acceptis true, every case must havecorrectness == "passed". overall.build_okmust be true.
How your existing tests map in. Your correctness assertions
(assert_allclose(out, expected, atol=..., rtol=...)) become an explicit
correctness check against workdir/expected/:
exp = np.load(expected / f"{tc_id}_o.npy")
abs_err = float(np.max(np.abs(out - exp)))
rel_err = float(np.max(np.abs(out - exp) / (np.abs(exp) + 1e-12)))
passed = abs_err <= tol["correctness_atol"] and rel_err <= tol["correctness_rtol"]Your micro-benchmark's timing becomes a metric, and your suite's pass/fail summary becomes the accept policy. A sensible default policy:
- each case accepts if correctness passes and its primary metric does not
regress more than a fixed tolerance (e.g. 5%) vs the current best in
bench_result.yaml; overall.accept = all(case.accept).
Anti-patterns (all cause rejections or runtime failures):
- printing logs to stdout (the JSON parser breaks);
correctness == "failed"on a case that still hasaccept: true(422);metrics[].valuebeingNaN/Inf;- omitting a test case from
casesbecause it is "not important"; - smuggling the result through a side channel (env vars, files) — the runner only reads stdout.
System interaction: the runner runs python benchmark.py <workdir>,
validates stdout against the schema, and posts the JSON (plus base64 runner
logs) to POST /api/submissions/{uuid}/result. On accept, the server
rewrites bench_result.yaml and commits via a two-phase commit — the SHA it
stamps is real and present in git log (it cannot be predicted ahead of
time, hence two commits, not an amend).
Even though only the server owns the real scoreboard, .kernel_package/
must contain all four files to be valid — so you ship a seed version. The
server overwrites it on the first accepted submission.
schema_version: "1.0.0"
kernel_id: gfx928/Attention/MHA/fp16/my_flash_attn # must equal the directory path
cases:
- test_case_id: small_b1_h1_s16_d16
metrics_best:
primary_latency_ms: {value: 0.20, submitter: seed, commit_sha: "0000000", recorded_at: "2026-08-02T00:00:00Z"}
tflops: {value: 0.001, submitter: seed, commit_sha: "0000000", recorded_at: "2026-08-02T00:00:00Z"}
metrics_current:
primary_latency_ms: {value: 0.20, submitter: seed, commit_sha: "0000000", recorded_at: "2026-08-02T00:00:00Z"}
tflops: {value: 0.001, submitter: seed, commit_sha: "0000000", recorded_at: "2026-08-02T00:00:00Z"}
correctness_current: passed
- test_case_id: large_b1_h2_s64_d32
# ... same structure ...Rules that matter:
- The
commit_shasentinel must be exactly"0000000". The server recognizes this special value and unconditionally overwrites the seed on the first accept (no best-value comparison), so a fresh package is never "stuck" behind its placeholder. - Only
metrics_bestandmetrics_currentare stored — history is recovered viagit log. This keeps the file from growing. - Do not put this file in the submission tarball (see §11).
# 1. Validate the four files exist and every schema passes.
kernel_zoo-validate-package <kernel_package directory>
# (or: python -m kernel_zoo.cli.validate_package <dir>)
# exit 0 = valid, 2 = validation issues, 3 = IO/usage error
# 2. Simulate the runner locally.
WORKDIR=$(mktemp -d)
python "$PWD/gfx928/Attention/MHA/fp16/my_flash_attn/.kernel_package/build_test_env.py" "$WORKDIR"
python "$PWD/gfx928/Attention/MHA/fp16/my_flash_attn/.kernel_package/benchmark.py" "$WORKDIR" > /tmp/result.json 2>/tmp/bench.log
# 3. Validate the emitted JSON against the schema.
python -c "import json, jsonschema; \
jsonschema.validate(json.load(open('/tmp/result.json')), \
json.load(open('schemas/benchmark_output.json')))"
# 4. Idempotency check (the hard requirement).
WORKDIR2=$(mktemp -d)
python "$PWD/gfx928/Attention/MHA/fp16/my_flash_attn/.kernel_package/build_test_env.py" "$WORKDIR2"
diff -r "$WORKDIR/inputs" "$WORKDIR2/inputs" && echo "inputs identical"
diff -r "$WORKDIR/expected" "$WORKDIR2/expected" && echo "expected identical"The tarball layout is not the on-disk layout. The top-level directory
inside the tar must be the kernel_id, and it must not contain
bench_result.yaml:
gfx928/Attention/MHA/fp16/my_flash_attn/ ← top-level dir name = kernel_id
├── .kernel_package/
│ ├── desc.yaml
│ ├── build_test_env.py
│ └── benchmark.py
└── src/... ← your implementation
import io, tarfile
from pathlib import Path
KERNEL_ID = "gfx928/Attention/MHA/fp16/my_flash_attn"
pkg_root = Path(KERNEL_ID) # local package root (has .kernel_package/ + src/)
buf = io.BytesIO()
with tarfile.open(fileobj=buf, mode="w:gz") as tf:
tf.add(pkg_root, arcname=KERNEL_ID) # top-level dir = kernel_id
# Exclude bench_result.yaml — or it is treated as scoreboard tampering.
for member in list(tf.getmembers()):
if member.name.endswith(".kernel_package/bench_result.yaml"):
tf.members.remove(member)
tarball_bytes = buf.getvalue()Submit (the long-poll returns the queued/running/done status; the default timeout is 30 s):
curl -X POST http://127.0.0.1:8000/api/submissions \
-H "X-Submitter-Id: alice" \
-F "kernel_id=$KERNEL_ID" \
-F "submitter_id=alice" \
-F "optimization_summary=vectorized loads; reduced register spills" \
-F "package=@package.tar.gz;type=application/gzip"
curl -H "X-Submitter-Id: alice" \
http://127.0.0.1:8000/api/submissions/<uuid>/statusoptimization_summary is optional free text describing what this submission
optimizes. On accept it is written into the source commit message as an
optimization_summary: block, so every attempt in git log records the
optimizer's own notes.
On accept, the server overwrites src/, updates bench_result.yaml, commits
twice (source commit → bench-result commit), and rebuilds index.json — your
operator is on the leaderboard.
| Decision | Why |
|---|---|
| git repo = database | no DBMS, no schema migration; git log is the entire history |
src/ separate from .kernel_package/ |
your implementation vs. the platform's records never mix; server can wholesale-overwrite src/ without touching the scoreboard |
bench_result.yaml server-owned |
submitters cannot fabricate scores; tampering is rejected |
benchmark.py decides accept |
the server stays a thin shell and never re-implements your operator's correctness/regression semantics |
build_test_env.py deterministic |
bit-identical inputs/expected ⇒ comparable, trustworthy leaderboard |
| construction separated from evaluation | reuse generated data across runs; keep build artifacts out of git |
| stdout = one JSON, logs to stderr | the runner parses stdout; any interleaving corrupts the result |
| two-phase git commit | the accept-commit's SHA cannot be known before committing; two commits keep the stamped SHA real |
reference_sources are file paths |
the server can verify the shipped sources actually exist |
closed schemas (additionalProperties: false) |
catches stale/foreign fields (e.g. old acceptance.*) at validation time, not in production |
Directory & naming
- every path segment matches
^[A-Za-z0-9._-]+$ -
.kernel_package/has all four files -
desc.yaml.category== first 4 path segments -
bench_result.yaml.kernel_id== directory path - the
kernel_idin benchmark output == directory path
desc.yaml
- all
reference_sourcespaths exist inside the package -
descriptionis self-contained (an agent could write a competitive impl from it alone) -
statusis notfrozen - no
acceptance.*/ URLreference_sources(schema rejects them)
build_test_env.py
- defines
build_env(workdir)and runs aspython build_test_env.py <workdir> - fixed random seed (determinism is mandatory)
- writes to
workdir/inputs/,workdir/expected/,workdir/build_manifest.json
benchmark.py
-
KERNEL_ZOO_TEST_CASE_METASmatchesbench_result.yaml.cases - each
primary_metricappears in that case'smetrics[].name - runs as
python benchmark.py <workdir> - stdout is exactly one valid JSON; all logs go to stderr
- a case with
correctness: failedalso hasaccept: false - no
NaN/Infmetric values - the accept policy is actually implemented (it decides, it doesn't just emit metrics)
Tarball
- top-level dir name == kernel_id
- no
.kernel_package/bench_result.yaml - filename ends in
.tar.gz
kernel_package_format.md— normative on-disk format and every schema (English).kernel_package_guide.md— 简体中文:从零创建 kernel_package 的完整中文编写指南.api_reference.md— HTTP API, roles, error envelope.examples/gfx928/Attention/MHA/fp16/reference_impl/— a complete, schema-validating example package you can copy as a skeleton.../api/openapi.yaml— the normative OpenAPI contract.../README.md— project overview (English) /../README.zh-CN.md(简体中文).