Skip to content

Latest commit

 

History

History
624 lines (502 loc) · 28.5 KB

File metadata and controls

624 lines (502 loc) · 28.5 KB

kernel_package authoring guide (English)

English | 简体中文(从零编写)

This guide is for an agent that already has a GPU kernel implementation — with its own code and its own tests — and wants to rewrite it into a kernel_package that the kernel_zoo platform can benchmark. If instead you are creating a package from a blank slate, read the step-by-step authoring guide at kernel_package_guide.md (简体中文) first; this document assumes you have code to port and explains the same contracts through that lens.

By the end of this guide you will know:

  • the exact on-disk layout of a kernel_package and why it is shaped that way;
  • what each file is responsible for, why the design forces that shape, and how the server/runner consume each file;
  • a concrete before/after recipe for moving your implementation and your test suite into the kernel_zoo protocol.

Normative references (read these when you need the exact field lists):

Document What it pins down
kernel_package_format.md on-disk format + every schema table
schemas/ the JSON Schemas (source of truth)
api_reference.md the HTTP API
examples/ a complete, validating example package

1. The system you are plugging into

kernel_zoo is a git-repository-as-database leaderboard. There is no external database: the kernel repo is the database, and every kernel_package directory in it is one operator implementation. Understanding the lifecycle is the fastest way to understand why every file contract exists.

┌──────────────┐  POST /api/submissions   ┌────────────────────────────────┐
│  You (agent) │ ───────────────────────▶ │              server             │
└──────────────┘   multipart tar.gz       │                                │
                                          │  validate → enqueue            │
                                          │  pending/<uuid>/               │
                                          │                                │
┌──────────────┐  GET /api/runner/claim   │                                │
│    runner    │ ◀─────────────────────── │                                │
│ (container)  │                          │                                │
│              │  GET /api/runner/package/│  downloads your tarball        │
│              │      <uuid> ────────────▶│  un-tars into a fresh workdir  │
│              │                          │                                │
│              │  python build_test_env.py│                                │
│              │      <workdir>           │                                │
│              │  python benchmark.py     │                                │
│              │      <workdir>           │                                │
│              │  ── stdout = JSON ──────▶│                                │
│              │  POST .../{uuid}/result  │  accept? → two-phase git commit│
└──────────────┘                          │  → index.json rebuild          │
                                          └────────────────────────────────┘

The runner drives your package through exactly two entry points, in order:

  1. python build_test_env.py <workdir>construct the experiment.
  2. python benchmark.py <workdir>run it, judge it, report a single JSON on stdout.

If either exits non-zero, or benchmark.py's JSON violates its schema, the result is recorded as rejected. There is no other way your code runs on the platform.


2. The target layout

A kernel_package is a directory inside the kernel repo. Its relative path is its kernel_id — that string appears in the URL, in desc.yaml's implicit location, in bench_result.yaml, and in the benchmark output, and every copy must match exactly.

<arch>/<op_class>/<sub_class>/<quant>/<your_name>/      ← this path = kernel_id
├── .kernel_package/
│   ├── desc.yaml             ← operator metadata (you write)
│   ├── bench_result.yaml     ← scoreboard (seed only; server owns it after first accept)
│   ├── build_test_env.py     ← generates inputs + expected outputs + build artifacts
│   └── benchmark.py          ← runs your kernel, checks correctness, measures, decides accept
└── src/                      ← your implementation, however you like to organize it
    ├── my_kernel.py          ← (example) your ported kernel code
    └── reference.py          ← (example) a naive reference your tests compare against

Hard rules

  • Every path segment matches ^[A-Za-z0-9._-]+$ — no spaces, no slashes, no .., no Unicode.
  • .kernel_package/ contains all four files. A package missing any one is marked invalid (visible in listings but not claimable).
  • desc.yaml.reference_sources lists relative file paths that must actually exist inside the package — the server checks this before accepting any submission.
  • Only the server writes bench_result.yaml. A submission tarball that contains bench_result.yaml is treated as scoreboard tampering and rejected with a VALIDATION_ERROR.

The split between src/ and .kernel_package/ is the core design decision: src/ is your code (freely shaped, fully overwritten on every accept), while .kernel_package/ is the contract area — metadata that belongs to the platform. Your implementation never touches the scoreboard, and the platform never merges your code; it replaces src/ wholesale.


3. Inventory your existing project

Most GPU kernel projects contain the pieces below. Do this mapping first — it tells you exactly which files to create.

Your existing artifact Maps to How
the kernel source (kernel.cu, kernel.py, Makefile, …) src/… copy/adapt it; reference it from desc.yaml
a reference / naive implementation used for validation src/… keep it; build_test_env.py imports it to produce expected outputs
test data generation (random tensors, fixtures, golden files) build_test_env.py writes workdir/inputs/ + workdir/expected/
correctness assertions (assert_allclose, pytest.approx, …) benchmark.py re-expressed as a correctness check against workdir/expected/
a latency / throughput micro-benchmark benchmark.py re-expressed as metrics
per-case test configurations (shapes, dtypes, batch sizes) KERNEL_ZOO_TEST_CASE_METAS + desc.yaml.io_signature the platform's notion of "test cases"
the build/install step build_test_env.py compile into workdir/build/ so the benchmark can load it
the "does the suite pass?" summary overall.accept + accept_reason your pass/fail policy becomes the accept policy

The key mental shift: you are not shipping your test runner to kernel_zoo. You are translating its intent into two files that the platform's own runner executes. Your assertions do not run under pytest — they run inside benchmark.py as the correctness gate. Your timings do not print a report — they become JSON metrics that drive the leaderboard.


4. Step 1 — choose kernel_id and your test cases

The first four path segments follow the classification convention and should mirror desc.yaml.category:

<arch>/<op_class>/<sub_class>/<quant>/<your_name>/
gfx928/Attention/MHA/fp16/my_flash_attn/

Decide this up front because it is replicated in four places that must agree: the directory, desc.yaml.category, bench_result.yaml.kernel_id, and every benchmark output's kernel_id.

Then decide your test cases. A test case is one (parameterized) run of the operator — a particular shape, batch, dtype configuration. Pick a small set that captures the operator's behaviour and its performance profile (e.g. a small shape for correctness sanity and a large shape for throughput). These become KERNEL_ZOO_TEST_CASE_METAS and the params are free-form but should be self-describing.


5. Step 2 — place your implementation in src/

Copy your kernel code into src/ (or any sibling directory you prefer — include/, Makefile, CMakeLists.txt, .cu, .hip, .cl are all fine; the platform does not care). The only constraint on src/ is that every path listed in desc.yaml.reference_sources exists.

Keep your reference/naive implementation in src/ too — build_test_env.py will import it to generate expected outputs. The example package does exactly this: build_test_env.py does from ref_attention import reference_attention.

To make your code importable from the package root, both build_test_env.py and benchmark.py prepend src/ to sys.path:

PKG_ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(PKG_ROOT / "src"))

If your kernel needs a compiler step (e.g. hipcc/nvcc), do that in build_test_env.py and emit artifacts into workdir/build/.


6. Step 3 — write desc.yaml

desc.yaml is your operator's "datasheet". It is read by the WebUI, by index.json, and above all by future optimizing agents that have never seen your code — so the description field is the most load-bearing piece of the whole package.

schema_version: "1.0.0"
name: MHA FP16 Reference
category:                        # must mirror the first 4 path segments
  arch: gfx928
  op_class: Attention
  sub_class: MHA
  quant: fp16
summary: Pure-Python reference multi-head attention used to exercise the contract layer.
description: |
  ## 数学定义
  S[b,h,i,:] = (Q[b,h,i,:] @ K[b,h,j,:].T) / sqrt(d_k)
  P = softmax(S);  O = P @ V

  ## 输入布局
  Q, K, V: [batch, heads, seq_len, head_dim], fp16.

  ## 性能提示
  - Use MFMA instructions for the batched GEMMs.
  - Use online safe-softmax.

  ## 已知陷阱
  - Layout of K/V (BHD vs BSH) changes KV-cache performance.
io_signature:
  - name: Q
    kind: in                     # in | out | inout
    dtype: fp16                  # generic dtype name, not vendor-specific
    shape: [batch, heads, seq_len, head_dim]
    shape_desc: "[B, H, S, D]"
    semantics: Query tensor.
  - name: O
    kind: out
    dtype: fp16
    shape: [batch, heads, seq_len, head_dim]
    shape_desc: "[B, H, S, D]"
    semantics: softmax(Q K^T / sqrt(d)) V.
quantization:
  scheme: none                   # or fp8_e4m3, int8, …; plus optional granularity/block_size/group_size
reference_sources:               # relative file paths; must exist at accept time
  - src/ref_attention.py
status: normal                   # normal | frozen | deprecated (frozen blocks claiming)
tags: [attention, fp16]          # optional, for WebUI filters
owner: alice                     # optional

Field-by-field (required unless marked optional):

Field Purpose & why
schema_version semver for future schema migration gating.
name human-readable display name for the WebUI.
category the four classification axes. Why: it duplicates the directory prefix so a single file can be classified without walking paths — but index.json reads this field, not the path, so the two must agree or the WebUI mis-classifies.
summary ≤300 chars; shown in /api/kernels listings.
description long markdown, the contract for agents. Why: an optimizer agent must be able to write a competitive implementation without reading your src/. Structure it as math definition / input layout / perf hints / known traps.
io_signature ordered I/O contract. kind is in/out/inout; dtype uses generic names (fp16, bf16, fp8_e4m3, int32) not vendor naming. shape entries may be integers or symbolic names.
quantization scheme required; granularityper_tensor/per_block/per_channel; sizes are optional. Use scheme: none for unquantized.
reference_sources array of file paths relative to the package. Why: the server verifies these exist before accepting a result — this is what stops a submission from silently referencing files that were never shipped. Note: it is not URLs/DOIs; put paper references in description.
status normal/frozen/deprecated. frozen prevents claiming (used to stop old packages being re-benchmarked).
min_arch_feature / tags / owner optional: runner arch-feature selection, WebUI filters, attribution.

Anti-patterns (all rejected by the schema, which is closed — additionalProperties: false):

  • acceptance.* fields — the accept policy lives in benchmark.py, not here.
  • reference_sources containing URLs or DOIs — only relative file paths.
  • io_signature items missing name/kind/dtype/shape, or using a vendor dtype like float16 instead of fp16.

7. Step 4 — port test-data generation into build_test_env.py

This file constructs the experiment: it generates every input tensor and every expected output tensor (and optionally compiles the kernel) so that benchmark.py only has to run and measure.

Contract

  • Must define build_env(workdir: Path) -> None.
  • Must be runnable as a script: python build_test_env.py <workdir>.
  • Writes (the runner reads exactly these locations):
    • workdir/inputs/<test_case_id>_*.npy — input tensors
    • workdir/expected/<test_case_id>_*.npy — expected outputs (for correctness comparison)
    • workdir/build/ — optional compile artifacts (.so, .cubin, …)
    • workdir/build_manifest.json — generation parameters + file sha256

The hard requirement: determinism. Under the same desc.yaml and the same workdir, build_env must produce bit-identical outputs every time. Use a fixed constant seed (np.random.default_rng(SEED)) — never time(), os.urandom, or random.random(). Why: the whole leaderboard is comparable only if two runs of the same code face identical inputs and expected outputs. A non-deterministic generator creates false regressions and destroys the ranking's meaning.

Why separate construction from evaluation? Building inputs/expected offline lets you reuse them across repeated benchmark runs without regenerating (random inputs would break correctness comparison), and keeps bulky, platform-specific compile artifacts out of git.

Minimal shape (this mirrors the example package):

from __future__ import annotations
import json, sys
from pathlib import Path
import numpy as np

PKG_ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(PKG_ROOT / "src"))
from my_kernel import reference_fn  # your reference implementation

TEST_CASE_PARAMS = {           # must mirror KERNEL_ZOO_TEST_CASE_METAS in benchmark.py
    "small_b1_h1_s16_d16": {"batch": 1, "heads": 1, "seq_len": 16, "head_dim": 16},
    "large_b1_h2_s64_d32": {"batch": 1, "heads": 2, "seq_len": 64, "head_dim": 32},
}
SEED = 0xBEEF                 # fixed; do not change

def build_env(workdir: Path) -> None:
    (workdir / "inputs").mkdir(parents=True, exist_ok=True)
    (workdir / "expected").mkdir(parents=True, exist_ok=True)
    rng = np.random.default_rng(SEED)
    for tc_id, params in TEST_CASE_PARAMS.items():
        q = rng.standard_normal((params["batch"], params["heads"], params["seq_len"], params["head_dim"])).astype(np.float32)
        o = reference_fn(q, q, q)              # your reference produces the expected output
        np.save(workdir / "inputs" / f"{tc_id}_q.npy", q)
        np.save(workdir / "expected" / f"{tc_id}_o.npy", o)
    (workdir / "build_manifest.json").write_text(json.dumps({"seed": SEED}))

def main(argv: list[str]) -> int:
    if len(argv) != 2:
        print("usage: build_test_env.py <workdir>", file=sys.stderr); return 2
    build_env(Path(argv[1])); return 0

if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

System interaction: the runner calls this before the benchmark, on a fresh workdir. If it exits non-zero, the runner reports build_ok=false and the server rejects the result outright.


8. Step 5 — port correctness + performance into benchmark.py

This is the heart of the package and where your existing tests do most of their work. It runs the kernel, checks correctness against the expected outputs, measures metrics, and — critically — decides accept or reject. The server trusts this decision.

Contract

  • Defines module-level KERNEL_ZOO_TEST_CASE_METAS: list[dict].
  • Runnable as python benchmark.py <workdir>.
  • stdout is exactly one JSON document (schema benchmark_output.json); every log/error line goes to stderr. The runner parses stdout as JSON — one stray line breaks the result.
  • Makes the accept/reject decision; the server does not re-judge.

KERNEL_ZOO_TEST_CASE_METAS — one entry per test case:

KERNEL_ZOO_TEST_CASE_METAS = [
    {
        "test_case_id": "small_b1_h1_s16_d16",       # ^[A-Za-z0-9._-]+$, unique
        "params": {"batch": 1, "heads": 1, "seq_len": 16, "head_dim": 16},  # free-form
        "primary_metric": "primary_latency_ms",       # must be one of metrics[].name
        "metrics": [
            {"name": "primary_latency_ms", "unit": "ms", "direction": "lower_better"},
            {"name": "tflops", "unit": "TFlops", "direction": "higher_better"},
        ],
        "tolerance": {"correctness_atol": 1e-4, "correctness_rtol": 1e-4},
    },
    # ... one per test case ...
]

Why direction lives here and not in bench_result.yaml: the best-value comparison rule is self-declared by the benchmark (per case), so the server can apply it without re-deriving intent; the scoreboard only stores numbers.

Output JSON (core fields; see ../schemas/benchmark_output.json):

{
  "schema_version": "1.0.0",
  "kernel_id": "gfx928/Attention/MHA/fp16/my_flash_attn",
  "ran_at": "2026-08-03T11:00:00Z",
  "runner_id": "runner-sh01",
  "arch": "gfx928",
  "overall": {"accept": true, "accept_reason": "all cases passed; within 5% of best", "build_ok": true},
  "cases": [
    {
      "test_case_id": "small_b1_h1_s16_d16",
      "correctness": "passed",
      "correctness_detail": {"max_abs_err": 1.2e-4, "max_rel_err": 8e-5},
      "accept": true,
      "accept_reason": "primary_latency 1.20ms vs current 1.23ms (-2.4%)",
      "metrics": [{"name": "primary_latency_ms", "value": 1.20}, {"name": "tflops", "value": 318.7}],
      "error": null
    }
  ]
}

Server-side hard constraints (violating any of them rejects the result):

  1. The JSON passes the benchmark_output.json schema.
  2. cases[].test_case_id is exactly the set in KERNEL_ZOO_TEST_CASE_METAS — no fewer, no more.
  3. If overall.accept is true, every case must have correctness == "passed".
  4. overall.build_ok must be true.

How your existing tests map in. Your correctness assertions (assert_allclose(out, expected, atol=..., rtol=...)) become an explicit correctness check against workdir/expected/:

exp = np.load(expected / f"{tc_id}_o.npy")
abs_err = float(np.max(np.abs(out - exp)))
rel_err = float(np.max(np.abs(out - exp) / (np.abs(exp) + 1e-12)))
passed = abs_err <= tol["correctness_atol"] and rel_err <= tol["correctness_rtol"]

Your micro-benchmark's timing becomes a metric, and your suite's pass/fail summary becomes the accept policy. A sensible default policy:

  • each case accepts if correctness passes and its primary metric does not regress more than a fixed tolerance (e.g. 5%) vs the current best in bench_result.yaml;
  • overall.accept = all(case.accept).

Anti-patterns (all cause rejections or runtime failures):

  • printing logs to stdout (the JSON parser breaks);
  • correctness == "failed" on a case that still has accept: true (422);
  • metrics[].value being NaN/Inf;
  • omitting a test case from cases because it is "not important";
  • smuggling the result through a side channel (env vars, files) — the runner only reads stdout.

System interaction: the runner runs python benchmark.py <workdir>, validates stdout against the schema, and posts the JSON (plus base64 runner logs) to POST /api/submissions/{uuid}/result. On accept, the server rewrites bench_result.yaml and commits via a two-phase commit — the SHA it stamps is real and present in git log (it cannot be predicted ahead of time, hence two commits, not an amend).


9. Step 6 — seed bench_result.yaml

Even though only the server owns the real scoreboard, .kernel_package/ must contain all four files to be valid — so you ship a seed version. The server overwrites it on the first accepted submission.

schema_version: "1.0.0"
kernel_id: gfx928/Attention/MHA/fp16/my_flash_attn   # must equal the directory path
cases:
  - test_case_id: small_b1_h1_s16_d16
    metrics_best:
      primary_latency_ms: {value: 0.20, submitter: seed, commit_sha: "0000000", recorded_at: "2026-08-02T00:00:00Z"}
      tflops:             {value: 0.001, submitter: seed, commit_sha: "0000000", recorded_at: "2026-08-02T00:00:00Z"}
    metrics_current:
      primary_latency_ms: {value: 0.20, submitter: seed, commit_sha: "0000000", recorded_at: "2026-08-02T00:00:00Z"}
      tflops:             {value: 0.001, submitter: seed, commit_sha: "0000000", recorded_at: "2026-08-02T00:00:00Z"}
    correctness_current: passed
  - test_case_id: large_b1_h2_s64_d32
    # ... same structure ...

Rules that matter:

  • The commit_sha sentinel must be exactly "0000000". The server recognizes this special value and unconditionally overwrites the seed on the first accept (no best-value comparison), so a fresh package is never "stuck" behind its placeholder.
  • Only metrics_best and metrics_current are stored — history is recovered via git log. This keeps the file from growing.
  • Do not put this file in the submission tarball (see §11).

10. Step 7 — local validation loop (run before you submit)

# 1. Validate the four files exist and every schema passes.
kernel_zoo-validate-package <kernel_package directory>
#    (or: python -m kernel_zoo.cli.validate_package <dir>)
#    exit 0 = valid, 2 = validation issues, 3 = IO/usage error

# 2. Simulate the runner locally.
WORKDIR=$(mktemp -d)
python "$PWD/gfx928/Attention/MHA/fp16/my_flash_attn/.kernel_package/build_test_env.py" "$WORKDIR"
python "$PWD/gfx928/Attention/MHA/fp16/my_flash_attn/.kernel_package/benchmark.py" "$WORKDIR" > /tmp/result.json 2>/tmp/bench.log

# 3. Validate the emitted JSON against the schema.
python -c "import json, jsonschema; \
  jsonschema.validate(json.load(open('/tmp/result.json')), \
  json.load(open('schemas/benchmark_output.json')))"

# 4. Idempotency check (the hard requirement).
WORKDIR2=$(mktemp -d)
python "$PWD/gfx928/Attention/MHA/fp16/my_flash_attn/.kernel_package/build_test_env.py" "$WORKDIR2"
diff -r "$WORKDIR/inputs" "$WORKDIR2/inputs" && echo "inputs identical"
diff -r "$WORKDIR/expected" "$WORKDIR2/expected" && echo "expected identical"

11. Step 8 — package and submit

The tarball layout is not the on-disk layout. The top-level directory inside the tar must be the kernel_id, and it must not contain bench_result.yaml:

gfx928/Attention/MHA/fp16/my_flash_attn/     ← top-level dir name = kernel_id
├── .kernel_package/
│   ├── desc.yaml
│   ├── build_test_env.py
│   └── benchmark.py
└── src/...                                  ← your implementation
import io, tarfile
from pathlib import Path

KERNEL_ID = "gfx928/Attention/MHA/fp16/my_flash_attn"
pkg_root = Path(KERNEL_ID)                     # local package root (has .kernel_package/ + src/)

buf = io.BytesIO()
with tarfile.open(fileobj=buf, mode="w:gz") as tf:
    tf.add(pkg_root, arcname=KERNEL_ID)        # top-level dir = kernel_id
    # Exclude bench_result.yaml — or it is treated as scoreboard tampering.
    for member in list(tf.getmembers()):
        if member.name.endswith(".kernel_package/bench_result.yaml"):
            tf.members.remove(member)
tarball_bytes = buf.getvalue()

Submit (the long-poll returns the queued/running/done status; the default timeout is 30 s):

curl -X POST http://127.0.0.1:8000/api/submissions \
  -H "X-Submitter-Id: alice" \
  -F "kernel_id=$KERNEL_ID" \
  -F "submitter_id=alice" \
  -F "optimization_summary=vectorized loads; reduced register spills" \
  -F "package=@package.tar.gz;type=application/gzip"

curl -H "X-Submitter-Id: alice" \
  http://127.0.0.1:8000/api/submissions/<uuid>/status

optimization_summary is optional free text describing what this submission optimizes. On accept it is written into the source commit message as an optimization_summary: block, so every attempt in git log records the optimizer's own notes.

On accept, the server overwrites src/, updates bench_result.yaml, commits twice (source commit → bench-result commit), and rebuilds index.json — your operator is on the leaderboard.


12. Design rationale, in one place

Decision Why
git repo = database no DBMS, no schema migration; git log is the entire history
src/ separate from .kernel_package/ your implementation vs. the platform's records never mix; server can wholesale-overwrite src/ without touching the scoreboard
bench_result.yaml server-owned submitters cannot fabricate scores; tampering is rejected
benchmark.py decides accept the server stays a thin shell and never re-implements your operator's correctness/regression semantics
build_test_env.py deterministic bit-identical inputs/expected ⇒ comparable, trustworthy leaderboard
construction separated from evaluation reuse generated data across runs; keep build artifacts out of git
stdout = one JSON, logs to stderr the runner parses stdout; any interleaving corrupts the result
two-phase git commit the accept-commit's SHA cannot be known before committing; two commits keep the stamped SHA real
reference_sources are file paths the server can verify the shipped sources actually exist
closed schemas (additionalProperties: false) catches stale/foreign fields (e.g. old acceptance.*) at validation time, not in production

13. Pre-submission checklist

Directory & naming

  • every path segment matches ^[A-Za-z0-9._-]+$
  • .kernel_package/ has all four files
  • desc.yaml.category == first 4 path segments
  • bench_result.yaml.kernel_id == directory path
  • the kernel_id in benchmark output == directory path

desc.yaml

  • all reference_sources paths exist inside the package
  • description is self-contained (an agent could write a competitive impl from it alone)
  • status is not frozen
  • no acceptance.* / URL reference_sources (schema rejects them)

build_test_env.py

  • defines build_env(workdir) and runs as python build_test_env.py <workdir>
  • fixed random seed (determinism is mandatory)
  • writes to workdir/inputs/, workdir/expected/, workdir/build_manifest.json

benchmark.py

  • KERNEL_ZOO_TEST_CASE_METAS matches bench_result.yaml.cases
  • each primary_metric appears in that case's metrics[].name
  • runs as python benchmark.py <workdir>
  • stdout is exactly one valid JSON; all logs go to stderr
  • a case with correctness: failed also has accept: false
  • no NaN/Inf metric values
  • the accept policy is actually implemented (it decides, it doesn't just emit metrics)

Tarball

  • top-level dir name == kernel_id
  • no .kernel_package/bench_result.yaml
  • filename ends in .tar.gz

14. References