MetaInfer

Let AI generate high-performance, compact, purpose-built LLM inference frameworks

Describe what you want — MetaInfer builds your optimized inference engine

(An LLM-powered AI Infra optimization toolkit)

---

中文文档

---

What is this? (Our vision)

• Let AI generate simple, compact inference frameworks that match or surpass SOTA performance. (Killer app)

  • Today's SOTA inference frameworks are buckling under their own weight.

    • The traditional approach — multi-layer abstractions to support every model, every hardware target, every parallelism strategy, every quantization scheme, every kernel optimization — no longer keeps pace with the pace of LLM innovation.
    • The vast majority of code in an inference framework is unnecessary once your deployment constraints are fixed, yet that compatibility logic drags down performance.

  • Tell MetaInfer what you need, and it writes a high-performance, purpose-built inference framework for your exact setup.

    • Which model? What hardware? What parallelism? What quantization? Optimize for throughput or latency?

• AI-assisted kernel optimization

  • Given a target kernel, the LLM drives an automated [optimize → benchmark] loop.

• Model porting

  • New model unsupported on older GPUs? Point MetaInfer at the upstream framework code and let it backport to a lower-version target.

• Embedded / edge inference

  • MetaInfer targets C/C++ and Rust as first-class output languages — laying the groundwork for edge computing and embodied AI.

Built-in task types

Type	Description
gen-infer-framework	Build a model-specific inference server with an OpenAI-compatible HTTP API. An immutable oracle boots serve.sh, sends fixed prompts, and dispatches an LLM judge to verdict correctness.
calc-theoretical-value	Compute theoretical FLOPs and memory-traffic for a single forward pass of an LLM. Fully read-only deterministic pipeline: model inspection → memory modeling → compute graph → visualization.
example	Canonical skeleton for new task types. Copy, rename, uncomment register(), implement your pipeline — no shared code touched.

Quick start

Step 1: Install ccb (MetaInfer currently uses the open-source Claude Code CLI; other coding agents are not yet supported — contributions welcome)

Open-source ccb repository: https://github.com/claude-code-best/claude-code

npm i -g claude-code-best

Step 2: Install MetaInfer

git clone https://github.com/MetaInfer/MetaInfer.git
cd MetaInfer
pip install -r requirements.txt
./serve.py

Open http://127.0.0.1:8765, click + New Task, pick a task type, fill in your requirements, and the LLM gets to work.

# Other ways to start
./serve.py --host 0.0.0.0 --port 9000
METAINFER_PORT=9000 ./serve.py
python -m metainfer.server.app

Multi-node setup (optional)

The Quick Start above runs everything on one node. To run GPU workloads on remote worker nodes (multi-task GPU isolation, cross-node PP2/TP debugging), start a worker daemon on each GPU machine:

# On each worker node (must share the same NFS mount as the orchestrator):
pip install -r requirements.txt   # same deps as the orchestrator
npm i -g claude-code-best         # ccb needed for agent-type jobs

# Pick a stable node id (defaults to $METAINFER_NODE_ID or hostname)
METAINFER_ROOT=/shared/metainfer \
METAINFER_NODE_ID=gpu-worker-1 \
python -m metainfer.worker --ip 10.0.0.5

The worker registers itself in cluster/workers/<node_id>.json and starts polling cluster/inbox/<node_id>/ for jobs. It touches cluster/workers/<node_id>.heartbeat every 15s — orchestrators and the WebUI consider it dead if the heartbeat goes >60s stale.

From the WebUI's Cluster tab you can see registered workers, GPU scoreboard (free/held per slot), and force-release any stuck slot. Tasks that accept a worker_nodes form field (e.g. evolve_kernel, port_model) will route GPU work to the listed workers when set.

Admin CLI:

metainfer-cluster workers ls          # list workers + alive status
metainfer-cluster scoreboard show     # show GPU claims across all workers
metainfer-cluster tail stdout gpu-worker-1 <job_id>   # read a job's stdout

See docs/multi-node-architecture.md for the NFS-safe claim algorithm, lease/reaper rules, and the single-reap-path invariant, and docs/agent-sdk-guide.md for the SDK cookbook.

License

MIT

Academic Research

The initial ideas and experimental data of MetaInfer are publicly available at https://arxiv.org/abs/2607.12875. The related code is on the arxiv-paper branch.

Citation:

@misc{miao2026metainferknowledgellminference,
      title={MetaInfer: A Knowledge Only LLM Inference Engine Generator SKILL Toolbox}, 
      author={Zhenwen Miao and Honglin Wang and Mingheng Mi},
      year={2026},
      eprint={2607.12875},
      archivePrefix={arXiv},
      primaryClass={cs.MA},
      url={https://arxiv.org/abs/2607.12875}, 
}

Contributing

See CONTRIBUTING.md for architecture details, design principles, and how to add new task types.