MiniTorch

My Detailed Notes: minitorch | 十派的玩具箱

A compact deep learning systems project based on the MiniTorch teaching framework. This repository focuses on implementing core autodiff and tensor operators, then using them to train CNN models for sentiment classification and image classification.

Features

• Implemented MiniTorch core components such as autodiff, tensor operations, module/parameter management, and fast operators.
• Implemented neural network operators including Conv1D, Conv2D, pooling, dropout, and logsoftmax.
• Ran end-to-end training on:
  • SST-2 sentiment classification
  • MNIST digit classification
• Backends: pure-Python SimpleOps, a dependency-free NumpyOps, and optional Numba FastOps / CudaOps, and an optional PyTorch-backed TorchOps (CPU/CUDA). Training scripts let you switch backends with --backend.
• Dependencies are managed with uv; training is one command via python -m minitorch <task>.

Structure

minitorch/
|- minitorch/                  # framework core (autodiff, tensor ops, backends)
|  |- numpy_ops.py             # pure-NumPy backend (no compiler needed)
|  |- fast_ops.py / cuda_ops.py# optional Numba JIT / CUDA backends
|  |- data.py                  # dependency-free MNIST / GloVe loaders
|  |- cli.py / __main__.py     # `python -m minitorch <task>` entry point
|- scripts/                    # training scripts (one per method)
|  |- train_mnist_minitorch.py # MiniTorch + NumpyOps/FastOps/CUDA
|  |- train_mnist_torch.py     # PyTorch reference
|  |- train_sentiment_minitorch.py # MiniTorch + GloVe CNN
|  |- train_sentiment_hf.py    # Hugging Face Trainer reference
|- project/                    # original MiniTorch demo scripts + Streamlit app
|- tests/
|- pyproject.toml / uv.lock    # uv-managed dependencies

Setup

Dependencies are managed with uv. The core framework only needs NumPy; everything else is an optional extra.

# core framework (NumPy backend, runs on any modern Python)
uv sync

# add the training demos (datasets + PyTorch reference scripts)
uv sync --extra demo

# add the Hugging Face Trainer sentiment script
uv sync --extra modern

# add the Numba JIT backends (FastOps / CudaOps), on supported CPython
uv sync --extra fast

# everything (demos + modern + fast + viz + tests)
uv sync --extra all

Prefer the existing Conda env? pip install -e ".[demo,fast,test]" works too. Run commands through uv with uv run python ..., or activate the venv that uv sync creates under .venv/.

Linux / WSL note for the CUDA/Triton backend

On WSL2, a stale system libcuda can shadow the WSL GPU driver and make PyTorch report an "NVIDIA driver too old" error. Point the loader at the WSL driver library first:

export LD_LIBRARY_PATH=/usr/lib/wsl/lib:$LD_LIBRARY_PATH
uv sync --extra demo --extra fast --extra bench
uv run python benchmarks/benchmark.py   # exercises Triton on the GPU

Run commands through uv with uv run python ..., or activate the venv that uv sync creates under .venv/.

Data

Data loading is self-contained — no extra packages are required.

• MNIST (minitorch.load_mnist): reads the IDX files from project/data/ and auto-downloads them from the canonical MNIST mirrors on first use, so you usually do not need to download anything by hand. To pre-populate the files:

  mkdir -p project/data
  cd project/data
  wget -c https://storage.googleapis.com/cvdf-datasets/mnist/train-images-idx3-ubyte.gz
  wget -c https://storage.googleapis.com/cvdf-datasets/mnist/train-labels-idx1-ubyte.gz
  gunzip -kf train-images-idx3-ubyte.gz train-labels-idx1-ubyte.gz
  cd ../..

• SST-2 is loaded via datasets (the demo/modern extra). Point HF_HOME at a shared cache if you like: export HF_HOME=project/data/hf_cache.
• GloVe (minitorch.load_glove): used by the MiniTorch sentiment script; it downloads glove.6B.zip from Stanford NLP on first use and caches the extracted vectors under project/data/. The Hugging Face sentiment script does not use GloVe.

Run

The easiest way is the unified CLI, which dispatches to the right script:

# MiniTorch MNIST (NumPy backend by default; --backend fast uses Numba)
uv run python -m minitorch mnist-minitorch --backend numpy --epochs 5

# PyTorch reference on MNIST
uv run python -m minitorch mnist-torch --epochs 5

# MiniTorch SST-2 sentiment (Kim CNN + GloVe)
uv run python -m minitorch sentiment-minitorch --epochs 100

# Hugging Face Trainer sentiment (DistilBERT)
uv run python -m minitorch sentiment-hf --max-train-samples 2000

mnist and sentiment are aliases for the MiniTorch variants. You can also call any script directly, e.g. uv run python scripts/train_mnist_minitorch.py --help for all flags (backend, batch size, learning rate, data ranges, output dir, ...). Each MiniTorch training run writes metrics.json under its output directory.

Original demo scripts

The original MiniTorch scripts are kept under project/:

python project/run_mnist_multiclass.py
python project/run_sentiment.py

Visualization

streamlit run project/app.py

Benchmarks

benchmarks/benchmark.py times matrix multiply, elementwise/reduce kernels, 2D convolution, and a full MNIST training step across the available MiniTorch backends and against PyTorch, then writes charts and a Markdown report:

uv sync --extra demo --extra bench
uv run python benchmarks/benchmark.py --out-dir docs/benchmarks

The report (docs/benchmarks/benchmark_report.md) includes a deep-dive on matrix multiply: keeping operands resident on the GPU (fp32, via the tiled Triton matmul in minitorch/torch_ops.py on Linux+CUDA+Triton, or cuBLAS otherwise) is ~20-60x faster than the NumPy CPU backend for large matrices. A torch / cuda-torch backend is also selectable with --backend torch / --backend cuda-torch in the MiniTorch training scripts.

Resume Summary

Built and extended a MiniTorch-based deep learning systems project, implementing autodiff, tensor operators, and CNN modules, then validating the framework on SST-2 sentiment classification and MNIST image classification.