I wanted to understand what actually happens between "user watches movies" and "you might like this" on a real streaming service, so I built the full retrieval → ranking → serving path myself: a TensorFlow two-tower retrieval model, an MLP ranker, a FAISS approximate-nearest-neighbour index, and a FastAPI service, benchmarked against a matrix-factorisation baseline with an offline A/B framework on MovieLens.
The point of the project was not to win a leaderboard but to feel every stage: why retrieval and ranking are separate models, why a pure cosine two-tower can lose to plain matrix factorisation until you give it a popularity term, how a temporal split changes the numbers, and what the serving path costs in milliseconds once FAISS and a ranker are in the request.
- Two-tower retrieval (Keras): user tower + item tower, cosine score with a learnable per-item bias, trained with sampled softmax.
- Ranking model (Keras MLP): reranks the retrieved candidates using id embeddings + item side-features.
- MF baselines: implicit ALS (the
implicitlibrary) as the anchor, plus a Keras BPR-MF. - FAISS IVF/flat inner-product index for ANN serving of the item embeddings.
- Feature engineering in DuckDB SQL + pandas: streaming aggregations, k-core filtering, the temporal split (window functions), and per-user/per-item features — all from the train partition only.
- Offline A/B framework: 9 variants scored on recall@10 / NDCG@10 / hit-rate@10 with a per-user temporal split.
- FastAPI serving on port 8120, benchmarked for latency through the HTTP endpoint.
MovieLens ml-32m (ratings.csv, movies.csv)
│
┌───────────────▼────────────────┐
│ DuckDB SQL + pandas features │
│ filter · k-core · id remap · │
│ temporal split · user/item aggs│
└───────────────┬────────────────┘
train interactions │ + item/user features
┌──────────────────────────┼───────────────────────────┐
▼ ▼ ▼
┌───────────────────┐ ┌────────────────────┐ ┌──────────────────┐
│ MF baseline │ │ Two-tower retrieval│ │ Ranker (MLP) │
│ ALS / BPR-MF │ │ user & item towers │ │ rerank top-N │
│ (anchor) │ │ cosine + item bias │ │ binary relevance│
└─────────┬─────────┘ └──────────┬─────────┘ └────────┬─────────┘
│ user/item embeddings │ user/item embeddings │
└──────────────┬─────────────┘ │
▼ │
┌───────────────────┐ │
│ Offline A/B eval │ recall@10 / NDCG@10 / hit │
│ temporal split │ (9 variants, same harness) │
└───────────────────┘ │
│ best two-tower │
▼ │
┌───────────────────┐ │
│ FAISS ANN index │◄─────────────────────────────┘
└─────────┬─────────┘
▼
┌───────────────────────────────────────┐
request ──────►│ FastAPI :8120 │──────► top-k titles
{user_id,k} │ embed lookup → FAISS top-N → mask seen │ + scores
│ → ranker rerank → top-k │
└───────────────────────────────────────┘
Retrieval narrows ~13k–90k movies to a few hundred candidates in one ANN lookup; the ranker then does the expensive, feature-rich scoring on just those candidates. That two-stage split is the whole reason a streaming service can answer in milliseconds.
Scale actually trained/measured: MovieLens ml-32m, 15% user sample = 2,143,244 positive interactions (rating ≥ 4.0), 29,841 users, 12,697 movies, per-user temporal split, 29,840 warm validation users. Committed configs run the full 32M; this scale was chosen to fit a shared 4-vCPU CPU-only box. Every number below is raw output of a command shown in BENCHMARKS.md.
Headline (recall@10 / NDCG@10 / hit-rate@10, 4,000 warm validation users, full-catalogue scoring):
| Model | recall@10 | NDCG@10 | hit-rate@10 |
|---|---|---|---|
MF-ALS baseline (implicit) |
0.1053 | 0.0848 | 0.3578 |
| Best two-tower, 8-epoch sweep (d=128 +genre) | 0.0995 | 0.0774 | 0.3465 |
| Best two-tower, longer-trained (d=64 +genre, 22 ep) | 0.1056 | 0.0802 | 0.3588 |
Measured delta vs the MF baseline: recall@10 +0.3%, hit-rate@10 +0.3%, NDCG@10 −5.4% — parity, not the +20% target. That is the honest result at this scale and I report it as such: on id-only MovieLens a two-tower is structurally close to matrix factorisation, so a well-tuned ALS is a strong baseline. The item-bias fix (which rescued the two-tower from −50%) and genre features are the real deltas; user-side content features are the documented next lever to actually beat ALS. Full analysis in BENCHMARKS.md.
Serving latency through the FastAPI endpoint (single query, warm): p50 8.63 ms / p99 10.23 ms with the ranker, p99 2.07 ms retrieval-only — under the 30 ms target. Full 9-variant table, commands, and raw output: BENCHMARKS.md.
# 1. env (kept outside the repo on a shared box)
python -m venv /path/to/.venv && source /path/to/.venv/bin/activate
pip install -r requirements.txt
cp .env.example .env # edit DATA_DIR / MODEL_DIR / SERVING_PORT
# 2. get MovieLens ml-32m
curl -o $DATA_DIR/ml-32m.zip https://files.grouplens.org/datasets/movielens/ml-32m.zip
unzip $DATA_DIR/ml-32m.zip -d $DATA_DIR
# 3. build features (DuckDB SQL + pandas)
python scripts/prepare.py --config configs/sample.yaml
# 4. run the A/B sweep (trains every variant, writes the serving bundle)
python scripts/run_ab.py --config configs/sample.yaml --write-serving
# 5. serve + benchmark latency THROUGH the endpoint
uvicorn cinerank.serving:app --port 8120 &
python scripts/bench_latency.py --port 8120 --out latency.jsonUse configs/full_32m.yaml in steps 3–4 to train on all 32M ratings.
pytest runs the ranking-metric unit tests, the DuckDB feature pipeline on tiny
synthetic data, the FAISS index test, and the serving HTTP contract — no
TensorFlow, no training, so CI stays fast (.github/workflows/ci.yml).
cinerank/ config, DuckDB+pandas features, models, eval, FAISS index, FastAPI app
scripts/ prepare.py · run_ab.py · bench_latency.py
configs/ sample.yaml (benchmarked scale) · full_32m.yaml (full dataset)
tests/ metrics · pipeline · index · serving (fast, no TF)
MIT — see LICENSE.