Macrodata Refinement

A CRT terminal that makes you sort "scary" numbers into bins, inspired by the "macrodata refinement" work of Lumon Industries in Apple TV's Severance. Underneath the terminal is a real text-analytics tool: feed it a batch of text records and it will embed them, project them to 2D, cluster them, and label each cluster with an LLM. Sorting the numbers on screen is, literally, sorting your data into its real clusters.

This project has no affiliation with Apple or Severance. It's just a homage built for fun and to learn a few things.

Why this even exists

Severance frames its central mystery around meaningless-looking work: Lumon's refiners stare at a grid of numbers that "feel" a certain way and sort them into bins, with zero visibility into why, or what any of it means. That's a surprisingly good metaphor for a lot of real data work, and a fun excuse to build something.

What it does

The terminal has two modes, chosen entirely by whether you paste/upload data on the login screen.

Classic mode (no data entered)

A decorative version of the MDR minigame. Synthetic "scary" clusters spawn on the grid at all times; dragging a selection box over one sweeps it into one of five generic bins. Filling all five bins to 100% ends the session ("Cold Harbor") and logs a score to the local leaderboard. Nothing here talks to the backend.

Data mode (real records entered)

1. On the login screen, give your employee name plus records: paste text (one per line, or a JSON array), upload a .txt/.csv/.json file, or load the built-in sample dataset. Provide the access key. (kier)
2. The frontend validates the batch (at least 10 records, and no more than the grid can physically display) and POSTs it to /analyze.
3. The backend runs the batch through an async pipeline: embed with Voyage, project to 2D with UMAP, cluster with KMeans, persist the rows, then label each cluster with Claude (name, one-sentence summary, sentiment split).
4. Then, the grid switches to data mode: each record becomes a cell on the lattice. Records "activate" (start shaking) a few at a time; only active records can be swept. One bin exists per real cluster until you drag every one of its records in, at which point it reveals the cluster's real label and summary.
5. Sorting every record unlocks the Refinement Report: per-cluster label/summary/sentiment breakdown and CSV export (per cluster, or the whole labeled dataset).
6. Progress is checkpointed to sessionStorage, so a reload mid-session resumes instead of dropping you back to a blank login screen losing your progress.

Architecture

index.html + css/style.css   static shell, CRT overlay, all screens/modals
js/app.js                    hash routing (#/login #/mdr #/leaderboard), menu, session glue
js/bins.js                   the five (or N, in data mode) storage bins
js/grid.js                   the canvas: cell lattice, trembling animation, sweep/refine logic
js/refine.js                 data intake, backend calls, polling, report/export

backend/main.py              FastAPI app: CORS, routers, serves the frontend same-origin
backend/db.py                SQLite schema (rows, clusters) + connection helper
backend/jobs.py              background job orchestration (in-memory status dict)
backend/routers/analyze.py   POST /analyze
backend/routers/results.py   GET /results/{job_id}
backend/routers/export.py    GET /export/{job_id}/{cluster_id}
backend/services/embeddings.py     Voyage AI embeddings (batched, retried)
backend/services/reduce_cluster.py UMAP -> 2D, KMeans clustering
backend/services/labeling.py       Claude cluster labeling (label/summary/sentiment)

There's no build step and no framework: the frontend is four plain <script> files sharing state through global objects (App, Bins, Grid, Refine) and custom DOM events (screenchange, mdrprogress, filecomplete, clusterdetail). The backend is plain FastAPI with hand-written SQL over aiosqlite, no ORM.

The analysis pipeline, step by step

POST /analyze
  -> validate batch, gate on X-Access-Key, kick off a background task
  -> return { job_id }

background job (jobs.run_analysis_job):
  1. embed every text with Voyage (voyage-4), batched, retried on 5xx/timeout
  2. reduce those embeddings to 2D coordinates with UMAP
  3. cluster the (full-dimensional) embeddings with KMeans
  4. persist every row (id, text, x, y, cluster_id) to SQLite
  5. for each cluster, sample up to 15 texts and ask Claude for a
     {label, summary, sentiment_pos/neu/neg} JSON object
  6. persist cluster metadata to SQLite
  7. mark the job done (or error, with the exception message attached)

GET /results/{job_id}   -> status while pending; full rows + clusters once done
GET /export/.../{id}    -> one cluster's rows as a CSV download

Job status lives in memory and resets on a backend restart, but finished results are already in SQLite, so /results falls back to the database if the in-memory status is gone.

Tech stack

Frontend: vanilla HTML/CSS/JS, <canvas> for the number grid, no build tooling, no dependencies.

Backend: Python, FastAPI, aiosqlite (SQLite), numpy, scikit-learn (KMeans), umap-learn, voyageai (embeddings), anthropic (cluster labeling).

Deployment: Docker, deployed to Fly.io (fly.toml), SQLite on a persistent volume, scales to zero when idle.

Project layout

index.html, css/style.css, js/*.js   frontend (served statically by the backend)
backend/main.py                      FastAPI app entrypoint
backend/db.py, jobs.py               storage + job orchestration
backend/routers/                     HTTP endpoints
backend/services/                    embeddings / reduction+clustering / labeling
Dockerfile, .dockerignore, fly.toml  container + Fly.io deploy config
launch.json                          VS Code debug config (runs uvicorn)

Getting started

Classic mode only (no backend needed)

The frontend has no build step. Serve the repo root with any static file server and open it:

python -m http.server 4173
# visit http://localhost:4173

Log in with just a name and leave the data field empty. This gets you the decorative game with no external calls.

Full data mode (backend required)

Requires Python 3.11+, a Voyage AI API key, and an Anthropic API key.

cd backend
python -m venv .venv
.venv/Scripts/activate        # .venv/bin/activate on macOS/Linux
pip install -r requirements.txt

cp .env.example .env          # then fill in the values below
uvicorn main:app --app-dir backend --port 8000
# visit http://localhost:8000

backend/.env:

Variable	Purpose
VOYAGE_API_KEY	embeddings
ANTHROPIC_API_KEY	cluster labeling
ACCESS_KEY	value clients must send as X-Access-Key to call /analyze
ALLOWED_ORIGINS	comma-separated CORS origins
DB_PATH	optional; defaults to backend/data.db
VOYAGE_MODEL	optional; defaults to voyage-4
CLAUDE_MODEL	optional; defaults to claude-sonnet-4-6

The access key is a light gate against bots burning API credits, not real authentication, by design (see Known limitations below).

API reference

Method	Path	Purpose
POST	/analyze	submit a batch (max 2000 rows), start a job
GET	/results/{job_id}	job status, plus rows/clusters when done
GET	/export/{job_id}/{cluster_id}	one cluster's rows as CSV
GET	/health	liveness check
GET	/	the frontend (index.html)

Deployment

fly launch --copy-config --no-deploy
fly volumes create data --size 1
fly secrets set VOYAGE_API_KEY=... ANTHROPIC_API_KEY=... ACCESS_KEY=... ALLOWED_ORIGINS=...
fly deploy

The Docker image installs backend/requirements.txt and serves index.html/js//css/ same-origin from FastAPI, so there's only one service to deploy. SQLite lives on a Fly volume mounted at /app/data; the machine scales to zero when idle.

Known limitations

These are deliberate, documented trade-offs for a project at this scale, not oversights, but worth knowing if you're building on top of this:

• The access key is not real auth. It's an hmac.compare_digest check against one shared secret, meant only to keep casual bots from burning API credits. There's no per-user identity anywhere in the backend.
• Job status is in-memory and resets on a backend restart. Finished results survive in SQLite; an in-flight job does not.
• The leaderboard is local only. Scores live in localStorage on the device that played them; there's no shared/global leaderboard.
• Outliers aren't actually detected yet. The frontend has full plumbing for flagging anomalous records (styling, report section, tooltip tag), but Refine.mapResults() currently hardcodes every point's isOutlier to false since the backend doesn't compute it. This is the most obviously unfinished wire in the codebase.
• No automated test suite. Correctness has been checked ad hoc (syntax checks, a manual SQLite round-trip, etc.) but nothing is committed as a repeatable test.

Credits

"Severance" and all associated names (Lumon Industries, Kier Eagan, Macrodata Refinement, Cold Harbor) belong to their creators and Apple TV. This project is an unaffiliated fan work, built for the fun of it and to learn.