RadReport AI

Upload a chest X-ray → get back an AI-generated radiology report (Findings + Impression), in real time, backed by retrieval-augmented generation over a Postgres vector store.

A full-stack ML application: a fine-tuned vision-language model, a RAG layer over pgvector, live status updates via Supabase Realtime, and a serverless-first deployment that costs $0 to run publicly.

⚠️ For research/educational use only — this is not a diagnostic device and must never be used for real clinical decisions.

Live demo · License · Deployment runbook

---

What this is

A chest X-ray, once uploaded, flows through:

1. Auth + storage — Supabase Auth and a private Storage bucket, gated by Row Level Security so users only ever see their own reports.
2. Retrieval — the image is embedded (BiomedCLIP) and the top-3 most similar reports are pulled from a 6,172-row pgvector index built from the IU-Xray corpus, injected as few-shot context.
3. Generation — a LoRA-fine-tuned vision-language model (BiomedCLIP encoder + BioGPT-Large decoder) produces Findings and an Impression, conditioned on the retrieved context.
4. Live delivery — the result streams back to the browser via Supabase Realtime the instant it's written — no polling, no page refresh.

The public demo runs in stub mode (see Known limitations below) so it's free and always-on; the full real-model pipeline above is implemented and runnable locally or on a GPU host.

Try the demo

1. Open ai-rep-gen.vercel.app.
2. Sign up with any email + password (6+ characters).
3. Upload a chest X-ray image (PNG/JPEG) and click Generate report.
4. Watch the report card flip from pending to done live — Supabase Realtime, no refresh.

Upload something that isn't an X-ray (a photo, a random image) and the app will reject it with a specific message instead of silently generating a report — see app/api/process/route.ts.

Architecture

Next.js Frontend (Vercel)
   -> Supabase client SDK -> Supabase (Postgres+pgvector, Auth, Storage, Realtime)
   -> HTTP call -> Inference Service (FastAPI + Torch + LoRA-fine-tuned VLM + RAG)
      hosted on Render/Fly.io/RunPod (GPU) or run locally, reads/writes Supabase Postgres directly

Piece	Where it runs
Frontend (Next.js)	Vercel
Auth, Postgres + pgvector, Storage, Realtime	Supabase
Inference (FastAPI + LoRA VLM + RAG)	Any GPU host, or local machine (currently not deployed publicly — see below)

Model stack

• Primary architecture (designed, not yet trained): StanfordAIMI/CheXagent-2-3b, a chest-X-ray-pretrained vision-language model, fine-tuned via LoRA (r=16, alpha=32, target q_proj/v_proj).
• Fallback architecture (actually trained and running): BiomedCLIP (ViT-B/16 image encoder) + BioGPT-Large (text decoder), bridged by a trainable linear projection, LoRA-fine-tuned on the decoder. Chosen because it fits on a free-tier GPU (Colab T4) — CheXagent needs ~16–24 GB and a paid GPU tier.
• Training data: 6,172 frontal (image, report) pairs from IU-Xray/Open-I, after filtering and cleaning.
• RAG: BiomedCLIP embeds the query image; Postgres pgvector (<-> cosine distance) returns the top-3 nearest reports from the indexed training set as few-shot context.
• Evaluation: BLEU / ROUGE-L / METEOR / RadGraph-F1 scoring script exists (training/evaluate_model.py) — see limitations below on why no scores are published yet.
• Experiment tracking: MLflow.

The trained adapter is published in this repo at inference/weights/lora-adapter/ — see Known limitations for exactly what it is and isn't.

Known limitations

Being direct about what this project does not yet do, because a portfolio project that hides its gaps is less useful than one that names them clearly:

• The public demo does not run the trained model. Serving it needs a GPU host, which costs money; the live site instead runs a serverless stub that returns a fixed canned report so the upload → Realtime → dashboard flow is demonstrable for free. The real model is fully implemented and can be run locally (see Quickstart) or deployed to any GPU host — flipping one config value (NEXT_PUBLIC_INFERENCE_URL) switches the live site to it.
• The trained model is the lightweight fallback stack, not the primary architecture. BiomedCLIP+BioGPT was chosen specifically because it fits on a free Colab GPU; it is a smaller, less capable combination than a purpose-built chest-X-ray VLM like CheXagent, which the codebase supports but has not been trained (it needs a paid GPU tier).
• Trained for one epoch on 6,172 image-report pairs. This validates that the full training pipeline runs correctly end-to-end; it is not enough data or training time to expect strong report quality. Modern clinical report-generation models train on multiple, much larger datasets for many more epochs.
• No published quantitative evaluation yet. The BLEU/ROUGE-L/METEOR/RadGraph-F1 evaluation script exists and works, but hasn't been run against this checkpoint to produce reported numbers — so there's no benchmarked accuracy claim here, intentionally, rather than an inflated one.
• CPU inference is slow. Without a GPU, generating one report measured ~160 seconds locally. Fine for a demo, not for production latency.
• The non-X-ray image filter is a heuristic, not a classifier. It checks whether an image is grayscale-like (real X-rays are; most photos aren't) — this reliably catches color photos submitted by mistake, but a grayscale photo of something else would still pass. It is not a substitute for real image validation.
• Not clinically validated in any way. No radiologist review, no FDA clearance, no comparison against real diagnostic outcomes. Existing entirely for demonstrating the engineering of the pipeline, not for producing trustworthy medical output.

Repo layout

• frontend/ — Next.js app (Vercel) — auth, upload, dashboard, and the serverless stub route
• inference/ — FastAPI inference microservice (Docker, GPU host) — model loading, RAG, generation
• training/ — data pipeline, LoRA fine-tuning (both architectures), evaluation, embedding indexing
• supabase/migrations/ — SQL schema migrations (Supabase CLI)
• .github/workflows/ — CI/CD for the inference service
• notebooks/ — Colab notebook that runs the full training pipeline on a free GPU
• data/ — IU-Xray dataset (downloaded by the pipeline, not committed)

Dataset usage terms

IU-Xray (Open-I) is provided by the NIH National Library of Medicine for research use. Reports are de-identified. See https://openi.nlm.nih.gov/faq for terms. Do not attempt re-identification. Cite: Demner-Fushman et al., Preparing a collection of radiology examinations for distribution and retrieval, JAMIA 2016.

Quickstart (development)

# training environment
python -m venv .venv && .venv/Scripts/activate
pip install -r training/requirements.txt

# data pipeline
python training/data_pipeline.py

# train (fallback stack — runs on a free Colab GPU; see notebooks/train_on_colab.ipynb)
python training/train_lora.py --fallback --epochs 1

# inference service — real model
pip install -r inference/requirements.txt
# set STUB_MODE=0 and FALLBACK_MODEL=1 in inference/.env (see inference/.env.example)
uvicorn app.main:app --port 8001 --app-dir inference

# frontend, pointed at your local inference service
cd frontend && npm install
# set NEXT_PUBLIC_INFERENCE_URL=http://localhost:8001 in frontend/.env.local
npm run dev

See .env.example files in frontend/ and inference/ for the full configuration reference.

Free public demo (no GPU, always-on)

The frontend can run a self-contained demo with no separate inference host: if NEXT_PUBLIC_INFERENCE_URL is unset, uploads are handled by a built-in serverless route (frontend/app/api/process/route.ts) that validates the image looks like an X-ray, then writes a canned report back to Supabase. The full upload → Realtime → dashboard flow works for free, with nothing running on your own machine.

Deploy the demo on Vercel:

1. Push the repo to GitHub.

2. Import it at vercel.com/new; set Root Directory to frontend.

3. Add environment variables (Settings → Environment Variables):

   • NEXT_PUBLIC_SUPABASE_URL, NEXT_PUBLIC_SUPABASE_ANON_KEY
   • SUPABASE_URL, SUPABASE_SERVICE_ROLE_KEY (server-side; used only by the stub route)
   • leave NEXT_PUBLIC_INFERENCE_URL unset

4. Deploy → you get a public https://<project>.vercel.app URL anyone can use.

5. Configure Supabase Auth redirects (required for signup emails to work): in the Supabase dashboard go to Authentication → URL Configuration and

   • set Site URL to your Vercel URL
   • add http://localhost:3000 to Redirect URLs so local dev keeps working.

   Without this, confirmation emails link users back to localhost:3000, which fails on any machine but your own. (Alternatively, disable email confirmation under Authentication → Sign In / Up → Email for a friction-free demo.)

To switch to the real model, deploy the FastAPI service (inference/) to a GPU host with STUB_MODE=0 and FALLBACK_MODEL=1, then set NEXT_PUBLIC_INFERENCE_URL on Vercel to that host — the frontend bypasses the stub automatically.

License

MIT — see LICENSE.