OpenGrad

Automated pipeline that keeps U.S. master's program information fresh: polite scraping → LLM structured extraction → per-field accuracy evals → change monitoring → searchable site.

Why: applicants overpay agencies for information that is public but scattered and stale. Community sites (e.g. OpenCSApp) solved openness but rely on manual upkeep; OpenGrad automates the upkeep.

• Design doc: docs/SCOPE.md
• Status: pilot phase — 10 schools, CS vertical

Fetch snapshots

Install the two runtime dependencies, then run the fetcher from the repository root:

python3 -m pip install -r requirements.txt
python3 scripts/fetch_snapshots.py

The fetcher reads data/programs_seed.yaml and writes raw HTML plus metadata to snapshots/YYYY-MM-DD/<school_slug>/<program_slug>.{html,meta.json}. Its report is snapshots/YYYY-MM-DD/fetch_report.json. Requests identify OpenGrad, obey robots.txt, wait at least three seconds between requests to the same domain, and never attempt to bypass anti-bot challenges. Same-day terminal outcomes are cached, so rerunning makes no network requests and reports every record as skipped. In the report, success/failure counts describe the complete dated cache; skipped additionally says how many records reused that cache in the current run, so it intentionally overlaps the outcome counts. Delete that dated record's .meta.json only when an intentional retry is needed.

Monitor changes

Preview the current refresh without spending LLM tokens:

python3 scripts/monitor.py --dry-run

Run the refresh and incrementally re-extract only changed projects:

read -s "DEEPSEEK_API_KEY?DeepSeek API key: "
echo
export DEEPSEEK_API_KEY
python3 scripts/monitor.py
unset DEEPSEEK_API_KEY

Both commands politely refresh known main and aux pages under snapshots/YYYY-MM-DD/ and compare cleaned page text with the latest historical snapshot. The full run carries unchanged extraction results forward, writes changed results to data/extracted/YYYY-MM-DD/, and produces the human-readable weekly summary plus machine-readable details at data/diffs/YYYY-MM-DD.{md,json}. The diff reports page buckets, field changes, and DeepSeek API cost. Extraction uses DeepSeek V4 Pro through its native JSON output API. --dry-run reports the buckets with zero LLM calls and writes data/diffs/YYYY-MM-DD.dry-run.{md,json}, so a preview cannot overwrite that day's completed report.