Skip to content

Add reproducible PAI public data pipeline - #1

Draft
amit-spatial wants to merge 1 commit into
mainfrom
agent/pai-public-dataset
Draft

Add reproducible PAI public data pipeline#1
amit-spatial wants to merge 1 commit into
mainfrom
agent/pai-public-dataset

Conversation

@amit-spatial

Copy link
Copy Markdown
Owner

What changed

  • adds a browser-free, resumable downloader for FY 2022–23 and FY 2023–24 GP-level PAI scores
  • validates portal hierarchy parity and preserves unpublished, malformed, and external-LGD exception records separately
  • adds official grade metadata, GP peer ranks, administrative summaries, tests, and reproducible analysis commands
  • adds concise scraping, handoff, CoRE Stack, and professor-facing documentation

Why

The public portal does not provide one complete national download, and state-wide pagination can lose rows at tied-score boundaries. The block-level pipeline validates every GP ID against the portal roster before accepting a partition.

Data and privacy

Raw datasets, scrape partitions, source PDFs/spreadsheets, prompts, environment files, and local history remain ignored and are not included in this PR.

Checks

  • uvx ruff format --check src tests main.py
  • uvx ruff check src tests main.py
  • uv run python -m unittest discover -s tests -v — 13 tests passed
  • staged privacy scan — clean

@amit-spatial
amit-spatial force-pushed the agent/pai-public-dataset branch from 99e7e68 to 9a93b87 Compare July 30, 2026 09:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant