Panchayat Advancement Index public-data downloader

This project downloads the public Gram Panchayat score table from the PAI portal without running a browser or loading JavaScript, images, fonts, analytics, or translation tools.

The downloader first reads the portal's public, year-specific district, block, and GP JSON handlers. It then requests the legacy score table one block at a time, following ASP.NET ViewState pagination only when a block exceeds 100 GPs. Several blocks run concurrently.

Every block is finalized only when every handler GP ID is accounted for as either a published score row, an explicit block-level no-data response, or an individually checked “score card not available” response for an ID missing from a partially published block. This matters because state-wide pagination is not stable when many GPs share the same score: rows can overlap at a 100-row page boundary and silently displace other rows.

The live portal currently exposes 34 State/UT options for financial-year ID 2 (shown by the portal as 2023-2024).

Quick start

uv sync
uv run pai-scrape list-states
uv run pai-scrape --year-id 2 scrape --workers 8
uv run pai-scrape --year-id 2 combine
uv run pai-scrape --year-id 2 derive
uv run pai-scrape --year-id 2 audit-reference

uv run pai-scrape --year-id 1 scrape --workers 8
uv run pai-scrape --year-id 1 combine
uv run pai-scrape --year-id 1 derive
uv run pai-scrape --year-id 1 audit-reference

Omitting --states selects all states. A state may be selected by its LGD code or exact name. The national command writes block partitions under data/raw/fy_2/blocks/, the combined dataset to data/raw/fy_2/blocks/pai_scores.csv, and the year-specific hierarchy roster to data/catalog/fy_2/gp_catalog.csv. A full combine also writes unpublished_gps.csv, which identifies roster GPs for which the portal confirms that no score is available at block or GP level, and state_summary.csv, which reports scored/unpublished counts and block coverage for each State/UT. data/ is deliberately gitignored because downloaded data is a reproducible artifact, not source code.

derive writes a ready-to-analyse layer under data/derived/fy_<year-id>/: official grades for the overall and nine theme scores, GP peer ranks and percentiles, and block/district/state summaries. audit-reference writes row-level reconciliation flags and state/district/block summaries under data/audit/fy_<year-id>/. Portal records that do not match the supplied external LGD snapshot are retained and flagged.

Use --force to replace completed block files. Without it, a rerun skips a completed block only when its metadata and handler-roster fingerprint still match. Multi-page blocks also have page-level resume checkpoints.

combine requires the complete live state set, verifies every block checksum, published row count, and handler-roster count, rejects nationally duplicated GP IDs, revalidates score ranges, and writes pai_scores.csv plus pai_scores.json. For a deliberate subset, use combine --allow-partial.

Output contract

Each state_<state code>/district_<district code>/block_<block code>.csv contains:

financial_year_id
pai_version
state_lgd_code
state_name
district_lgd_code
district_name
block_lgd_code
block_name
gp_lgd_code
gp_name
overall_pai_score
t1_score
...
t9_score

Portal scores are percentages in the range 0–100, not fractions in the range 0–1. Values are preserved exactly as published rather than rounded again.

The row's scorecard URL contains base64-encoded financial-year, state, district, block, and GP IDs. The downloader decodes those IDs and cross-checks the visible bracketed GP code. It does not infer identifiers from names.

Grades can be derived without scraping extra columns: A+ is 90–100, A is 75–below 90, B is 60–below 75, C is 40–below 60, and D is below 40. The canonical raw output retains the published numeric values and keeps derived labels out of the source table.

Tracking and validation

Every completed block has a block_<code>.json sidecar containing:

• row count;
• server-page count;
• state, district, and block LGD IDs and names;
• scored and catalog GP counts plus fingerprints of both ID sets;
• unavailable GP IDs, the block-level no-data flag, and any page-boundary duplicate IDs;
• SHA-256 checksum of the CSV;
• exact source URL and financial-year ID;
• elapsed time and UTC completion time.

manifest.jsonl records each completed, skipped, or failed block as the run progresses. CSV files are first written with a .part suffix and atomically renamed only after validation, so an interrupted response is never mistaken for a complete block. After every non-terminal 100-row page, a block_<code>.checkpoint.json stores the updated ViewState, next page number, completed row count, handler-roster fingerprint, and page fingerprints. A rerun truncates any post-checkpoint bytes and resumes at the next server page.

The downloader rejects a block result when:

• the 11 live source headers differ from the observed contract;
• the scorecard IDs cannot be decoded or disagree with the visible GP ID;
• a row has the wrong number of cells or a blank hierarchy value;
• any of the ten scores is not numeric or is outside 0–100;
• a row belongs to a different year, state, district, or block;
• a returned GP ID is outside the year-specific handler roster;
• any handler GP ID is neither recovered from the table/scorecard nor explicitly confirmed unavailable;
• a GP LGD code repeats within one page; or
• a server page repeats an earlier page.

For a full combine, the command also requires exact parity with gp_catalog.csv: every catalog block must have a completion sidecar, every catalog GP ID must be accounted for exactly once as scored or unpublished, and each GP's state/district/block assignment must agree. This live invariant is stronger than relying on a national count alone.

The current live FY2 handlers expose 266,999 GP roster entries. The official edition describes 259,867 submitted GP/equivalent records. The difference is represented explicitly rather than discarded: the main dataset contains published score rows, while unpublished_gps.csv records handler IDs whose block table or individual scorecard endpoint reports that no score is available.

In this snapshot, all 7,132 unavailable IDs are concentrated in West Bengal (3,339; no published block scores), Meghalaya (3,790), and Goa (3). These are explicit portal availability results, not scraper failures.

Why this is fast

The browser page is an ASP.NET form. Its dependent geography selectors use public JSON handlers such as:

/Handlers/Y_Lgd_Districts.ashx?SID=<state>&YID=<year>
/Handlers/Y_LGD_Blocks.ashx?SID=<state>&ZID=<district>&YID=<year>
/Handlers/Y_GPs_By_LGD_Block.ashx?SID=<state>&ZID=<district>&BID=<block>&YID=<year>

The handlers establish the expected set before each block is accepted. The ID-bearing TW-GP.aspx response publishes at most 100 records at a time, and each Next request uses only the updated ViewState and hidden hierarchy fields. Only handler IDs absent from the terminal block table are checked individually against SC.aspx. If the block table explicitly reports no data for the whole request, that response accounts for the block without redundant GP requests. The fast block table remains the primary path. No server CSV/XLSX endpoint was found: the apparent Excel button is a browser-side DataTables export of only the currently loaded page.

The supplied LGD boundary Parquets are retained for optional geometry enrichment and QA, not used as the source of truth for PAI parentage. The panchayat geometry release contains missing, repeated, and conflicting GP records, while the portal score row directly supplies its own hierarchy IDs.

Project handoff

• reports/PAI_QUALITATIVE_REPORT.md is the professor-facing qualitative interpretation and one-page CoRE Stack use note.
• reports/PROFESSOR_BRIEF.md is a concise five-minute speaking brief.
• metadata/ contains official themes and grade bands, the data dictionary, and the ranking/aggregation methodology.
• docs/SCRAPING_AND_VALIDATION.md is the separate concise technical extraction note.
• HANDOFF.md records final artifacts, validation evidence, known issues and reproducible next steps.

The default concurrency is eight block/handler requests to keep the run quick without creating an unnecessarily aggressive load on the public server. Transient 408, 425, 429, and 5xx responses are retried with backoff and Retry-After support.

Development checks

uv run python -m unittest discover -s tests -v
uvx ruff check src tests main.py
uv run pai-scrape scrape --states 35 --workers 4 --output data/test_run --force
uv run pai-scrape combine --input data/test_run --allow-partial

The unit suite tests the ASP.NET form, current download-table parser, ID decoder, hierarchy parser, combiner, and the exploratory newer-page parser. The live smoke command is currently expected to reconcile 7 blocks and 70 GPs for Andaman and Nicobar Islands; treat those counts as dated observations, not permanent hard-coded invariants. A separate live multi-page test on Maharashtra block 4566 reconciled all 238 handler IDs over pages of 100, 100, and 38 rows.

The portal is the source and should be credited when the resulting dataset is republished. This repository does not invent a licence for the source data; verify the Government of India's current reuse terms for the intended release.