Create link health check script

[AJaccP]

🧠 Context

Every URL we ingest lives in one of two curated files: data/webpages/list.json (the real set of links to ingest content) and data/webpages/test_list.json (a small subset used by tests, ~12 URLs). Both are flat JSON arrays of URL strings. When one of these pages moves or is taken down, our answers silently lose a source, and nobody notices until a student gets a worse answer.

Ingestion already logs a failure (with the HTTP status) when it can't fetch a page, but that only helps while ingestion is actually running, and only for someone reading the logs. A dedicated CI check adds two things: it runs on a schedule, so we catch link rot in between ingests, and it runs on any PR that edits the lists, so a newly-added dead link is caught before it merges.

This is a self-contained job: a script that reads both lists, checks each URL, and reports the ones that are broken.

---

🛠 Implementation Plan

1. A checker script — scripts/check_links.py. Read both data/webpages/list.json and data/webpages/test_list.json, request every URL, and classify the result. The project already uses httpx, so use its async client with a small concurrency cap and follow redirects — see the httpx async docs.

2. Classify results into three buckets, not two. A naive "any non-200 is broken" check produces false alarms and gets ignored. Distinguish:

   • Broken (fail the job): definitive dead links — 404, 410, connection refused etc.
   • Unverifiable (warn, do not fail): transient or bot-blocking responses — timeouts, 429, 5xx, and 403. Carleton pages can potentially block unfamiliar clients, so 403 is not proof the page is gone. Report these separately so a flaky network doesn't turn the whole job red.
   • OK: any 2xx (after redirects).

3. Reduce false positives. Send a real browser-like User-Agent (some servers reject the default and return 403), and retry the transient/unverifiable cases a couple of times with a short backoff before reporting them. Keep concurrency modest.

4. Report clearly. Print a summary grouping URLs by bucket, and write the same summary to the job summary ($GITHUB_STEP_SUMMARY) so it's visible without opening logs. Exit non-zero only when the Broken bucket is non-empty.

5. A workflow — .github/workflows/link-check.yml. Three triggers (see the events docs):

   • schedule — a weekly cron (e.g. Monday morning UTC).
   • pull_request with a paths: filter on the two JSON files, so it only runs on PRs that actually change a list.
   • workflow_dispatch — so it can be run on demand.

   The job checks out the repo, sets up uv (mirror the existing ci.yml lint job), and runs the script.

6. Send a Discord alert on scheduled failure — only on scheduled runs. When a scheduled run finds broken links, post a short message to a Discord channel via an incoming webhook: just a heads-up that broken links were found plus a link back to the workflow run — the full list lives in the job summary. Gate this step on failure() && github.event_name == 'schedule' so it does not fire on PR runs (the failed check is already visible there) or manual dispatch. Store the webhook URL as a repository secret (e.g. DISCORD_WEBHOOK_URL) and read it from the environment instead of hardcoding it.

---

🔑 The webhook url secret

You do not need to create the real repository secret — Jacc will add DISCORD_WEBHOOK_URL in the repo settings while reviewing/merging. Write the workflow assuming that secret will exist: reference secrets.DISCORD_WEBHOOK_URL as shown. This should be safe because the alert step only runs on scheduled runs, so your PR never touches the secret, and an undefined secret resolves to an empty string rather than a build error — nothing breaks in the meantime.

You can use your own webhook url to test locally during development.

---

✅ Acceptance Criteria

• scripts/check_links.py reads both list.json and test_list.json, checks every URL, and classifies each as OK / broken / unverifiable, exiting non-zero only when something is broken.
• Broken vs unverifiable is distinguished as described (404/410/DNS = broken; timeout/429/5xx/403 = unverifiable), with a real User-Agent, redirect following, and a small retry on transient failures.
• A summary of broken and unverifiable URLs is written to the job summary.
• .github/workflows/link-check.yml runs weekly, on PRs that change either list (via a paths: filter), and on manual dispatch.
• On a scheduled run that finds broken links, a Discord webhook alert is sent with the broken URLs and a link to the run; the alert does not fire on PR or manually-dispatched runs.
• The Discord webhook URL comes from a repository secret, not hardcoded.
• make lint passes.