Migrate from cron-job.org to self-hosted Grafana

[aditeyabaral]

📌 Feature Summary

We should deploy a new endpoint on a new render called pesu-auth-dash so we can visualize data returned from the metrics.

💡 Motivation

Migrating to Grafana gives us control over the type of health checks we perform, and also lets us visualize the metrics that will be returned from /metrics in #129

🧩 Suggested Implementation

No response

🧠 Estimated Complexity

Medium (Changes multiple files, some model updates)

🔗 Related Issues / PRs

• Follow up to Metric Logging #129

🧠 Additional Notes

No response

[aditeyabaral]

Closing this as done, though not in the shape the issue describes.

What shipped: a Grafana Cloud dashboard reading /metrics?fmt=json directly through the Infinity datasource — uptime, request and authentication counts, status codes, request outcomes and upstream outcomes across 14 panels. No exporter, no scrape job, no new service: Grafana fetches the JSON view of the endpoint that shipped in #157 as v4.2.0.

What did not happen, and why — recorded so it is not re-litigated

Self-hosted Grafana beside the app is not viable on the current hosting. Grafana's own documented minimum is 512 MB of memory and 1 CPU core. A Render free instance is 512 MB and 0.1 CPU, shared with the API — so Grafana alone wants the whole instance's memory and ten times its CPU, before Prometheus or the API itself. Free services also get no persistent disk, and the filesystem is wiped on every deploy, restart and spin-down, so Prometheus's TSDB and Grafana's SQLite would reset constantly — which destroys the history a dashboard exists to show. Worst of all the failure mode is inverted: an OOM in the monitoring stack would take down the API it is monitoring.

A separate pesu-auth-dash service does not get around that. The workspace has 750 instance-hours per calendar month against a ~730-hour month, already spent keeping the API awake — and exhausting it suspends every free service in the workspace, production included.

cron-job.org stays. The four checks — Docs, Health, Authentication, Authentication with Profile Data — still run, and the status page linked in the README is unchanged. Migrating them would move the test account's credentials into a third-party check config for very little gain.

What the dashboard cannot show, and what each would take

• No history, rates or alerting. Infinity fetches live and stores nothing, so every figure is cumulative-since-restart and the time picker has no effect. Getting real time series means writing the values into a TSDB first. The cheapest route is Grafana Cloud's agentless Metrics Endpoint integration, which scrapes any public HTTPS Prometheus URL from Grafana's side every 60s with no infrastructure to run — /metrics already serves the Prometheus exposition format by default. It requires the endpoint to be behind Basic or Bearer auth, which the API does not support yet.
• No percentiles. The collector's summaries expose _sum and _count only, so every latency figure is a mean, and a mean hides the tail users feel. p95 needs histogram buckets in app/metrics/collector.py.
• Production is still on v4.1.0 and returns 404 on /metrics, so the dashboard currently reads staging.

Two things the metrics surfaced almost immediately

• profileParseErrors: {"page_structure": 1} on staging, with a matching 422 in responsesByStatus — PESU Academy served a profile page the parser could not read. Worth noting that a dashboard enumerating status codes by hand cannot show this, because that object only carries keys for codes that have actually occurred.
• The CSRF prefetch cache is running at a 0.67 hit ratio, so one authentication in three still pays a ~0.43 s token fetch inline. That accounts for 61% of the 0.22 s of overhead the API adds to a login — the other 87% of a login is time spent waiting on PESU Academy, which is now measurable for the first time.