Skip to content

feat(api/dashboard): per-project uptime monitors with incident alerts#207

Open
Ahmedk-n wants to merge 1 commit into
oblien:mainfrom
Ahmedk-n:feat/uptime-monitors
Open

feat(api/dashboard): per-project uptime monitors with incident alerts#207
Ahmedk-n wants to merge 1 commit into
oblien:mainfrom
Ahmedk-n:feat/uptime-monitors

Conversation

@Ahmedk-n

@Ahmedk-n Ahmedk-n commented Jul 24, 2026

Copy link
Copy Markdown

Problem

OpenShip deploys and manages apps, but has no way to tell you one of them went down, the only signal is your users noticing. There is no uptime checking, no incident history, and nothing that feeds the notification system when an app stops responding. The dashboard even ships a MonitoringTab component, but it was never wired into the project tab list.

Solution

Per-project HTTP uptime monitors, built on the platform's existing machinery:

  • packages/db: new monitor, monitor_check, and monitor_incident tables (generated migration 0057; additive-safe, passes migrations-additive.test.ts). Check history is pruned after 7 days so it can't grow unbounded.
  • apps/api/lib/monitor-runner.ts: a check runner using the same singleton-interval pattern as the notification runner (15s tick, in-flight guard so a slow probe can never overlap the next tick and double-alert). Probes due monitors (GET, per-monitor timeout, success = expected status or < 400), tracks consecutive failures against a per-monitor threshold, opens/resolves incidents, and emits monitor.down / monitor.recovered through the existing notification dispatcher — alerts reach email, webhook, Slack, Discord, Microsoft Teams, and the in-app bell with zero channel changes. Recovery events carry the downtime as durationMs.
  • apps/api/modules/monitors: CRUD + checks + incidents endpoints nested under projects (project:read/project:write, localOnly — same pattern and scope as route-rules). 24h uptime is computed from check history. Kept self-hosted-only deliberately: user-defined probe URLs in multi-tenant cloud mode would be an SSRF surface; cloud support is future work.
  • apps/dashboard: wires in the previously-unwired Monitoring tab and adds an Uptime monitors section — monitor list with status/uptime/response time, add/enable/disable/delete, expandable response-time chart (same hand-rolled SVG style as TrafficChart), and incident history. Strings localized across all nine locales.
  • apps/web: docs page for monitors, plus the two new notification categories in the notifications doc.

Future work: public status pages, TCP/keyword checks, multi-region probes, per-monitor alert routing, cloud-mode support.

Verification

  • bun run --cwd apps/api linttsc --noEmit passes.
  • bun run --cwd apps/api test — 64 test files / 755 tests pass (3 skipped), including new e2e CRUD/ownership tests (real PGlite + real auth via the jobs-harness pattern) and unit + integration tests for the runner state machine.
  • bunx tsc --noEmit in apps/dashboard passes; dashboard tests pass.
  • Live end-to-end on a dev stack: created a monitor against a local app through the dashboard; killed the app → monitor flipped to down within the interval, an incident opened with the real connection error, and an HMAC-signed monitor.down webhook was delivered to a test receiver; restarted the app → monitor.recovered arrived carrying durationMs matching the actual 43s outage, and the incident resolved. Dashboard verified showing status, uptime %, response-time chart, and incident history throughout.
  • One note: because notification channels currently can't be verified (Notification channels created in the dashboard can never be verified, so they never deliver #179), the delivery leg of the E2E required flipping the test channel's verified flag in the dev DB. Once fix(api/dashboard): add notification channel test/verify flow #189 lands, monitors compose with it cleanly.

Related

- New monitor/monitor_check/monitor_incident tables with generated migration; checks pruned after 7 days.

- A monitor runner (same singleton-interval pattern as the notification runner) probes due monitors, tracks consecutive failures against a per-monitor threshold, opens/resolves incidents, and emits monitor.down / monitor.recovered through the existing notification dispatcher — so alerts reach email, webhook, Slack, Discord, Microsoft Teams, and the in-app bell with no channel changes.

- Monitors CRUD nested under projects (project:read/write, localOnly) following the route-rules pattern, with 24h uptime computed from check history.

- Wires in the dashboard's previously unwired Monitoring tab and adds an uptime section: monitor list with status and uptime, add/enable/disable/delete, response-time chart, incident history. Strings localized across all nine locales.

- API docs page for monitors plus the two new notification categories.
@Hydralerne

Copy link
Copy Markdown
Member

can you please share screen shot of that module in ui

@Ahmedk-n

Copy link
Copy Markdown
Author

can you please share screen shot of that module in ui

  • doing sm checks and testing rn and I will change the pr status afterwards
image image

@Ahmedk-n
Ahmedk-n marked this pull request as ready for review July 24, 2026 14:45
@Hydralerne

Copy link
Copy Markdown
Member

Really clean and good to have. However, monitoring is definitely a feature I was already planning to build. Our architecture is monitoring-first thanks to OpenResty.

I actually plan to make it an Analytics tab rather than just a monitoring tab. It would include uptime, visits per country, live hits on a world map (we already support the backend for this in the Server Logs tab), and much more - including monitoring itself.

If you're open to working on it, I can prepare a technical brief with the right scope and architecture so we build it the way we want.

But anyway, amazing work.

@Ahmedk-n

Copy link
Copy Markdown
Author

Awesome, this sounds great.
It honestly makes sense to use the bigger Analytics tab rather than just monitoring especially with OpenResty features since they are way more solid than what I was picturing. Send over the technical brief whenever it's ready, happy to work on it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants