Skip to content

fix: bound container resources and harden failure cleanup - #103

Open
keethliu wants to merge 1 commit into
LilithGames:mainfrom
keethliu:feature/incident-hardening-20260908
Open

fix: bound container resources and harden failure cleanup#103
keethliu wants to merge 1 commit into
LilithGames:mainfrom
keethliu:feature/incident-hardening-20260908

Conversation

@keethliu

@keethliu keethliu commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Summary

Heavy Agent tasks currently share an unbounded application container, status probes can leave descendants behind after a timeout, and a failed terminal transaction can leave a Run failed while its step still says running.

  • Apply configurable defaults of 2 CPUs, 3 GiB RAM, 3.5 GiB RAM plus swap, and 512 PIDs to repository Compose, new CLI-generated installs, and remote deployment. Validate resource settings locally and with a stopped-container Docker preflight before replacing the live container.
  • Give status probes the shared process-tree termination path, including delayed SIGKILL after the leader exits, and bound each output stream to 1 MiB. Discard incomplete output on overflow.
  • Recover the Run and its running step in one transaction. If recovery also fails, roll back both rows so startup recovery can still find them; preserve existing terminal states.

Update operator documentation and the in-app FAQ. Full E2E validation also exposed existing assertions that had drifted from login redirects and list rendering; align those assertions without changing the corresponding product behavior.

Type of change

  • Bug fix

Testing checklist

  • pnpm lint: 0 errors, 751 existing warnings (752 on the base branch; no new warnings).
  • pnpm typecheck: API, CLI, web, and shared packages pass.
  • pnpm test: 10,030 passed, 5 existing platform/conditional skips. Regression tests were observed failing before the production fixes.
  • pnpm test:e2e: 191 passed, 16 existing conditional skips; no failures.
  • pnpm test:e2e:recovery: all 4 restart-recovery scenarios pass.
  • pnpm test:e2e:onboarding: fresh clone, installation, first startup/setup/login, and shutdown pass.
  • Real Linux process smoke: a timed-out probe also terminates a SIGTERM-resistant descendant after the leader exits.
  • Repository commit gates pass; remote-deployment tests cover invalid settings and failed preflight without removing the existing service.
  • Configuration docs, product constraints, and the existing Chinese manual updated; no new UI translation keys or navigation changes.

AI assistance disclosure

This change was developed and reviewed with Codex assistance. The automated checks above were executed against the changed source.

Upgrade notes

The limits cover the entire application container, including all Agents and tool children. Heavy workloads may need larger operator-selected budgets; database sidecars and other services need their own budgets.

Existing CLI installs preserve their Compose file during image upgrades and must add the documented resource fields once. Drain active tasks, validate Compose configuration, and recreate the container to apply limits. Pulling an image or restarting the API alone does not apply Docker resource settings. Native/PM2 deployments require equivalent systemd/cgroup limits.

No database schema migration is required.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant