scripts : add Strix host-memory watchdog

[ajaxdude]

Overview

Add an independent external Linux host-memory watchdog for the DeepSeek V4.1 Strix Halo validation stack.

• launch a supplied command in its own guarded process group
• read meminfo and swaps from a configurable procfs root, default /proc
• reject startup when /proc/swaps contains any configured swap entry
• calculate host-wide used memory as MemTotal - MemAvailable using integer bytes
• send SIGTERM at the default 116 GiB soft threshold
• send SIGKILL at the default 118 GiB emergency threshold or after the bounded grace period
• fail closed if swap appears or procfs becomes invalid during execution
• forward wrapper SIGHUP, SIGINT, and SIGTERM, terminate remaining descendants, reap the child, emit an authoritative final record, and return shell signal convention
• publish optional v2 lease/heartbeat evidence and exclusive persistent JSONL audit records
• use an internal guardian with a private pulse pipe and Linux parent-death handling so wrapper failure or stall cannot leave the process group unmonitored
• preserve the primary safety classification while recording artifact persistence failures in structured secondary_errors

The 118 GiB emergency threshold leaves a documented 2 GiB sampling margin below the strict 120 GiB ceiling. Grace is capped at 30 seconds, sampling at one second, and heartbeat age at five seconds.

This is the independent watchdog slice from halo-box#48. It is stacked on the corrected halo-box#49 branch head 65fdc19831d30b12f06b981c3ca74dd18731f90e. After ggml-org#49 merges, this PR will be recreated or retargeted to halo-box/strix-llama.cpp with the merged base.

Frozen candidate: bbf182a48123f64c65d56ba48c0fc0c2b1d375f7.

This head retains independently approved watchdog ancestry 778db6f50eae04e6c232c69b9575bdbd0747962b, normally merges the corrected ggml-org#49 base, and makes real-process test fixtures wait for complete parseable PID/JSON state instead of racing file creation. It fixes later exact-head review findings: lease validation closes its held pidfd on every success or failure path, parent-signal graceful cleanup refreshes the payload-visible heartbeat while it continues private guardian pulses, and cleanup rechecks the deadline after each synchronous heartbeat attempt so delayed success or failure cannot cause a negative sleep or bypass SIGKILL/reap. Production script SHA-256: 98313ee0edabe42455f103f68bcbc6cba0e8e117afae8db658e17a18b87038f9.

It deliberately does not implement model admission, the DeepSeek V4.1 graph, Engram, expert caching, or model execution. No model/GGUF was loaded, no inference or model server was started, and no Strix runtime or ROCm package was changed.

Validation

Model-free validation for exact head bbf182a48123f64c65d56ba48c0fc0c2b1d375f7:

• macOS python3 -m unittest tests.test_strix_memory_watchdog - 37 discovered, 29 passed, 8 Linux-only skipped
• macOS python3 -W error::ResourceWarning -m unittest tests.test_strix_memory_watchdog - 37 discovered, 29 passed, 8 Linux-only skipped
• Linux Debian container normal suite, repeated twice - 37/37 each run
• Linux Debian container with ResourceWarning promoted to error, repeated twice - 37/37 each run
• registered Linux CTest: ctest --test-dir /tmp/watchdog-build -R '^test-strix-memory-watchdog$' --output-on-failure - 1/1 passed
• corrected-base schema: PYTHONPATH=gguf-py python3 -m unittest gguf-py/tests/test_deepseek41_schema.py - 4/4 passed
• python3 -m py_compile scripts/strix_memory_watchdog.py tests/test_strix_memory_watchdog.py - passed
• flake8 7.3.0 with flake8-no-print 0.1.1 on the watchdog script and tests - passed
• ty 0.0.78 on the watchdog script and tests - passed
• git diff --check, ASCII-only scan, and leftover-process inspection - passed

The deterministic suite covers parsing, swap gates, thresholds, grace and emergency escalation, malformed/missing procfs, launch failures, child exit propagation, wrapper signals, real child/grandchild cleanup, guardian failure/stall, live launch identity, artifact failure precedence, all closed/blocked guardian-control FD x TERM/KILL/final/lease failure combinations, and the documented setsid() boundary limitation. The corrective regressions run a real guarded payload whose SIGTERM handler completes after heartbeat max age but before grace, a guarded unresponsive payload that remains alive until grace escalation, repeated successful/rejected lease validation while asserting every injected pidfd is closed and the process descriptor set is unchanged, and successful plus ArtifactError heartbeat writes that cross the grace deadline while proving TERM, SIGKILL, reap, and authoritative final classification still execute without a negative sleep.

Independent exact-SHA review approved bbf182a48123f64c65d56ba48c0fc0c2b1d375f7 with no remaining blockers. The reviewer rechecked all nine earlier lifecycle/evidence areas, the complete eight-cell closed/blocked guardian-control FD x TERM/KILL/final/lease matrix, pidfd closure, guarded long-handler and unresponsive-child grace behavior, and both heartbeat deadline-overrun cases. Earlier restacked heads 061389280e8df1926f89d51596caeceb41dc2720 and c3ddb694db9f94d0108cb937071af07f6daefadc remain rejected and must not be used.

Hosted Server CI passed on Ubuntu and Windows and the labeler passed for exact head bbf182a48123f64c65d56ba48c0fc0c2b1d375f7. Hosted flake8 and Python type-check jobs remain queued because this fork has no [self-hosted, fast] runner; equivalent targeted local checks pass. No runner was installed or registered.

Requirements

• I have read and agree with the contributing guidelines.
• This change is Strix Halo specific and implements the independent watchdog requested by Feature Request: support DeepSeek V4.1 Flash with bounded SSD expert caching halo-box/strix-llama.cpp#48.
• AI usage disclosure: agent-authored by GPT-5.6 Sol.
• Not verified: model-backed execution, published-GGUF loading, inference, ROCm execution, or swap reconfiguration were intentionally not performed.

[ajaxdude]

Hardware-use freeze: independent review found four high-severity fail-closed defects in watchdog head 59833018814c0883f995848912cd5a52889c5303: launch/artifact identity can be self-forged; hard watchdog death can leave the protected process group alive; synchronous artifact fsync occurs before emergency enforcement; and audit failure can abort cleanup escalation/reaping. This head is superseded for Strix validation. A corrected replacement with focused negative/lifecycle tests is in progress. Do not use it for the authorized swapoff run.

Written by GPT-5.6 Sol.

[ajaxdude]

STOP - superseded for hardware use: 59833018814c0883f995848912cd5a52889c5303 has four open fail-closed lifecycle/authentication findings and must not be used for Strix validation. A corrected replacement will be pushed and independently re-reviewed before consumption. Do not merge or run the current head on hardware.

[ajaxdude]

Corrected replacement pushed at c4598ee747fdb811b474c0b41bdb38eeda3bbc0c; 59833018814c0883f995848912cd5a52889c5303 remains superseded. Mapping: (1) strict /proc reconstruction now requires the exact Python executable, canonical script at argv[1], complete options/command, stable pid/start ticks, watchdog->guardian->payload topology, O_NOFOLLOW 0600 owner/inode checks, and the watchdog-held audit lock; negative tests cover helper/inert args, -c, -m, interpreter options, command/policy/env path, inode, start tick, arbitrary heartbeat, stale heartbeat, and outside-PGID cases. (2) an internal guardian owns the process group and private CLOEXEC nonblocking pulse pipe, with PDEATHSIG and parent-race checks; Linux tests cover watchdog SIGKILL, SIGSTOP, pipe EOF, control-FD noninheritance, forged file heartbeat, child/grandchild death, and the documented deliberate-setsid() limit. (3) swap/emergency/soft enforcement and private pulse now occur before audit/heartbeat writes; regression proves SIGKILL precedes simulated blocked artifact persistence. (4) graceful and emergency cleanup defer ArtifactError until TERM/KILL/wait/reap complete; regression forces persistent write failure and verifies TERM->KILL plus reaping. Validation: Linux 28/28 normal and ResourceWarning; macOS 25 passed/3 Linux-only skipped in both modes; py_compile, flake8+no-print, ty, diff, and ASCII passed. This head is pending independent re-review and is not yet authorized for hardware use.

[ajaxdude]

Independent exact-SHA rereview of c4598ee747fdb811b474c0b41bdb38eeda3bbc0c closed prior HIGH findings 1-3, but hardware use remains blocked. Prior HIGH #4 is still open: after audit emit() fails, disable_component() can raise again while closing the persistent stream, escaping cleanup before SIGKILL escalation and reap; the reviewer reproduced a surviving child. Two additional corrections are required: keep guardian pulses active throughout intentional graceful cleanup so the pulse timeout does not silently shorten the configured grace period, and version the incompatible guardian lease/heartbeat topology as v2 with an atomic consumer update. A further pushed SHA and fresh independent review are required.

Written by GPT-5.6 Sol.

[ajaxdude]

Replacement candidate: 0071e8f21701b74b3cc4dfe5c1131950ce338143

This supersedes c4598ee747fdb811b474c0b41bdb38eeda3bbc0c and remains blocked from hardware use pending independent exact-SHA closure.

Finding -> invariant -> regression mapping:

1. HIGH forged lease identity -> validation reconstructs the exact executable, watchdog argv position/options, PID/start ticks, command, artifact identities, audit lock, and watchdog -> guardian -> payload topology from live procfs -> test_active_lease_validation_rejects_tamper_and_stale.
2. HIGH wrapper loss/stall leaves payload -> private noninherited control pipe, PDEATHSIG parent-race checks, guardian group kill, and payload-side stale-evidence guard -> test_guardian_pipe_close_kills_group_without_fd_leak, test_guard_kills_group_after_watchdog_loss_or_stall, and test_payload_guard_fails_closed_on_artifact_error.
3. HIGH artifact I/O before enforcement -> swap/emergency/soft signals and private pulse occur before audit/heartbeat persistence -> test_emergency_signal_precedes_failing_audit.
4. HIGH artifact failure/close aborts cleanup -> audit sink is cleared before nonthrowing close; write, flush, close, and stderr failures are secondary to unconditional TERM -> bounded wait -> SIGKILL -> reap -> test_artifact_failure_does_not_skip_cleanup, test_stderr_failure_does_not_bypass_cleanup, and test_audit_write_and_close_failures_do_not_bypass_cleanup.
5. MEDIUM guardian pulse timeout truncates intentional grace -> graceful signal sends a private grace-mode control record, cleanup continues pulses, and the guardian fallback deadline is one second beyond wrapper grace -> test_parent_signal_grace_outlives_guardian_pulse_timeout and test_parent_signal_allows_exit_after_pulse_deadline.
6. MEDIUM incompatible topology under v1 -> lease and heartbeat are now strict version 2, bind grace/sample/heartbeat timing to the reconstructed live command, and reject v1 -> test_active_lease_validation_rejects_tamper_and_stale plus test_configuration_rejects_weakened_liveness_timing.

Additional lifecycle coverage confirms signal-mask restoration, delayed SIGTERM handler exit without escalation, direct-leader descendant cleanup, soft-limit descendant grace_timeout, payload signal-status preservation, SIGHUP/SIGINT/SIGTERM forwarding, private-FD noninheritance, and documented setsid() scope.

Validation:

• Linux Python 3.11 container: 35/35 normal; 35/35 with ResourceWarning as errors.
• macOS Python 3.14: 28 passed, 7 Linux-only skipped, normal and with ResourceWarning as errors.
• py_compile, flake8 7.3 + flake8-no-print, ty check, git diff --check, and ASCII scan passed.
• No Strix host, model, ROCm, swap, or remote workload was accessed.

Hosted CI is running for this exact SHA. Do not consume this candidate until independent review closes all six findings.

[ajaxdude]

Independent exact-SHA review of 0071e8f21701b74b3cc4dfe5c1131950ce338143 closes all six previously recorded findings, but hardware use remains blocked by a new HIGH lifecycle defect. During intentional cleanup, begin_grace() or a cleanup pulse can raise ProcessGroupError; the current outer handler returns signal_error without guaranteed SIGKILL escalation and reap. The reviewer reproduced a closed control-FD case where only SIGTERM was sent and the child remained alive. The next correction must force immediate process-group kill and bounded reap on both closed and blocked guardian-control failures, with exact regressions and another fresh independent review.

Written by GPT-5.6 Sol.

[ajaxdude]

New exact review candidate: cd97b50b664bd170819b4225ae52cad97d762ada

0071e8f21701b74b3cc4dfe5c1131950ce338143 is superseded and must not be used for hardware validation. Independent review closed its prior six findings, then found one new HIGH: GuardianProcess.begin_grace() or a cleanup pulse could raise ProcessGroupError and return signal_error before killing/reaping the live process group.

Fix invariant:

• the graceful signal audit is emitted immediately after TERM/INT/HUP forwarding
• a failed grace-control write or cleanup pulse is recorded as the primary guardian-control error
• cleanup immediately calls the independent process-group signal path with SIGKILL
• it then performs the existing bounded wait/reap before finalization
• process-group kill failure remains a distinct signal_error; artifact errors remain secondary
• the authoritative final record is signal_error/exit 7 with the guardian-control cause and the reaped child return code

Regression: test_guardian_control_failure_still_kills_and_reaps_group uses real child/grandchild process groups for both a closed control FD and a nonblocking full/blocked control FD. Both cases assert SIGTERM then fallback SIGKILL, bounded leader reap with -SIGKILL, final signal_error, and no surviving child or grandchild.

Full validation for this exact tree:

• Linux Python 3.11 container: 36/36 normal; 36/36 with ResourceWarning as errors.
• macOS Python 3.14: 28 passed, 8 Linux-only skipped, normal and with ResourceWarning as errors.
• py_compile, flake8 7.3 + flake8-no-print, ty check, git diff --check, and ASCII scan passed.
• No Strix host, model, ROCm, swap, or remote workload was accessed.

Hosted CI and independent exact-SHA review are pending. No hardware or integration authorization is granted.

[ajaxdude]

Independent exact-SHA review of cd97b50b664bd170819b4225ae52cad97d762ada confirms all previous six findings remain closed and guardian-control termination/reaping is fixed. One medium correctness blocker remains: after a guardian-control failure establishes authoritative signal_error/exit 7, a final audit or lease-finalization failure can overwrite it with lease_error/exit 8. The reviewer reproduced this combined failure while confirming the child was killed and reaped. The next correction must preserve the primary guardian-control classification and record final artifact failure only as secondary metadata, with combined closed/blocked-control-FD plus final-audit/lease-failure regressions. Hardware use remains blocked.

Written by GPT-5.6 Sol.