[Nexthop] run_test.py: fail a test when a unit it left running crashes

[hillol-nexthop]

Summary

Every run_test.py runner leaves some production systemd units running while its test binary executes (sai_test keeps fsdb and the platform services up, sai_agent/link keep qsfp_service up, bsp_tests keeps both agents up while unloading kmods, the fboss2 CLI suite runs against the agents themselves), and all of those units run with Restart=always (RestartSec=30). A crash the test caused is therefore invisible to gtest: systemd brings the unit back, the binary's own readiness wait rides through it, and the test reports OK while a core sits on disk and the hardware may never have received the config. The motivating case was a CLI test whose commit crashed the hw_agent on every run and still passed 4/4 times, one hw_agent core each. gtest cannot see that, so the harness has to.

TestRunner._run_test_guarded() now wraps every _run_test call (cold- and warm-boot phases, and PlatformServicesTestRunner's own loop) and, once the binary exits, looks for a crash inside the window. When it finds one, _apply_agent_crash() downgrades every OK/SKIPPED result of that test to FAILED (so the console/CSV/JSON summary counts it) and appends a banner naming the reason to the binary's transcript.

Two signals, both read after the binary exits, in the new fboss_test_runner/crash_detection.py:

• A unit's main process died uncleanly inside the window, taken from PID 1's own journal records (journalctl -o json --since=@<start> _PID=1, find_unclean_unit_exits): code=dumped (any signal), code=killed by anything but SIGTERM/SIGINT/SIGHUP, a non-zero exit status (other than the 128+N a wrapper relays for those signals), or an OOM kill. Clean stops and restarts — the systemctl restart the fboss2 CLI performs on commit, a warm-boot phase bouncing the test-owned OSS unit, a console logout bouncing serial-getty — exit 0 or die by SIGTERM and do not register. Units the runner masked are inactive and produce nothing, so no per-runner watch list is needed.
• A core file that was not there before the binary started in /var/lib/systemd/coredump or /var/core (list_core_dumps before and after, set difference), from any process except the test binary itself (that test already failed on its own). Comparing sets rather than mtimes charges one core to exactly one test, even when systemd-coredump finishes writing it after the check or the next test starts a second later.

Why the journal rather than systemctl show: NRestarts only moves for Restart=-triggered restarts (a manual restart followed by a crash reads "unchanged"), and Result is reset to success the moment the unit is started again — by the time the check runs after the binary exits, a crash 30 s earlier has already been papered over. The journal line keeps the exit code and signal, and its microsecond timestamp decides membership in the window, so a cold boot the runner performed right before launching the binary is not charged to the test. When journalctl is unavailable (workstation, container) that signal is simply empty; the core check still runs.

Known limitations: a crash that lands between tests (after one binary exits, before the next starts) is attributed to no test. Any unit on the box is watched, so a unit crash-looping for reasons unrelated to the test (a mis-configured platform daemon, say) fails every test in the run — that is a true statement about the device, but it is not attributed to a cause. An agent that aborts on every graceful stop would flag every committing CLI test. The production agents failing to come back after the post-run enable_services restart is outside every window and is a follow-up.

Test Plan

• python3 -m pytest fboss/oss/scripts/run_scripts/fboss_test_runner/unittests → 338 passed on top of current main (323 existing + 15 new: crash_detection ×7 — crash/clean-stop/OOM classification incl. serial-getty exit 0 and wrapper status 143, record-timestamp window, unparseable lines, journalctl missing, core listing, comm truncation; _run_test_guarded ×6 — unit crash, new core, pre-existing core ignored, own-binary core ignored, both, healthy window, downgrade rules keep the transcript; end-to-end through _run_tests ×2).
• ruff check + format clean.
• Journal record shape verified on a lab switch (read-only): a real fboss_sw_agent abort is logged by PID 1 as fboss_sw_agent.service: Main process exited, code=dumped, status=6/ABRT with EXIT_CODE=dumped, and --since=@<epoch> is accepted. _classify_unit_exit reports it.
• Python-only change to the on-device test harness; the cmake/getdeps build is not affected.