FIPS: expand testing + prove runtime behaviors - #1376
Merged
Merged
Conversation
The outbound-chain test waited for the eager GET common stream to show up at the mock, but rmcp spawns that GET into a JoinSet and aborts the set when the session tears down. In a serve -> tool-call -> drop flow the GET can therefore be aborted before it ever reaches the wire, so the wait was never a latency problem and no budget could fix it. A 5s poll and then a 30s one both still flaked, on plain runs as well as under coverage. The DELETE cleanup already proves the same property. rmcp awaits it on the shutdown path so it is always sent, and GET and DELETE reach the wire through the same staging, which means routing through the outbound chain is the same code for either. So assert on the DELETE alone, move the GET's session-id check down to a transport unit test where it is deterministic, and have the remaining wait report what the mock actually recorded instead of only that it expired. The mock still serves the GET stream, so the transport's GET success arm is still exercised end to end. Signed-off-by: Shane Utt <shaneutt@linux.com>
A standard build registers the policy engine and the response store, whose dependencies (jsonwebtoken on aws-lc-rs, sqlx's sha2 and SCRAM) do their own cryptography outside the system OpenSSL, yet it started happily under PRAXIS_REQUIRE_FIPS as long as the host was in FIPS mode. The check now covers the binary itself: a registered non-FIPS filter refuses startup (and --validate) by name, so only the FIPS build can honor the variable. Checked against the registry, not the config, so a hot reload cannot add the filter later; praxis core has made the same refusal since its FIPS work. Signed-off-by: Shane Utt <shaneutt@linux.com>
The integration and schema suites now compile with exactly the feature set the FIPS binary ships (make test-integration-fips, make test-schema-fips): the proxy runs in-process, so the test binary's dependency graph is the FIPS build's graph. Getting there removed every second cryptography stack from that graph: rustls, tokio-rustls and reqwest lose their aws-lc-rs activators (rmcp forwards the same choice, so even full builds carry no bundled provider), digest oracles moved to the system OpenSSL, and tests of filters the FIPS build leaves out are gated on their features, including fourteen agentic-loop unit tests that drove the loop with an MCP call and so were asserting openai-mcp-tools behavior in every build. New FIPS behavior tests assert both branches on whether the installed provider offers approved algorithms only: what a listener negotiates and refuses (raw hello probes), what the upstream client offers and insists on (a rogue peer records the offer), what a FIPS deployment rejects (short RSA key, SHA-1 chain, ChaCha20-only config), and what the binary does under PRAXIS_REQUIRE_FIPS. PRAXIS_FIPS_HOST=1 makes the approved branch mandatory and the harness fail closed, so a green run on the FIPS runner cannot have happened on OpenSSL's default provider. The listener tests honor PRAXIS_FIPS_PROBE_ADDR, so the same probes drive the shipped image. Signed-off-by: Shane Utt <shaneutt@linux.com>
cargo xtask fips host-check states the facts the module's Security Policy requires of a host (the kernel flag, the boot parameter, the crypto policy, fips-mode-setup, the module OpenSSL loads) and of the FIPS image (the policy podman propagates into it, the build of fips.so it carries), grades that build against a compiled-in list of validated and in-validation module builds with sources, and writes the attestation for CI to keep. The pinned UBI 9 images carry a build still in validation; the check says so on every run rather than letting the package name imply a certificate, and --require-certified turns it into a failure once that is the policy. cargo xtask fips runtime-probe runs the shipped image under PRAXIS_REQUIRE_FIPS=1 with a generated listener, waits for a real TLS handshake, drives the listener probes of the integration suite against it, and checks the startup status line: the check against the bits that ship. Signed-off-by: Shane Utt <shaneutt@linux.com>
The FIPS workflow gets a second job on the self-hosted RHEL 9 runner in FIPS mode (labels fips and rhel). It tests the exact image the hosted job built and scanned, handed over as an artifact and checked by id: the host and the image's module build are attested, the suites run as the FIPS build inside the new toolchain stage of Containerfile.fips on that host (failing closed unless the container is in FIPS mode, with the provider assertions armed), and the image is run under PRAXIS_REQUIRE_FIPS=1 and probed from outside. The release workflow attests and probes the pushed -fips image, pulled back by digest, before the release is cut; the release gate accepts a skipped run, so a missing digest never wedges a release, and a runner outage surfaces as a queued job a maintainer can act on. The job never runs fork code and stays out of the merge queue, so queue throughput never depends on the single runner. Signed-off-by: Shane Utt <shaneutt@linux.com>
The operator guide gains the FIPS-host verification targets, the refusal of non-FIPS filters under PRAXIS_REQUIRE_FIPS, and a plain statement of what is validated and what is not: the module build the pinned images carry is in validation, the certificate lists tested operating environments, and rustls over the validated provider is an architecture a compliance program judges, not a fact this repository can assert. The developer guide documents the new xtask commands, the FIPS host run and the runner job. Signed-off-by: Shane Utt <shaneutt@linux.com>
Signed-off-by: Shane Utt <shaneutt@linux.com>
nerdalert
requested changes
Sep 25, 2026
nerdalert
left a comment
nerdalert
left a comment
Member
There was a problem hiding this comment.
Thanks for adding the FIPS-host runtime checks. I found two release-workflow issues worth addressing before relying on them as release evidence:
-
An offline runner leaves the release waiting, rather than taking the documented fallback. In
.github/workflows/release.yaml,fips-hostis scheduled on[self-hosted, fips, rhel]andgithub-releaseneeds its result. If no matching runner is online, GitHub keeps the job queued; it does not mark itskipped, so theneeds.fips-host.result == 'skipped'allowance cannot help. GitHub documents a queued-job timeout of up to 24 hours. Suggested fix: choose the intended policy explicitly. If host qualification is required, let an unavailable runner block and report that requirement clearly. If it is optional, run it independently ofgithub-release.needsand publish its result separately. Please do not rely onskippedto represent runner unavailability. -
The release does not establish that the tagged image passed the full FIPS-host suites. The FIPS workflow runs on main/PR/schedule/manual events, not release tags, while the release job invokes
fips-hostwithrun-suites: "false". The comment that the suites already ran for this commit is not enforced, and a green release could therefore lack that evidence for its exact source and image. Suggested fix: run the existing suites for the release tag (the simplest, slower choice), or make the release verify a successful FIPS-host suite result tied to the tagged commit and the exact image digest before claiming runtime-suite qualification.
I did not run the FIPS-host suite locally; these findings are based on the workflow wiring and GitHub's self-hosted runner behavior.
Signed-off-by: Shane Utt <shaneutt@linux.com>
Signed-off-by: Shane Utt <shaneutt@linux.com>
Signed-off-by: Shane Utt <shaneutt@linux.com>
…ference Signed-off-by: Shane Utt <shaneutt@linux.com>
Signed-off-by: Shane Utt <shaneutt@linux.com>
shaneutt
marked this pull request as ready for review
September 25, 2026 18:30
nerdalert
approved these changes
Sep 25, 2026
|
testing locally on rhel vm |
|
test verified locally |
hexfusion
approved these changes
Sep 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a follow-up to #1325 which expands the tooling and testing for FIPS-mode Praxis. The previous patch tested that the binary was structurally FIPS, this patch runs the entire unit and integration test suite on a FIPS-enabled RHEL host to prove the runtime behaviors.
This is the reciprocal of praxis-proxy/praxis#1263.