fix(cli): stream decrypt and inspect instead of buffering by dmihalcik-virtru · Pull Request #3939 · opentdf/platform

dmihalcik-virtru · 2026-09-01T02:57:22Z

Part 10 of 20 in the DSPX-2604 re-cut. Base branch: dspx-2604-09-stream-encrypt.

This stack replaces #3782 / #3865 / #3921, which stay open and untouched
until it lands. Nothing here is a rebase of those branches — the work was
re-cut from the ticket so each PR stands on its own.

Proposed Changes

otdfctl decrypt read the whole TDF into memory, handed the slice to
DecryptBytes, which accumulated the whole plaintext in a bytes.Buffer, and then
-- for stdout -- called Buffer.String(), allocating a third full copy. Peak RSS
was roughly 3.6x the payload; a 1 GiB file cost ~3.7 GiB of RAM and a large
enough file simply OOMed on a machine with plenty of disk for it.

The plaintext now streams from the SDK reader to the destination. Handler.Decrypt
takes an io.ReadSeeker and an io.Writer, with DecryptOptions replacing the
positional parameter list, and inspect reaches the manifest through the same
seekable reader rather than buffering the archive to get at its tail.

io.Copy is what does the streaming, and it does so only because sdk.Reader
implements WriteTo, which decrypts one segment at a time. Its Read delegates to
ReadAt, which grows an internal bytes.Buffer holding every segment decrypted so
far -- so dropping WriteTo would silently restore the old memory profile with no
test failure to show for it. A compile-time assertion pins the interface.

Removes MaxFileSize. The 10 GB cap existed to bound RAM; the real limit is the
SDK maxFileSizeSupported at 64 GiB, which enforces itself.

Output to a file is atomic, as on the encrypt side: the plaintext goes to a
temporary sibling and is renamed into place only on success. Since
cli.ExitWithError calls os.Exit and skips deferred functions, the spooled input
and the partial output are discarded explicitly on every exit path -- including
inspect's success path, which exits through ExitWithJSON.

e2e coverage lands in a new otdfctl/e2e/streaming.bats rather than in
encrypt-decrypt.bats, which carries a file-level skip pending the
namespaced-subject-mappings migration and would have swallowed the new cases
without running them. Nothing in the new file needs an entitlement, so it needs
no policy fixtures: the round-trips use no attributes, and the two failure cases
are forced with an unresolvable attribute FQN and a KAS allowlist that excludes
the platform. As of this change it is the only e2e coverage of encrypt, decrypt
and inspect that actually executes in CI.

The file is tagged payload_streaming and action.yaml gives it its own pass
ahead of the parallel batch. That ordering is load-bearing, not tidiness. An
encrypt with no attributes falls back to the platform base key, and
key-base.bats sets one pointing at https://test-kas-for-base-keys.com, which
does not resolve. It cannot put things back afterwards: a base key can be
replaced but never cleared, so every unattributed encrypt scheduled after that
file yields a TDF nothing can decrypt. Under --jobs 4 the file order is
nondeterministic, so overlapping the two made this suite flaky rather than
merely broken -- which is how it presented, a different subset of round-trips
failing per run. Running alone also keeps the 1 GiB peak-RSS case from
measuring itself against three neighbours competing for the same memory.

That leak is worth closing on its own -- encrypt-decrypt.bats walks into it the
day its skip is lifted -- but the fix belongs with the file that opens it
rather than here.

Checklist

I have added or updated unit tests
I have added or updated integration tests (if appropriate)
I have added or updated documentation

Testing Instructions

cd otdfctl && go test ./... -race

e2e, against a running platform:

cd otdfctl && bats --tap e2e --filter-tags payload_streaming

The first CI run of this file failed 308–311 and 317, all of them the cases
that need a successful decrypt. Cause was not the code under test: an encrypt
with no attributes falls back to the platform base key, and key-base.bats
sets one pointing at https://test-kas-for-base-keys.com, which does not
resolve — and cannot unset it, because a base key can only be replaced. Under
--jobs 4 the file order is nondeterministic, so which subset failed varied
per run. Fixed here by tagging the file payload_streaming and giving it its
own pass before the parallel batch. Tag arithmetic checks out: 14 + 10 + 330 =
354, the same total as before.

The memory case needs GNU time (gtime on macOS) and skips without it. It
allocates a 1 GiB file; peak RSS was ~3.6 GiB per command before this change
and the assertion threshold is 512 MiB.

The full DSPX-2604 stack — 20 PRs

#	PR	Based on
01	#3930 chore: bump go.work toolchain to go1.25.12 and simplify an rt_test condition	`main`
02	#3931 feat(sdk): make the zipstream clock injectable for deterministic ZIP output	`main`
03	#3932 fix(sdk): reject a zipstream write set that omits segment 0	#3931
04	#3933 fix(sdk): map ReadAt plaintext offsets from cumulative segment sizes	`main`
05	#3934 chore(sdk): extract integrityAlgorithmString, createPolicyBinding, signAssertions	`main`
06	#3935 chore(sdk): add direct tests for createKeyAccess, encryptMetadata and tdfSalt	`main`
07	#3936 fix(sdk): fill each segment with io.ReadFull and size the buffer to the input	`main`
08	#3937 chore(cli): move streaming IO helpers into pkg	`main`
09	#3938 fix(cli): stream encrypt instead of buffering the whole payload	#3937
10	#3939 fix(cli): stream decrypt and inspect instead of buffering	#3938
11	#3940 feat(sdk): add a chunked segment writer (experimental)	`dspx-2604-base-11` = #3932 + #3934 + #3935
12	#3941 fix(sdk): stop GetManifest from splitting the key under the lock	#3940
13	#3942 fix(sdk): reject a chunked split naming a KAS with no resolved public key	#3941
14	#3943 chore(sdk): alias experimental/tdf manifest and assertion types	#3942
15	#3944 fix(sdk): emit spec-compliant key access in experimental/tdf and delegate Writer	#3943
16	#3945 feat(sdk): accept io.Reader in CreateTDF and drop the 64 GB payload cap	#3936
17	#3946 chore(sdk): rewrite CreateTDF on top of the chunked writer	`dspx-2604-base-17` = #3944 + #3945
18	#3947 chore(sdk): drop dead TDFConfig fields and deprecate the TDFFormat enum	#3946
19	#3948 fix(cli): drop the encrypt-side stdin spool	`dspx-2604-base-19` = #3947 + #3939
20	#3949 feat(sdk): graduate the chunked writer to stable API	#3948

Reviewable in parallel right now, since they sit directly on main and depend on
nothing else: 01, 02, 04, 05, 06, 07, 08.

Why three PRs have a dspx-2604-base-* base. A GitHub PR takes one base branch,
but 11, 17 and 19 each build on more than one parent. The base-* branches are empty
merge commits that exist only to join those parents so the PR diff shows exactly its
own change and nothing else. They contain no code, have no PR of their own, and go
away once their parents land — retarget the child onto main at that point.

Wants a cross-SDK xtest run before merge: 15, 17 (and therefore 20). They touch
the KAS wire format.

Red checks you may see are network flakes, not this stack. Four distinct ones hit
this batch and all clear on re-run: golangci-lint config verify timing out on
https://golangci-lint.run/.../golangci.v2.8.jsonschema.json (fails the whole go (<module>) job and fail-fast cancels its siblings), the bats installer getting a 403,
Docker Hub timing out on keycloak/keycloak:26.4, and buf reporting "the server
hosted at that remote is unavailable" while the Java SDK generates sources. The
govulncheck step also emits ##[error] annotations against the go1.25.11 stdlib, but
it is continue-on-error: true and never fails a job — 01 bumps the toolchain and
clears those annotations.

coderabbitai · 2026-09-01T02:57:30Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 80687f42-8545-4e8b-b53c-01b39ecede69

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-09-01T03:01:59Z

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	253.225118ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	120.611852ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	412.967343ms
Throughput	242.15 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	41.987733638s
Average Latency	419.130854ms
Throughput	119.08 requests/second

github-actions · 2026-09-01T03:11:09Z

github-actions · 2026-09-01T03:31:10Z

X-Test Failure Report

opentdf~~platform~~XEO3RP.dockerbuild
govulncheck-failure-2
govulncheck-failure-8
govulncheck-failure-3
govulncheck-failure-1
govulncheck-failure-5
govulncheck-failure-0
govulncheck-failure-4

github-actions · 2026-09-01T03:34:07Z

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	256.568381ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	130.771436ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	414.107871ms
Throughput	241.48 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	43.194539476s
Average Latency	431.232655ms
Throughput	115.76 requests/second

github-actions · 2026-09-01T03:35:54Z

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	175.344915ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	85.346041ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	299.004165ms
Throughput	334.44 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	33.923613778s
Average Latency	338.529397ms
Throughput	147.39 requests/second

github-actions · 2026-09-01T03:36:14Z

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	240.89153ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	126.172214ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	448.143408ms
Throughput	223.14 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	42.368805139s
Average Latency	422.509278ms
Throughput	118.01 requests/second

github-actions · 2026-09-01T03:37:19Z

X-Test Failure Report

opentdf~~platform~~ANEK91.dockerbuild
govulncheck-failure-8
govulncheck-failure-3
govulncheck-failure-1
govulncheck-failure-4
govulncheck-failure-0
govulncheck-failure-2
govulncheck-failure-5

github-actions · 2026-09-01T03:41:30Z

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	165.106314ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	93.527817ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	306.97483ms
Throughput	325.76 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	34.737845354s
Average Latency	346.796044ms
Throughput	143.94 requests/second

github-actions · 2026-09-01T03:41:45Z

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	266.777517ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	132.530299ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	427.315995ms
Throughput	234.02 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	45.814723351s
Average Latency	457.337739ms
Throughput	109.14 requests/second

github-actions · 2026-09-01T03:56:35Z

github-actions · 2026-09-03T14:18:02Z

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	230.648168ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	121.644365ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	448.064997ms
Throughput	223.18 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	57.062357235s
Average Latency	569.187829ms
Throughput	87.62 requests/second

github-actions · 2026-09-03T14:26:45Z

X-Test Results

✅ go@pull-3939-main
✅ go@main-pull-3939
✅ go@pull-3939-v0.9.0
✅ go@main-main
✅ java@main-pull-3939
✅ go@main-v0.9.0
✅ java@main-main
cukes-report
✅ go@pull-3939-pull-3939
✅ js@main-pull-3939
✅ java@main-v0.9.0
✅ js@main-v0.9.0
✅ js@main-main
opentdf~~platform~~TDMV7V.dockerbuild
govulncheck-failure-3
govulncheck-failure-8
govulncheck-failure-1

`otdfctl decrypt` read the whole TDF into memory, handed the slice to DecryptBytes, which accumulated the whole plaintext in a bytes.Buffer, and then -- for stdout -- called Buffer.String(), allocating a third full copy. Peak RSS was roughly 3.6x the payload; a 1 GiB file cost ~3.7 GiB of RAM and a large enough file simply OOMed on a machine with plenty of disk for it. The plaintext now streams from the SDK reader to the destination. Handler.Decrypt takes an io.ReadSeeker and an io.Writer, with DecryptOptions replacing the positional parameter list, and inspect reaches the manifest through the same seekable reader rather than buffering the archive to get at its tail. io.Copy is what does the streaming, and it does so only because sdk.Reader implements WriteTo, which decrypts one segment at a time. Its Read delegates to ReadAt, which grows an internal bytes.Buffer holding every segment decrypted so far -- so dropping WriteTo would silently restore the old memory profile with no test failure to show for it. A compile-time assertion pins the interface. Removes MaxFileSize. The 10 GB cap existed to bound RAM; the real limit is the SDK maxFileSizeSupported at 64 GiB, which enforces itself. Output to a file is atomic, as on the encrypt side: the plaintext goes to a temporary sibling and is renamed into place only on success. Since cli.ExitWithError calls os.Exit and skips deferred functions, the spooled input and the partial output are discarded explicitly on every exit path -- including inspect's success path, which exits through ExitWithJSON. e2e coverage lands in a new otdfctl/e2e/streaming.bats rather than in encrypt-decrypt.bats, which carries a file-level skip pending the namespaced-subject-mappings migration and would have swallowed the new cases without running them. Nothing in the new file needs an entitlement, so it needs no policy fixtures: the round-trips use no attributes, and the two failure cases are forced with an unresolvable attribute FQN and a KAS allowlist that excludes the platform. As of this change it is the only e2e coverage of encrypt, decrypt and inspect that actually executes in CI. The file is tagged payload_streaming and action.yaml gives it its own pass ahead of the parallel batch. That ordering is load-bearing, not tidiness. An encrypt with no attributes falls back to the platform base key, and key-base.bats sets one pointing at https://test-kas-for-base-keys.com, which does not resolve. It cannot put things back afterwards: a base key can be replaced but never cleared, so every unattributed encrypt scheduled after that file yields a TDF nothing can decrypt. Under --jobs 4 the file order is nondeterministic, so overlapping the two made this suite flaky rather than merely broken -- which is how it presented, a different subset of round-trips failing per run. Running alone also keeps the 1 GiB peak-RSS case from measuring itself against three neighbours competing for the same memory. That leak is worth closing on its own -- encrypt-decrypt.bats walks into it the day its skip is lifted -- but the fix belongs with the file that opens it rather than here. Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>

github-actions · 2026-09-08T16:36:34Z

X-Test Failure Report

github-actions · 2026-09-08T16:42:27Z

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	190.340344ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	108.485295ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	401.760137ms
Throughput	248.90 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	51.32081585s
Average Latency	512.331924ms
Throughput	97.43 requests/second

github-actions · 2026-09-08T16:43:07Z

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	246.166925ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	137.476312ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	433.15911ms
Throughput	230.86 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	1m2.136220845s
Average Latency	619.944443ms
Throughput	80.47 requests/second

github-actions · 2026-09-08T16:44:13Z

⚠️ Govulncheck found vulnerabilities ⚠️

The following modules have known vulnerabilities:

otdfctl
service
tests-bdd

See the workflow run for details.

github-actions · 2026-09-08T16:52:20Z

X-Test Results

bats-test-results
✅ go@pull-3939-main
✅ go@pull-3939-pull-3939
✅ go@main-pull-3939
✅ go@pull-3939-v0.9.0
✅ go@main-v0.9.0
✅ go@main-main
✅ java@main-pull-3939
✅ js@main-pull-3939
✅ java@main-main
✅ java@main-v0.9.0
cukes-report
✅ js@main-v0.9.0
✅ js@main-main
opentdf~~platform~~0M0C2W.dockerbuild
govulncheck-failure-8
govulncheck-failure-3
govulncheck-failure-1

> **Part 08 of 20** in the DSPX-2604 re-cut. Base branch: `main`. > > This stack replaces #3782 / #3865 / #3921, which stay open and untouched > until it lands. Nothing here is a rebase of those branches — the work was > re-cut from the ticket so each PR stands on its own. ### Proposed Changes Adds otdfctl/pkg/streamio, holding the input and output plumbing that the streaming encrypt and decrypt work needs, and migrates `inspect` onto it so nothing is left calling the buffered helpers it supersedes. This is groundwork with one user-visible consequence: `inspect` no longer reads the whole TDF into memory. Everything else is a move. Why a new package rather than pkg/cli. The helpers in pkg/cli/pipe.go call ExitWithError -- which calls os.Exit -- from inside the read, so they cannot be used from anywhere that wants to handle the failure itself, and they read the entire input into memory. streamio returns errors and leaves the decision to exit with the command layer. What moved in: - PipeReader establishes whether stdin is a non-empty pipe with a one-byte Peek instead of a read, so the payload still reaches the caller. - Spool copies a pipe to a temporary file and rewinds it. A TDF's manifest sits at the end of the archive, so decrypt and inspect have to seek and cannot consume a pipe directly. - OpenSeekable resolves "file argument or piped stdin" to one seekable handle, reporting ErrNoInput for the shared "nothing to read" case. - OutputFile writes to a temporary sibling of the destination and renames it into place on Commit, so a failed run leaves no partial output. The temp file is a sibling so the rename stays atomic rather than degrading to a cross-filesystem copy. Per review feedback on #3921: - readPipedStdin now delegates its detection to streamio.PipeReader rather than answering "is there piped input?" a second way. Its read is still unbounded; the callers that must stop buffering are changed separately. - pkg/cli/pipe.go is deprecated rather than deleted, since the package is exported and may have callers outside this repository. Worth noting that ReadFromFile has no size cap at all -- not even the 10 GB the tdf commands apply -- which is its own argument for the notice. InspectTDF takes an io.ReadSeeker instead of a byte slice. GetTdfType already rewinds to the start, so the reader is positioned for LoadTDF. Because cli.ExitWithError calls os.Exit and skips deferred functions, inspectRun invokes cleanup explicitly on every exit path, including the successful one: piped input is spooled to disk and the temp file would otherwise survive. ### Checklist - [x] I have added or updated unit tests - [ ] I have added or updated integration tests (if appropriate) - [ ] I have added or updated documentation ### Testing Instructions ``` cd otdfctl && go test ./pkg/streamio/... ./cmd/... -race ``` `inspect` is the only command migrated in this PR; check it still reads both a file argument and piped stdin, and that no `otdfctl-spool-*` file survives either run. <details> <summary><b>The full DSPX-2604 stack — 20 PRs</b></summary> | # | PR | Based on | |---|----|----------| | 01 | #3930 chore: bump go.work toolchain to go1.25.12 and simplify an rt_test condition | `main` | | 02 | #3931 feat(sdk): make the zipstream clock injectable for deterministic ZIP output | `main` | | 03 | #3932 fix(sdk): reject a zipstream write set that omits segment 0 | #3931 | | 04 | #3933 fix(sdk): map ReadAt plaintext offsets from cumulative segment sizes | `main` | | 05 | #3934 chore(sdk): extract integrityAlgorithmString, createPolicyBinding, signAssertions | `main` | | 06 | #3935 chore(sdk): add direct tests for createKeyAccess, encryptMetadata and tdfSalt | `main` | | 07 | #3936 fix(sdk): fill each segment with io.ReadFull and size the buffer to the input | `main` | | 08 | #3937 chore(cli): move streaming IO helpers into pkg | `main` | | 09 | #3938 fix(cli): stream encrypt instead of buffering the whole payload | #3937 | | 10 | #3939 fix(cli): stream decrypt and inspect instead of buffering | #3938 | | 11 | #3940 feat(sdk): add a chunked segment writer (experimental) | `dspx-2604-base-11` = #3932 + #3934 + #3935 | | 12 | #3941 fix(sdk): stop GetManifest from splitting the key under the lock | #3940 | | 13 | #3942 fix(sdk): reject a chunked split naming a KAS with no resolved public key | #3941 | | 14 | #3943 chore(sdk): alias experimental/tdf manifest and assertion types | #3942 | | 15 | #3944 fix(sdk): emit spec-compliant key access in experimental/tdf and delegate Writer | #3943 | | 16 | #3945 feat(sdk): accept io.Reader in CreateTDF and drop the 64 GB payload cap | #3936 | | 17 | #3946 chore(sdk): rewrite CreateTDF on top of the chunked writer | `dspx-2604-base-17` = #3944 + #3945 | | 18 | #3947 chore(sdk): drop dead TDFConfig fields and deprecate the TDFFormat enum | #3946 | | 19 | #3948 fix(cli): drop the encrypt-side stdin spool | `dspx-2604-base-19` = #3947 + #3939 | | 20 | #3949 feat(sdk): graduate the chunked writer to stable API | #3948 | **Reviewable in parallel right now**, since they sit directly on `main` and depend on nothing else: 01, 02, 04, 05, 06, 07, 08. **Why three PRs have a `dspx-2604-base-*` base.** A GitHub PR takes one base branch, but 11, 17 and 19 each build on more than one parent. The `base-*` branches are empty merge commits that exist only to join those parents so the PR diff shows exactly its own change and nothing else. They contain no code, have no PR of their own, and go away once their parents land — retarget the child onto `main` at that point. **Wants a cross-SDK xtest run before merge:** 15, 17 (and therefore 20). They touch the KAS wire format. **Red checks you may see are network flakes, not this stack.** Four distinct ones hit this batch and all clear on re-run: `golangci-lint config verify` timing out on `https://golangci-lint.run/.../golangci.v2.8.jsonschema.json` (fails the whole `go (<module>)` job and fail-fast cancels its siblings), the bats installer getting a 403, Docker Hub timing out on `keycloak/keycloak:26.4`, and `buf` reporting "the server hosted at that remote is unavailable" while the Java SDK generates sources. The `govulncheck` step also emits `##[error]` annotations against the go1.25.11 stdlib, but it is `continue-on-error: true` and never fails a job — 01 bumps the toolchain and clears those annotations. </details>  ## Summary by CodeRabbit - **New Features** - Added reliable support for inspecting TDF content from files, piped input, and standard input. - Added safer output handling that prevents incomplete files from replacing existing results. - Added clearer input errors when no content is provided or an input cannot be opened. - **Bug Fixes** - Improved handling of large and non-seekable input streams. - Preserved piped input correctly while processing and inspecting content. - Non-fatal inspection issues are now reported as warnings where possible, allowing processing to continue.  Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>

dmihalcik-virtru requested review from a team as code owners September 1, 2026 02:57

github-actions Bot added the size/l label Sep 1, 2026

dmihalcik-virtru force-pushed the dspx-2604-10-stream-decrypt branch 2 times, most recently from 4ff797c to e98fcfd Compare September 1, 2026 03:30

dmihalcik-virtru force-pushed the dspx-2604-09-stream-encrypt branch from b28dc50 to c9b8343 Compare September 1, 2026 03:30

dmihalcik-virtru changed the title ~~fix(otdfctl): stream decrypt and inspect instead of buffering~~ fix(cli): stream decrypt and inspect instead of buffering Sep 1, 2026

dmihalcik-virtru force-pushed the dspx-2604-09-stream-encrypt branch from c9b8343 to eb523f9 Compare September 1, 2026 03:36

dmihalcik-virtru force-pushed the dspx-2604-10-stream-decrypt branch from e98fcfd to 9a7949d Compare September 1, 2026 03:36

dmihalcik-virtru force-pushed the dspx-2604-10-stream-decrypt branch from 9a7949d to f8ef543 Compare September 3, 2026 14:13

dmihalcik-virtru force-pushed the dspx-2604-09-stream-encrypt branch from 3750b81 to c1ce2ce Compare September 8, 2026 16:36

dmihalcik-virtru force-pushed the dspx-2604-10-stream-decrypt branch from f8ef543 to d9515ba Compare September 8, 2026 16:36

Uh oh!

Conversation

dmihalcik-virtru commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed Changes

Checklist

Testing Instructions

Uh oh!

coderabbitai Bot commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

github-actions Bot commented Sep 1, 2026

Benchmark authorization.GetDecisions Results:

Benchmark authorization.v2.GetMultiResourceDecision Results:

Benchmark Statistics

Bulk Benchmark Results

TDF3 Benchmark Results:

Uh oh!

github-actions Bot commented Sep 1, 2026

X-Test Results

Uh oh!

github-actions Bot commented Sep 1, 2026

X-Test Failure Report

Uh oh!

github-actions Bot commented Sep 1, 2026

Benchmark authorization.GetDecisions Results:

Benchmark authorization.v2.GetMultiResourceDecision Results:

Benchmark Statistics

Bulk Benchmark Results

TDF3 Benchmark Results:

Uh oh!

github-actions Bot commented Sep 1, 2026

Benchmark authorization.GetDecisions Results:

Benchmark authorization.v2.GetMultiResourceDecision Results:

Benchmark Statistics

Bulk Benchmark Results

TDF3 Benchmark Results:

Uh oh!

github-actions Bot commented Sep 1, 2026

Benchmark authorization.GetDecisions Results:

Benchmark authorization.v2.GetMultiResourceDecision Results:

Benchmark Statistics

Bulk Benchmark Results

TDF3 Benchmark Results:

Uh oh!

github-actions Bot commented Sep 1, 2026

X-Test Failure Report

Uh oh!

github-actions Bot commented Sep 1, 2026

Benchmark authorization.GetDecisions Results:

Benchmark authorization.v2.GetMultiResourceDecision Results:

Benchmark Statistics

Bulk Benchmark Results

TDF3 Benchmark Results:

Uh oh!

github-actions Bot commented Sep 1, 2026

Benchmark authorization.GetDecisions Results:

Benchmark authorization.v2.GetMultiResourceDecision Results:

Benchmark Statistics

Bulk Benchmark Results

TDF3 Benchmark Results:

Uh oh!

github-actions Bot commented Sep 1, 2026

X-Test Results

Uh oh!

github-actions Bot commented Sep 3, 2026

Benchmark authorization.GetDecisions Results:

Benchmark authorization.v2.GetMultiResourceDecision Results:

Benchmark Statistics

Bulk Benchmark Results

TDF3 Benchmark Results:

Uh oh!

github-actions Bot commented Sep 3, 2026

X-Test Results

Uh oh!

github-actions Bot commented Sep 8, 2026

X-Test Failure Report

Uh oh!

github-actions Bot commented Sep 8, 2026

dmihalcik-virtru commented Sep 1, 2026 •

edited

Loading

coderabbitai Bot commented Sep 1, 2026 •

edited

Loading