chore(sdk): rewrite CreateTDF on top of the chunked writer by dmihalcik-virtru · Pull Request #3946 · opentdf/platform

dmihalcik-virtru · 2026-09-01T02:57:36Z

Part 17 of 20 in the DSPX-2604 re-cut. Base branch: dspx-2604-base-17.

This stack replaces #3782 / #3865 / #3921, which stay open and untouched
until it lands. Nothing here is a rebase of those branches — the work was
re-cut from the ticket so each PR stands on its own.

Proposed Changes

CreateTDFContext and the chunked writer had grown two full implementations of
the same thing: build a policy, split the DEK, wrap each share to its KAS,
encrypt segments, accumulate an aggregate hash, sign assertions, emit a
manifest. Two copies of TDF construction is one too many — every spec change
has to land twice, and the second copy is the one that gets forgotten.

CreateTDF now delegates. It keeps the parts that are genuinely its own — the
KAO template, autoconfigure, the input-size resolution and read loop from the
previous commit — and hands each segment to the chunked writer, which owns
manifest assembly for both paths from here on.

The two paths differ in when key access is resolved, so that is what the
writer is now parameterized on. The chunked writer defers to a KeySplitter at
Finalize, because a caller may still be adding attributes while segments are
in flight. CreateTDF cannot: it knows its attributes up front and wants an
unreachable KAS to fail the call before a single payload byte reaches the
output writer. Both are expressed as a keyAccessResolver, with the DEK now
injectable so CreateTDF can wrap it ahead of time and hand the writer a
staticKeyAccess.

Everything downstream of that split is shared: resolvePolicyAndKeyAccess and
buildKeyAccessObjects replace prepareManifest's inline loop and the chunked
writer's buildChunkedPolicy/buildChunkedKeyAccessObjects, so both paths now
emit byte-identical policy and key access for the same attributes.

Two things fall out of the unification, both moving the chunked writer onto
the shipped classic behavior:

With zero attributes the policy body's "dataAttributes" and "dissem" are
now null rather than []. createPolicyObjectFromFQNs initializes them
inside the attribute loop; the deleted buildChunkedPolicy did so
unconditionally. The classic path has always emitted null here.
A KAS named by a split but missing a public key is still rejected rather
than skipped (the check moved into buildKeyAccessObjects), but the error
now names the missing PEM rather than the absent map entry — the two cases
were indistinguishable in practice and only the outcome matters.

The writer also gained an explicit segment size. It previously reported the
first segment's actual length as defaultSegmentSize, which is only correct
when every segment is full; a single-segment TDF would advertise a short
default. CreateTDF knows the configured size and now says so.

TDFObject loses aesGcm and payloadKey, which only ever existed to carry state
between prepareManifest and the encrypt loop.

Deliberately not in this commit: removing enableEncryption, tdfFormat, and
readActionName, which are dead but unrelated; they are a separate cleanup.

Testing: the existing TDFSuite round trips pin byte-level output across
integrity algorithms, segment sizes, target modes, and multi-KAS splits, and
pass unchanged. Also verified against the streaming-input and input-size
coverage added in the previous commit, the experimental chunked writer suite,
and cross-module builds of examples, otdfctl, service, and tests-bdd.

Checklist

I have added or updated unit tests
I have added or updated integration tests (if appropriate)
I have added or updated documentation

Testing Instructions

cd sdk && go test ./... -race

The existing TDFSuite round trips pin byte-level output across integrity
algorithms, segment sizes, target modes and multi-KAS splits, and pass
unchanged — that is the main assurance here.

This touches key access construction on the shipped path, so it wants a
cross-SDK run before merge:

gh workflow run xtest.yml --repo opentdf/tests --ref main \
  -f platform-ref=dspx-2604-17-createtdf-delegates \
  -f otdfctl-ref=main -f java-ref=main -f js-ref=main

The full DSPX-2604 stack — 20 PRs

#	PR	Based on
01	#3930 chore: bump go.work toolchain to go1.25.12 and simplify an rt_test condition	`main`
02	#3931 feat(sdk): make the zipstream clock injectable for deterministic ZIP output	`main`
03	#3932 fix(sdk): reject a zipstream write set that omits segment 0	#3931
04	#3933 fix(sdk): map ReadAt plaintext offsets from cumulative segment sizes	`main`
05	#3934 chore(sdk): extract integrityAlgorithmString, createPolicyBinding, signAssertions	`main`
06	#3935 chore(sdk): add direct tests for createKeyAccess, encryptMetadata and tdfSalt	`main`
07	#3936 fix(sdk): fill each segment with io.ReadFull and size the buffer to the input	`main`
08	#3937 chore(cli): move streaming IO helpers into pkg	`main`
09	#3938 fix(cli): stream encrypt instead of buffering the whole payload	#3937
10	#3939 fix(cli): stream decrypt and inspect instead of buffering	#3938
11	#3940 feat(sdk): add a chunked segment writer (experimental)	`dspx-2604-base-11` = #3932 + #3934 + #3935
12	#3941 fix(sdk): stop GetManifest from splitting the key under the lock	#3940
13	#3942 fix(sdk): reject a chunked split naming a KAS with no resolved public key	#3941
14	#3943 chore(sdk): alias experimental/tdf manifest and assertion types	#3942
15	#3944 fix(sdk): emit spec-compliant key access in experimental/tdf and delegate Writer	#3943
16	#3945 feat(sdk): accept io.Reader in CreateTDF and drop the 64 GB payload cap	#3936
17	#3946 chore(sdk): rewrite CreateTDF on top of the chunked writer	`dspx-2604-base-17` = #3944 + #3945
18	#3947 chore(sdk): drop dead TDFConfig fields and deprecate the TDFFormat enum	#3946
19	#3948 fix(cli): drop the encrypt-side stdin spool	`dspx-2604-base-19` = #3947 + #3939
20	#3949 feat(sdk): graduate the chunked writer to stable API	#3948

Reviewable in parallel right now, since they sit directly on main and depend on
nothing else: 01, 02, 04, 05, 06, 07, 08.

Why three PRs have a dspx-2604-base-* base. A GitHub PR takes one base branch,
but 11, 17 and 19 each build on more than one parent. The base-* branches are empty
merge commits that exist only to join those parents so the PR diff shows exactly its
own change and nothing else. They contain no code, have no PR of their own, and go
away once their parents land — retarget the child onto main at that point.

Wants a cross-SDK xtest run before merge: 15, 17 (and therefore 20). They touch
the KAS wire format.

Red checks you may see are network flakes, not this stack. Four distinct ones hit
this batch and all clear on re-run: golangci-lint config verify timing out on
https://golangci-lint.run/.../golangci.v2.8.jsonschema.json (fails the whole go (<module>) job and fail-fast cancels its siblings), the bats installer getting a 403,
Docker Hub timing out on keycloak/keycloak:26.4, and buf reporting "the server
hosted at that remote is unavailable" while the Java SDK generates sources. The
govulncheck step also emits ##[error] annotations against the go1.25.11 stdlib, but
it is continue-on-error: true and never fails a job — 01 bumps the toolchain and
clears those annotations.

coderabbitai · 2026-09-01T02:57:43Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 8bd601ca-6cdc-4674-9f4a-37da7048af71

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-09-01T03:03:29Z

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	260.573394ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	135.553264ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	417.368815ms
Throughput	239.60 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	43.737562519s
Average Latency	436.48954ms
Throughput	114.32 requests/second

github-actions · 2026-09-01T03:11:30Z

github-actions · 2026-09-01T03:30:53Z

X-Test Failure Report

github-actions · 2026-09-01T03:36:34Z

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	275.391918ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	139.401877ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	443.371855ms
Throughput	225.54 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	44.279606203s
Average Latency	442.099151ms
Throughput	112.92 requests/second

github-actions · 2026-09-03T14:45:36Z

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	225.346848ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	129.183876ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	441.873494ms
Throughput	226.31 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	56.50767164s
Average Latency	563.64472ms
Throughput	88.48 requests/second

github-actions · 2026-09-03T14:45:39Z

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	239.884947ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	135.306657ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	434.602682ms
Throughput	230.10 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	1m1.717860027s
Average Latency	615.831103ms
Throughput	81.01 requests/second

github-actions · 2026-09-03T14:55:24Z

X-Test Results

✅ java@main-main
✅ go@pull-3946-pull-3946
✅ go@main-main
✅ go@main-pull-3946
✅ go@pull-3946-main
✅ go@main-v0.9.0
✅ go@pull-3946-v0.9.0
✅ java@main-pull-3946
bats-test-results
✅ java@main-v0.9.0
✅ js@main-main
✅ js@main-pull-3946
cukes-report
✅ js@main-v0.9.0
govulncheck-failure-1
govulncheck-failure-8
opentdf~~platform~~KN52S7.dockerbuild
govulncheck-failure-3

github-actions · 2026-09-08T17:15:38Z

X-Test Failure Report

github-actions · 2026-09-08T17:21:29Z

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	173.702469ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	104.707741ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	363.953648ms
Throughput	274.76 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	50.501853457s
Average Latency	504.18192ms
Throughput	99.01 requests/second

github-actions · 2026-09-08T17:22:08Z

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	231.134322ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	129.168262ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	424.48979ms
Throughput	235.58 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	56.975034773s
Average Latency	567.967964ms
Throughput	87.76 requests/second

github-actions · 2026-09-08T17:31:23Z

X-Test Results

✅ go@main-pull-3946
✅ go@main-main
✅ go@pull-3946-pull-3946
✅ go@main-v0.9.0
✅ go@pull-3946-v0.9.0
✅ java@main-main
bats-test-results
✅ java@main-pull-3946
✅ go@pull-3946-main
✅ java@main-v0.9.0
✅ js@main-main
✅ js@main-pull-3946
cukes-report
✅ js@main-v0.9.0
opentdf~~platform~~DMACI6.dockerbuild
govulncheck-failure-8
govulncheck-failure-3
govulncheck-failure-1

github-actions · 2026-09-08T17:37:13Z

X-Test Failure Report

github-actions · 2026-09-08T17:43:04Z

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	245.183523ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	133.853218ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	421.738355ms
Throughput	237.11 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	59.838239515s
Average Latency	596.97852ms
Throughput	83.56 requests/second

github-actions · 2026-09-08T17:43:12Z

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	250.324489ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	141.370812ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	429.285847ms
Throughput	232.95 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	1m2.624905124s
Average Latency	624.920224ms
Throughput	79.84 requests/second

github-actions · 2026-09-08T17:53:46Z

X-Test Results

✅ go@main-main
✅ go@pull-3946-pull-3946
bats-test-results
✅ go@main-pull-3946
✅ java@main-pull-3946
✅ go@pull-3946-main
✅ js@main-main
✅ go@main-v0.9.0
✅ js@main-v0.9.0
✅ js@main-pull-3946
✅ go@pull-3946-v0.9.0
✅ java@main-main
cukes-report
✅ java@main-v0.9.0
opentdf~~platform~~MJAUHO.dockerbuild
govulncheck-failure-1
govulncheck-failure-8
govulncheck-failure-3

github-actions · 2026-09-08T20:35:11Z

X-Test Failure Report

> **Part 08 of 20** in the DSPX-2604 re-cut. Base branch: `main`. > > This stack replaces #3782 / #3865 / #3921, which stay open and untouched > until it lands. Nothing here is a rebase of those branches — the work was > re-cut from the ticket so each PR stands on its own. ### Proposed Changes Adds otdfctl/pkg/streamio, holding the input and output plumbing that the streaming encrypt and decrypt work needs, and migrates `inspect` onto it so nothing is left calling the buffered helpers it supersedes. This is groundwork with one user-visible consequence: `inspect` no longer reads the whole TDF into memory. Everything else is a move. Why a new package rather than pkg/cli. The helpers in pkg/cli/pipe.go call ExitWithError -- which calls os.Exit -- from inside the read, so they cannot be used from anywhere that wants to handle the failure itself, and they read the entire input into memory. streamio returns errors and leaves the decision to exit with the command layer. What moved in: - PipeReader establishes whether stdin is a non-empty pipe with a one-byte Peek instead of a read, so the payload still reaches the caller. - Spool copies a pipe to a temporary file and rewinds it. A TDF's manifest sits at the end of the archive, so decrypt and inspect have to seek and cannot consume a pipe directly. - OpenSeekable resolves "file argument or piped stdin" to one seekable handle, reporting ErrNoInput for the shared "nothing to read" case. - OutputFile writes to a temporary sibling of the destination and renames it into place on Commit, so a failed run leaves no partial output. The temp file is a sibling so the rename stays atomic rather than degrading to a cross-filesystem copy. Per review feedback on #3921: - readPipedStdin now delegates its detection to streamio.PipeReader rather than answering "is there piped input?" a second way. Its read is still unbounded; the callers that must stop buffering are changed separately. - pkg/cli/pipe.go is deprecated rather than deleted, since the package is exported and may have callers outside this repository. Worth noting that ReadFromFile has no size cap at all -- not even the 10 GB the tdf commands apply -- which is its own argument for the notice. InspectTDF takes an io.ReadSeeker instead of a byte slice. GetTdfType already rewinds to the start, so the reader is positioned for LoadTDF. Because cli.ExitWithError calls os.Exit and skips deferred functions, inspectRun invokes cleanup explicitly on every exit path, including the successful one: piped input is spooled to disk and the temp file would otherwise survive. ### Checklist - [x] I have added or updated unit tests - [ ] I have added or updated integration tests (if appropriate) - [ ] I have added or updated documentation ### Testing Instructions ``` cd otdfctl && go test ./pkg/streamio/... ./cmd/... -race ``` `inspect` is the only command migrated in this PR; check it still reads both a file argument and piped stdin, and that no `otdfctl-spool-*` file survives either run. <details> <summary><b>The full DSPX-2604 stack — 20 PRs</b></summary> | # | PR | Based on | |---|----|----------| | 01 | #3930 chore: bump go.work toolchain to go1.25.12 and simplify an rt_test condition | `main` | | 02 | #3931 feat(sdk): make the zipstream clock injectable for deterministic ZIP output | `main` | | 03 | #3932 fix(sdk): reject a zipstream write set that omits segment 0 | #3931 | | 04 | #3933 fix(sdk): map ReadAt plaintext offsets from cumulative segment sizes | `main` | | 05 | #3934 chore(sdk): extract integrityAlgorithmString, createPolicyBinding, signAssertions | `main` | | 06 | #3935 chore(sdk): add direct tests for createKeyAccess, encryptMetadata and tdfSalt | `main` | | 07 | #3936 fix(sdk): fill each segment with io.ReadFull and size the buffer to the input | `main` | | 08 | #3937 chore(cli): move streaming IO helpers into pkg | `main` | | 09 | #3938 fix(cli): stream encrypt instead of buffering the whole payload | #3937 | | 10 | #3939 fix(cli): stream decrypt and inspect instead of buffering | #3938 | | 11 | #3940 feat(sdk): add a chunked segment writer (experimental) | `dspx-2604-base-11` = #3932 + #3934 + #3935 | | 12 | #3941 fix(sdk): stop GetManifest from splitting the key under the lock | #3940 | | 13 | #3942 fix(sdk): reject a chunked split naming a KAS with no resolved public key | #3941 | | 14 | #3943 chore(sdk): alias experimental/tdf manifest and assertion types | #3942 | | 15 | #3944 fix(sdk): emit spec-compliant key access in experimental/tdf and delegate Writer | #3943 | | 16 | #3945 feat(sdk): accept io.Reader in CreateTDF and drop the 64 GB payload cap | #3936 | | 17 | #3946 chore(sdk): rewrite CreateTDF on top of the chunked writer | `dspx-2604-base-17` = #3944 + #3945 | | 18 | #3947 chore(sdk): drop dead TDFConfig fields and deprecate the TDFFormat enum | #3946 | | 19 | #3948 fix(cli): drop the encrypt-side stdin spool | `dspx-2604-base-19` = #3947 + #3939 | | 20 | #3949 feat(sdk): graduate the chunked writer to stable API | #3948 | **Reviewable in parallel right now**, since they sit directly on `main` and depend on nothing else: 01, 02, 04, 05, 06, 07, 08. **Why three PRs have a `dspx-2604-base-*` base.** A GitHub PR takes one base branch, but 11, 17 and 19 each build on more than one parent. The `base-*` branches are empty merge commits that exist only to join those parents so the PR diff shows exactly its own change and nothing else. They contain no code, have no PR of their own, and go away once their parents land — retarget the child onto `main` at that point. **Wants a cross-SDK xtest run before merge:** 15, 17 (and therefore 20). They touch the KAS wire format. **Red checks you may see are network flakes, not this stack.** Four distinct ones hit this batch and all clear on re-run: `golangci-lint config verify` timing out on `https://golangci-lint.run/.../golangci.v2.8.jsonschema.json` (fails the whole `go (<module>)` job and fail-fast cancels its siblings), the bats installer getting a 403, Docker Hub timing out on `keycloak/keycloak:26.4`, and `buf` reporting "the server hosted at that remote is unavailable" while the Java SDK generates sources. The `govulncheck` step also emits `##[error]` annotations against the go1.25.11 stdlib, but it is `continue-on-error: true` and never fails a job — 01 bumps the toolchain and clears those annotations. </details>  ## Summary by CodeRabbit - **New Features** - Added reliable support for inspecting TDF content from files, piped input, and standard input. - Added safer output handling that prevents incomplete files from replacing existing results. - Added clearer input errors when no content is provided or an input cannot be opened. - **Bug Fixes** - Improved handling of large and non-seekable input streams. - Preserved piped input correctly while processing and inspecting content. - Non-fatal inspection issues are now reported as warnings where possible, allowing processing to continue.  Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>

github-actions · 2026-09-08T20:40:56Z

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	216.335079ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	119.272078ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	460.193636ms
Throughput	217.30 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	56.175909622s
Average Latency	560.478635ms
Throughput	89.01 requests/second

github-actions · 2026-09-08T20:41:15Z

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	238.76595ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	139.193742ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	433.056665ms
Throughput	230.92 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	1m1.946890607s
Average Latency	617.994435ms
Throughput	80.71 requests/second

CreateTDFContext and the chunked writer had grown two full implementations of the same thing: build a policy, split the DEK, wrap each share to its KAS, encrypt segments, accumulate an aggregate hash, sign assertions, emit a manifest. Two copies of TDF construction is one too many — every spec change has to land twice, and the second copy is the one that gets forgotten. CreateTDF now delegates. It keeps the parts that are genuinely its own — the KAO template, autoconfigure, the input-size resolution and read loop from the previous commit — and hands each segment to the chunked writer, which owns manifest assembly for both paths from here on. The two paths differ in when key access is resolved, so that is what the writer is now parameterized on. The chunked writer defers to a KeySplitter at Finalize, because a caller may still be adding attributes while segments are in flight. CreateTDF cannot: it knows its attributes up front and wants an unreachable KAS to fail the call before a single payload byte reaches the output writer. Both are expressed as a keyAccessResolver, with the DEK now injectable so CreateTDF can wrap it ahead of time and hand the writer a staticKeyAccess. Everything downstream of that split is shared: resolvePolicyAndKeyAccess and buildKeyAccessObjects replace prepareManifest's inline loop and the chunked writer's buildChunkedPolicy/buildChunkedKeyAccessObjects, so both paths now emit byte-identical policy and key access for the same attributes. Two things fall out of the unification, both moving the chunked writer onto the shipped classic behavior: - With zero attributes the policy body's "dataAttributes" and "dissem" are now null rather than []. createPolicyObjectFromFQNs initializes them inside the attribute loop; the deleted buildChunkedPolicy did so unconditionally. The classic path has always emitted null here. - A KAS named by a split but missing a public key is still rejected rather than skipped (the check moved into buildKeyAccessObjects), but the error now names the missing PEM rather than the absent map entry — the two cases were indistinguishable in practice and only the outcome matters. The writer also gained an explicit segment size. It previously reported the first segment's actual length as defaultSegmentSize, which is only correct when every segment is full; a single-segment TDF would advertise a short default. CreateTDF knows the configured size and now says so. TDFObject loses aesGcm and payloadKey, which only ever existed to carry state between prepareManifest and the encrypt loop. Deliberately not in this commit: removing enableEncryption, tdfFormat, and readActionName, which are dead but unrelated; they are a separate cleanup. Testing: the existing TDFSuite round trips pin byte-level output across integrity algorithms, segment sizes, target modes, and multi-KAS splits, and pass unchanged. Also verified against the streaming-input and input-size coverage added in the previous commit, the experimental chunked writer suite, and cross-module builds of examples, otdfctl, service, and tests-bdd. Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>

github-actions · 2026-09-08T20:48:01Z

X-Test Failure Report

bats-test-results
✅ go@main-v0.9.0
✅ java@main-main
✅ js@main-pull-3946
✅ js@main-main
✅ js@main-v0.9.0
✅ go@pull-3946-v0.9.0
✅ java@main-v0.9.0
opentdf~~platform~~JPB14N.dockerbuild
govulncheck-failure-8
govulncheck-failure-3
govulncheck-failure-1

github-actions · 2026-09-08T20:52:35Z

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	229.800357ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	128.95655ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	453.618841ms
Throughput	220.45 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	1m1.976409487s
Average Latency	618.286646ms
Throughput	80.68 requests/second

github-actions · 2026-09-08T20:52:54Z

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	222.879727ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	127.336281ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	450.061395ms
Throughput	222.19 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	55.224517132s
Average Latency	551.232237ms
Throughput	90.54 requests/second

github-actions · 2026-09-08T20:53:37Z

⚠️ Govulncheck found vulnerabilities ⚠️

The following modules have known vulnerabilities:

otdfctl
service
tests-bdd

See the workflow run for details.

github-actions · 2026-09-08T21:04:24Z

X-Test Results

✅ go@pull-3946-main
✅ go@pull-3946-pull-3946
✅ go@main-pull-3946
✅ go@main-main
✅ go@main-v0.9.0
bats-test-results
✅ java@main-main
✅ go@pull-3946-v0.9.0
✅ js@main-main
✅ js@main-pull-3946
✅ java@main-pull-3946
✅ java@main-v0.9.0
cukes-report
✅ js@main-v0.9.0
opentdf~~platform~~Z1P4TL.dockerbuild
govulncheck-failure-8
govulncheck-failure-3
govulncheck-failure-1

dmihalcik-virtru requested review from a team as code owners September 1, 2026 02:57

github-actions Bot added comp:sdk A software development kit, including library, for client applications and inter-service communicati size/l labels Sep 1, 2026

dmihalcik-virtru force-pushed the dspx-2604-17-createtdf-delegates branch from 02e414b to f6a3cf2 Compare September 1, 2026 03:30

dmihalcik-virtru force-pushed the dspx-2604-base-17 branch from 9693e0a to 74608c8 Compare September 1, 2026 03:30

dmihalcik-virtru force-pushed the dspx-2604-base-17 branch from 05eb20f to 39b0d00 Compare September 8, 2026 17:15

dmihalcik-virtru force-pushed the dspx-2604-17-createtdf-delegates branch from 464aef4 to 4be6d0d Compare September 8, 2026 17:15

dmihalcik-virtru force-pushed the dspx-2604-17-createtdf-delegates branch from 4be6d0d to 760fb30 Compare September 8, 2026 17:36

dmihalcik-virtru force-pushed the dspx-2604-base-17 branch from 39b0d00 to 0094781 Compare September 8, 2026 17:37

dmihalcik-virtru force-pushed the dspx-2604-17-createtdf-delegates branch from 760fb30 to 7c99393 Compare September 8, 2026 20:34

dmihalcik-virtru force-pushed the dspx-2604-base-17 branch from 0094781 to 5a5a5ff Compare September 8, 2026 20:34

dmihalcik-virtru force-pushed the dspx-2604-17-createtdf-delegates branch from 760fb30 to 7c99393 Compare September 8, 2026 20:34

dmihalcik-virtru force-pushed the dspx-2604-base-17 branch from 5a5a5ff to f9db686 Compare September 8, 2026 20:46

dmihalcik-virtru force-pushed the dspx-2604-17-createtdf-delegates branch from 7c99393 to bac467c Compare September 8, 2026 20:46

Uh oh!

Conversation

dmihalcik-virtru commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed Changes

Checklist

Testing Instructions

Uh oh!

coderabbitai Bot commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

github-actions Bot commented Sep 1, 2026

Benchmark authorization.GetDecisions Results:

Benchmark authorization.v2.GetMultiResourceDecision Results:

Benchmark Statistics

Bulk Benchmark Results

TDF3 Benchmark Results:

Uh oh!

github-actions Bot commented Sep 1, 2026

X-Test Results

Uh oh!

github-actions Bot commented Sep 1, 2026

X-Test Failure Report

Uh oh!

github-actions Bot commented Sep 1, 2026

Benchmark authorization.GetDecisions Results:

Benchmark authorization.v2.GetMultiResourceDecision Results:

Benchmark Statistics

Bulk Benchmark Results

TDF3 Benchmark Results:

Uh oh!

github-actions Bot commented Sep 3, 2026

Benchmark authorization.GetDecisions Results:

Benchmark authorization.v2.GetMultiResourceDecision Results:

Benchmark Statistics

Bulk Benchmark Results

TDF3 Benchmark Results:

Uh oh!

github-actions Bot commented Sep 3, 2026

Benchmark authorization.GetDecisions Results:

Benchmark authorization.v2.GetMultiResourceDecision Results:

Benchmark Statistics

Bulk Benchmark Results

TDF3 Benchmark Results:

Uh oh!

github-actions Bot commented Sep 3, 2026

X-Test Results

Uh oh!

github-actions Bot commented Sep 8, 2026

X-Test Failure Report

Uh oh!

github-actions Bot commented Sep 8, 2026

Benchmark authorization.GetDecisions Results:

Benchmark authorization.v2.GetMultiResourceDecision Results:

Benchmark Statistics

Bulk Benchmark Results

TDF3 Benchmark Results:

Uh oh!

github-actions Bot commented Sep 8, 2026

Benchmark authorization.GetDecisions Results:

Benchmark authorization.v2.GetMultiResourceDecision Results:

Benchmark Statistics

Bulk Benchmark Results

TDF3 Benchmark Results:

Uh oh!

github-actions Bot commented Sep 8, 2026

X-Test Results

Uh oh!

github-actions Bot commented Sep 8, 2026

X-Test Failure Report

Uh oh!

github-actions Bot commented Sep 8, 2026

Benchmark authorization.GetDecisions Results:

Benchmark authorization.v2.GetMultiResourceDecision Results:

Benchmark Statistics

Bulk Benchmark Results

TDF3 Benchmark Results:

Uh oh!

github-actions Bot commented Sep 8, 2026

dmihalcik-virtru commented Sep 1, 2026 •

edited

Loading

coderabbitai Bot commented Sep 1, 2026 •

edited

Loading