Skip to content

feat(sdk): add a chunked segment writer (experimental) - #3940

Open
dmihalcik-virtru wants to merge 1 commit into
dspx-2604-base-11from
dspx-2604-11-chunked-writer
Open

feat(sdk): add a chunked segment writer (experimental)#3940
dmihalcik-virtru wants to merge 1 commit into
dspx-2604-base-11from
dspx-2604-11-chunked-writer

Conversation

@dmihalcik-virtru

@dmihalcik-virtru dmihalcik-virtru commented Sep 1, 2026

Copy link
Copy Markdown
Member

Part 11 of 20 in the DSPX-2604 re-cut. Base branch: dspx-2604-base-11.

This stack replaces #3782 / #3865 / #3921, which stay open and untouched
until it lands. Nothing here is a rebase of those branches — the work was
re-cut from the ticket so each PR stands on its own.

Proposed Changes

Adds ChunkedWriter, a TDF creation path that accepts segments in any
order. Callers encrypt and upload each segment independently -- typically
off-thread or in parallel -- then call Finalize to get the ZIP closing
bytes. Contrast with SDK.CreateTDF, which needs the whole plaintext up
front behind an io.ReadSeeker.

This is the first half of DSPX-2604. It adds the implementation only; the
public face of out-of-order writing stays sdk/experimental/tdf, which is
rewired onto this writer in a later PR. Graduating ChunkedWriter to
supported API -- dropping the Experimental: markers and adding the
SDK.NewChunkedWriter method -- is deliberately deferred to the end of the
stack.

What is exported, and why that set

Go forces a compromise here. sdk/experimental/tdf is a separate package,
so it cannot reach unexported symbols in sdk, and the writer cannot move
under sdk/internal/ because it needs Manifest, KeyAccess, Segment,
createKeyAccess and calculateSignature. So: export the minimum bridge
the adapter needs, mark all of it experimental, unexport everything else.

Exported and Experimental:-marked: ChunkedWriter, NewChunkedWriter,
the two result structs, the two config structs and option types, the
caller-facing WithChunked* options, the sentinel errors, and the
KeySplitter / Split / SplitResult / KASPublicKey group that an
out-of-package splitter has to implement.

Unexported: the test seams. The clock, the segment cipher and its factory,
the archive writer factory, the entropy source, and the four options that
inject them. archiveWriterFactory in particular returns a
zipstream.SegmentWriter from internal/, so no external package could
have implemented it even when the type was exported -- an exported symbol
no caller can satisfy should not be exported. Those three files (clock.go,
segment.go, archive_writer.go -- 28, 29 and 23 lines) are folded into
chunked_writer.go now that nothing outside the package can see them.

Correctness guards included rather than deferred

Three small guards ship with the code they protect, since splitting "add
new code with a known hole" from "fix it" across two PRs of brand-new code
is churn with no review value:

  • Finalize rejects a write set missing segment 0
    (ErrChunkedMissingSegmentZero). Only segment 0 emits the payload's ZIP
    local file header and every recorded offset is measured from it, so a set
    without it silently produces a corrupt archive. It cannot be synthesized
    at Finalize time -- by then the caller has already encrypted and shipped
    the bytes.
  • The default splitter rejects a KAS key whose algorithm has no wrapping
    scheme (ErrSplitterUnsupportedAlgorithm) instead of letting the empty
    string reach createKeyAccess, where it selects the RSA branch while
    ocrypto.FromPublicPEM sniffs the PEM and wraps anyway -- producing a KAO
    that claims keyType "wrapped" with no ephemeral public key, i.e. a TDF
    nothing can decrypt.
  • The injection-seam options reject nil rather than storing it. A stored nil
    is indistinguishable from an unset field, so no default is installed and
    the nil surfaces as a panic partway through -- for the key splitter, not
    until Finalize.

WriteSegment also reserves an index with a negative-size placeholder and
rolls the reservation back if encryption, signing or the archive write
fails, so a failed attempt cannot leave a placeholder that blocks a retry
or that Finalize mistakes for a written segment. Release matches on
pointer identity and on the placeholder still being unwritten, so it can
never discard a segment another call has since completed.

Deferred to the next PR: the GetManifest-splits-under-RLock question,
which is a behavior change to an exported method rather than a hole in
what lands here.

Tests cover round-trip, out-of-order and sparse writes, segment trimming,
KAO shape for RSA and EC, legacy and current target modes, assertion
signing, deterministic ZIP timestamps, the error contracts above,
archive-failure rollback and retry, and concurrent WriteSegment calls to
distinct and to duplicate indices under -race.

Checklist

  • I have added or updated unit tests
  • I have added or updated integration tests (if appropriate)
  • I have added or updated documentation

Testing Instructions

cd sdk && go test ./... -race

sdk/chunked_test.go is the bulk of the diff. The concurrency cases
(WriteSegment to distinct and to duplicate indices) are the ones that want
-race specifically.

The full DSPX-2604 stack — 20 PRs
# PR Based on
01 #3930 chore: bump go.work toolchain to go1.25.12 and simplify an rt_test condition main
02 #3931 feat(sdk): make the zipstream clock injectable for deterministic ZIP output main
03 #3932 fix(sdk): reject a zipstream write set that omits segment 0 #3931
04 #3933 fix(sdk): map ReadAt plaintext offsets from cumulative segment sizes main
05 #3934 chore(sdk): extract integrityAlgorithmString, createPolicyBinding, signAssertions main
06 #3935 chore(sdk): add direct tests for createKeyAccess, encryptMetadata and tdfSalt main
07 #3936 fix(sdk): fill each segment with io.ReadFull and size the buffer to the input main
08 #3937 chore(cli): move streaming IO helpers into pkg main
09 #3938 fix(cli): stream encrypt instead of buffering the whole payload #3937
10 #3939 fix(cli): stream decrypt and inspect instead of buffering #3938
11 #3940 feat(sdk): add a chunked segment writer (experimental) dspx-2604-base-11 = #3932 + #3934 + #3935
12 #3941 fix(sdk): stop GetManifest from splitting the key under the lock #3940
13 #3942 fix(sdk): reject a chunked split naming a KAS with no resolved public key #3941
14 #3943 chore(sdk): alias experimental/tdf manifest and assertion types #3942
15 #3944 fix(sdk): emit spec-compliant key access in experimental/tdf and delegate Writer #3943
16 #3945 feat(sdk): accept io.Reader in CreateTDF and drop the 64 GB payload cap #3936
17 #3946 chore(sdk): rewrite CreateTDF on top of the chunked writer dspx-2604-base-17 = #3944 + #3945
18 #3947 chore(sdk): drop dead TDFConfig fields and deprecate the TDFFormat enum #3946
19 #3948 fix(cli): drop the encrypt-side stdin spool dspx-2604-base-19 = #3947 + #3939
20 #3949 feat(sdk): graduate the chunked writer to stable API #3948

Reviewable in parallel right now, since they sit directly on main and depend on
nothing else: 01, 02, 04, 05, 06, 07, 08.

Why three PRs have a dspx-2604-base-* base. A GitHub PR takes one base branch,
but 11, 17 and 19 each build on more than one parent. The base-* branches are empty
merge commits that exist only to join those parents so the PR diff shows exactly its
own change and nothing else. They contain no code, have no PR of their own, and go
away once their parents land — retarget the child onto main at that point.

Wants a cross-SDK xtest run before merge: 15, 17 (and therefore 20). They touch
the KAS wire format.

Red checks you may see are network flakes, not this stack. Four distinct ones hit
this batch and all clear on re-run: golangci-lint config verify timing out on
https://golangci-lint.run/.../golangci.v2.8.jsonschema.json (fails the whole go (<module>) job and fail-fast cancels its siblings), the bats installer getting a 403,
Docker Hub timing out on keycloak/keycloak:26.4, and buf reporting "the server
hosted at that remote is unavailable" while the Java SDK generates sources. The
govulncheck step also emits ##[error] annotations against the go1.25.11 stdlib, but
it is continue-on-error: true and never fails a job — 01 bumps the toolchain and
clears those annotations.

@dmihalcik-virtru
dmihalcik-virtru requested review from a team as code owners September 1, 2026 02:57
@coderabbitai

coderabbitai Bot commented Sep 1, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: ASSERTIVE

Plan: Team

Run ID: b187ac1e-0bc7-4b3f-b309-d4dd4e7097e6

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added comp:sdk A software development kit, including library, for client applications and inter-service communicati size/xl labels Sep 1, 2026
@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor
Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 249.367523ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 135.815882ms

Benchmark Statistics

Name № Requests Avg Duration Min Duration Max Duration

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 424.900496ms
Throughput 235.35 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 45.011917207s
Average Latency 449.359824ms
Throughput 111.08 requests/second

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor
Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 258.976987ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 120.08747ms

Benchmark Statistics

Name № Requests Avg Duration Min Duration Max Duration

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 418.384049ms
Throughput 239.01 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 43.440828764s
Average Latency 433.695524ms
Throughput 115.10 requests/second

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

X-Test Failure Report

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor
Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 266.148451ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 102.656526ms

Benchmark Statistics

Name № Requests Avg Duration Min Duration Max Duration

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 415.212131ms
Throughput 240.84 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 41.609055135s
Average Latency 415.319258ms
Throughput 120.17 requests/second

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor
Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 251.627571ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 128.260185ms

Benchmark Statistics

Name № Requests Avg Duration Min Duration Max Duration

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 455.481972ms
Throughput 219.55 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 43.741044707s
Average Latency 436.653998ms
Throughput 114.31 requests/second

@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

X-Test Failure Report

Adds `ChunkedWriter`, a TDF creation path that accepts segments in any
order. Callers encrypt and upload each segment independently -- typically
off-thread or in parallel -- then call `Finalize` to get the ZIP closing
bytes. Contrast with `SDK.CreateTDF`, which needs the whole plaintext up
front behind an `io.ReadSeeker`.

This is the first half of DSPX-2604. It adds the implementation only; the
public face of out-of-order writing stays `sdk/experimental/tdf`, which is
rewired onto this writer in a later PR. Graduating `ChunkedWriter` to
supported API -- dropping the `Experimental:` markers and adding the
`SDK.NewChunkedWriter` method -- is deliberately deferred to the end of the
stack.

What is exported, and why that set

Go forces a compromise here. `sdk/experimental/tdf` is a separate package,
so it cannot reach unexported symbols in `sdk`, and the writer cannot move
under `sdk/internal/` because it needs `Manifest`, `KeyAccess`, `Segment`,
`createKeyAccess` and `calculateSignature`. So: export the minimum bridge
the adapter needs, mark all of it experimental, unexport everything else.

Exported and `Experimental:`-marked: `ChunkedWriter`, `NewChunkedWriter`,
the two result structs, the two config structs and option types, the
caller-facing `WithChunked*` options, the sentinel errors, and the
`KeySplitter` / `Split` / `SplitResult` / `KASPublicKey` group that an
out-of-package splitter has to implement.

Unexported: the test seams. The clock, the segment cipher and its factory,
the archive writer factory, the entropy source, and the four options that
inject them. `archiveWriterFactory` in particular returns a
`zipstream.SegmentWriter` from `internal/`, so no external package could
have implemented it even when the type was exported -- an exported symbol
no caller can satisfy should not be exported. Those three files (`clock.go`,
`segment.go`, `archive_writer.go` -- 28, 29 and 23 lines) are folded into
`chunked_writer.go` now that nothing outside the package can see them.

Correctness guards included rather than deferred

Three small guards ship with the code they protect, since splitting "add
new code with a known hole" from "fix it" across two PRs of brand-new code
is churn with no review value:

- `Finalize` rejects a write set missing segment 0
  (`ErrChunkedMissingSegmentZero`). Only segment 0 emits the payload's ZIP
  local file header and every recorded offset is measured from it, so a set
  without it silently produces a corrupt archive. It cannot be synthesized
  at `Finalize` time -- by then the caller has already encrypted and shipped
  the bytes.
- The default splitter rejects a KAS key whose algorithm has no wrapping
  scheme (`ErrSplitterUnsupportedAlgorithm`) instead of letting the empty
  string reach `createKeyAccess`, where it selects the RSA branch while
  `ocrypto.FromPublicPEM` sniffs the PEM and wraps anyway -- producing a KAO
  that claims keyType "wrapped" with no ephemeral public key, i.e. a TDF
  nothing can decrypt.
- The injection-seam options reject nil rather than storing it. A stored nil
  is indistinguishable from an unset field, so no default is installed and
  the nil surfaces as a panic partway through -- for the key splitter, not
  until `Finalize`.

`WriteSegment` also reserves an index with a negative-size placeholder and
rolls the reservation back if encryption, signing or the archive write
fails, so a failed attempt cannot leave a placeholder that blocks a retry
or that `Finalize` mistakes for a written segment. Release matches on
pointer identity and on the placeholder still being unwritten, so it can
never discard a segment another call has since completed.

Deferred to the next PR: the `GetManifest`-splits-under-RLock question,
which is a behavior change to an exported method rather than a hole in
what lands here.

Tests cover round-trip, out-of-order and sparse writes, segment trimming,
KAO shape for RSA and EC, legacy and current target modes, assertion
signing, deterministic ZIP timestamps, the error contracts above,
archive-failure rollback and retry, and concurrent `WriteSegment` calls to
distinct and to duplicate indices under `-race`.

Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>
@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor
Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 175.955805ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 97.187474ms

Benchmark Statistics

Name № Requests Avg Duration Min Duration Max Duration

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 311.78007ms
Throughput 320.74 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 43.225708394s
Average Latency 431.248262ms
Throughput 115.67 requests/second

@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor
Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 246.874534ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 139.915779ms

Benchmark Statistics

Name № Requests Avg Duration Min Duration Max Duration

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 408.714078ms
Throughput 244.67 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 59.598580258s
Average Latency 594.738255ms
Throughput 83.89 requests/second

@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

X-Test Failure Report

@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor
Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 215.170249ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 125.417313ms

Benchmark Statistics

Name № Requests Avg Duration Min Duration Max Duration

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 443.633304ms
Throughput 225.41 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 55.899261014s
Average Latency 557.644184ms
Throughput 89.45 requests/second

@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor
Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 219.095027ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 129.290161ms

Benchmark Statistics

Name № Requests Avg Duration Min Duration Max Duration

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 450.308022ms
Throughput 222.07 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 58.027851465s
Average Latency 578.78608ms
Throughput 86.17 requests/second

@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

⚠️ Govulncheck found vulnerabilities ⚠️

The following modules have known vulnerabilities:

  • otdfctl
  • service
  • tests-bdd

See the workflow run for details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp:sdk A software development kit, including library, for client applications and inter-service communicati size/xl

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant