Adds `ChunkedWriter`, a TDF creation path that accepts segments in any
order. Callers encrypt and upload each segment independently -- typically
off-thread or in parallel -- then call `Finalize` to get the ZIP closing
bytes. Contrast with `SDK.CreateTDF`, which needs the whole plaintext up
front behind an `io.ReadSeeker`.
This is the first half of DSPX-2604. It adds the implementation only; the
public face of out-of-order writing stays `sdk/experimental/tdf`, which is
rewired onto this writer in a later PR. Graduating `ChunkedWriter` to
supported API -- dropping the `Experimental:` markers and adding the
`SDK.NewChunkedWriter` method -- is deliberately deferred to the end of the
stack.
What is exported, and why that set
Go forces a compromise here. `sdk/experimental/tdf` is a separate package,
so it cannot reach unexported symbols in `sdk`, and the writer cannot move
under `sdk/internal/` because it needs `Manifest`, `KeyAccess`, `Segment`,
`createKeyAccess` and `calculateSignature`. So: export the minimum bridge
the adapter needs, mark all of it experimental, unexport everything else.
Exported and `Experimental:`-marked: `ChunkedWriter`, `NewChunkedWriter`,
the two result structs, the two config structs and option types, the
caller-facing `WithChunked*` options, the sentinel errors, and the
`KeySplitter` / `Split` / `SplitResult` / `KASPublicKey` group that an
out-of-package splitter has to implement.
Unexported: the test seams. The clock, the segment cipher and its factory,
the archive writer factory, the entropy source, and the four options that
inject them. `archiveWriterFactory` in particular returns a
`zipstream.SegmentWriter` from `internal/`, so no external package could
have implemented it even when the type was exported -- an exported symbol
no caller can satisfy should not be exported. Those three files (`clock.go`,
`segment.go`, `archive_writer.go` -- 28, 29 and 23 lines) are folded into
`chunked_writer.go` now that nothing outside the package can see them.
Correctness guards included rather than deferred
Three small guards ship with the code they protect, since splitting "add
new code with a known hole" from "fix it" across two PRs of brand-new code
is churn with no review value:
- `Finalize` rejects a write set missing segment 0
(`ErrChunkedMissingSegmentZero`). Only segment 0 emits the payload's ZIP
local file header and every recorded offset is measured from it, so a set
without it silently produces a corrupt archive. It cannot be synthesized
at `Finalize` time -- by then the caller has already encrypted and shipped
the bytes.
- The default splitter rejects a KAS key whose algorithm has no wrapping
scheme (`ErrSplitterUnsupportedAlgorithm`) instead of letting the empty
string reach `createKeyAccess`, where it selects the RSA branch while
`ocrypto.FromPublicPEM` sniffs the PEM and wraps anyway -- producing a KAO
that claims keyType "wrapped" with no ephemeral public key, i.e. a TDF
nothing can decrypt.
- The injection-seam options reject nil rather than storing it. A stored nil
is indistinguishable from an unset field, so no default is installed and
the nil surfaces as a panic partway through -- for the key splitter, not
until `Finalize`.
`WriteSegment` also reserves an index with a negative-size placeholder and
rolls the reservation back if encryption, signing or the archive write
fails, so a failed attempt cannot leave a placeholder that blocks a retry
or that `Finalize` mistakes for a written segment. Release matches on
pointer identity and on the placeholder still being unwritten, so it can
never discard a segment another call has since completed.
Deferred to the next PR: the `GetManifest`-splits-under-RLock question,
which is a behavior change to an exported method rather than a hole in
what lands here.
Tests cover round-trip, out-of-order and sparse writes, segment trimming,
KAO shape for RSA and EC, legacy and current target modes, assertion
signing, deterministic ZIP timestamps, the error contracts above,
archive-failure rollback and retry, and concurrent `WriteSegment` calls to
distinct and to duplicate indices under `-race`.
Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>
Proposed Changes
Adds
ChunkedWriter, a TDF creation path that accepts segments in anyorder. Callers encrypt and upload each segment independently -- typically
off-thread or in parallel -- then call
Finalizeto get the ZIP closingbytes. Contrast with
SDK.CreateTDF, which needs the whole plaintext upfront behind an
io.ReadSeeker.This is the first half of DSPX-2604. It adds the implementation only; the
public face of out-of-order writing stays
sdk/experimental/tdf, which isrewired onto this writer in a later PR. Graduating
ChunkedWritertosupported API -- dropping the
Experimental:markers and adding theSDK.NewChunkedWritermethod -- is deliberately deferred to the end of thestack.
What is exported, and why that set
Go forces a compromise here.
sdk/experimental/tdfis a separate package,so it cannot reach unexported symbols in
sdk, and the writer cannot moveunder
sdk/internal/because it needsManifest,KeyAccess,Segment,createKeyAccessandcalculateSignature. So: export the minimum bridgethe adapter needs, mark all of it experimental, unexport everything else.
Exported and
Experimental:-marked:ChunkedWriter,NewChunkedWriter,the two result structs, the two config structs and option types, the
caller-facing
WithChunked*options, the sentinel errors, and theKeySplitter/Split/SplitResult/KASPublicKeygroup that anout-of-package splitter has to implement.
Unexported: the test seams. The clock, the segment cipher and its factory,
the archive writer factory, the entropy source, and the four options that
inject them.
archiveWriterFactoryin particular returns azipstream.SegmentWriterfrominternal/, so no external package couldhave implemented it even when the type was exported -- an exported symbol
no caller can satisfy should not be exported. Those three files (
clock.go,segment.go,archive_writer.go-- 28, 29 and 23 lines) are folded intochunked_writer.gonow that nothing outside the package can see them.Correctness guards included rather than deferred
Three small guards ship with the code they protect, since splitting "add
new code with a known hole" from "fix it" across two PRs of brand-new code
is churn with no review value:
Finalizerejects a write set missing segment 0(
ErrChunkedMissingSegmentZero). Only segment 0 emits the payload's ZIPlocal file header and every recorded offset is measured from it, so a set
without it silently produces a corrupt archive. It cannot be synthesized
at
Finalizetime -- by then the caller has already encrypted and shippedthe bytes.
scheme (
ErrSplitterUnsupportedAlgorithm) instead of letting the emptystring reach
createKeyAccess, where it selects the RSA branch whileocrypto.FromPublicPEMsniffs the PEM and wraps anyway -- producing a KAOthat claims keyType "wrapped" with no ephemeral public key, i.e. a TDF
nothing can decrypt.
is indistinguishable from an unset field, so no default is installed and
the nil surfaces as a panic partway through -- for the key splitter, not
until
Finalize.WriteSegmentalso reserves an index with a negative-size placeholder androlls the reservation back if encryption, signing or the archive write
fails, so a failed attempt cannot leave a placeholder that blocks a retry
or that
Finalizemistakes for a written segment. Release matches onpointer identity and on the placeholder still being unwritten, so it can
never discard a segment another call has since completed.
Deferred to the next PR: the
GetManifest-splits-under-RLock question,which is a behavior change to an exported method rather than a hole in
what lands here.
Tests cover round-trip, out-of-order and sparse writes, segment trimming,
KAO shape for RSA and EC, legacy and current target modes, assertion
signing, deterministic ZIP timestamps, the error contracts above,
archive-failure rollback and retry, and concurrent
WriteSegmentcalls todistinct and to duplicate indices under
-race.Checklist
Testing Instructions
sdk/chunked_test.gois the bulk of the diff. The concurrency cases(
WriteSegmentto distinct and to duplicate indices) are the ones that want-racespecifically.The full DSPX-2604 stack — 20 PRs
mainmainmainmainmainmainmaindspx-2604-base-11= #3932 + #3934 + #3935dspx-2604-base-17= #3944 + #3945dspx-2604-base-19= #3947 + #3939Reviewable in parallel right now, since they sit directly on
mainand depend onnothing else: 01, 02, 04, 05, 06, 07, 08.
Why three PRs have a
dspx-2604-base-*base. A GitHub PR takes one base branch,but 11, 17 and 19 each build on more than one parent. The
base-*branches are emptymerge commits that exist only to join those parents so the PR diff shows exactly its
own change and nothing else. They contain no code, have no PR of their own, and go
away once their parents land — retarget the child onto
mainat that point.Wants a cross-SDK xtest run before merge: 15, 17 (and therefore 20). They touch
the KAS wire format.
Red checks you may see are network flakes, not this stack. Four distinct ones hit
this batch and all clear on re-run:
golangci-lint config verifytiming out onhttps://golangci-lint.run/.../golangci.v2.8.jsonschema.json(fails the wholego (<module>)job and fail-fast cancels its siblings), the bats installer getting a 403,Docker Hub timing out on
keycloak/keycloak:26.4, andbufreporting "the serverhosted at that remote is unavailable" while the Java SDK generates sources. The
govulncheckstep also emits##[error]annotations against the go1.25.11 stdlib, butit is
continue-on-error: trueand never fails a job — 01 bumps the toolchain andclears those annotations.