> **Part 08 of 20** in the DSPX-2604 re-cut. Base branch: `main`.
>
> This stack replaces #3782 / #3865 / #3921, which stay open and
untouched
> until it lands. Nothing here is a rebase of those branches — the work
was
> re-cut from the ticket so each PR stands on its own.
### Proposed Changes
Adds otdfctl/pkg/streamio, holding the input and output plumbing that
the
streaming encrypt and decrypt work needs, and migrates `inspect` onto it
so
nothing is left calling the buffered helpers it supersedes.
This is groundwork with one user-visible consequence: `inspect` no
longer
reads the whole TDF into memory. Everything else is a move.
Why a new package rather than pkg/cli. The helpers in pkg/cli/pipe.go
call
ExitWithError -- which calls os.Exit -- from inside the read, so they
cannot
be used from anywhere that wants to handle the failure itself, and they
read
the entire input into memory. streamio returns errors and leaves the
decision
to exit with the command layer.
What moved in:
- PipeReader establishes whether stdin is a non-empty pipe with a
one-byte
Peek instead of a read, so the payload still reaches the caller.
- Spool copies a pipe to a temporary file and rewinds it. A TDF's
manifest
sits at the end of the archive, so decrypt and inspect have to seek and
cannot consume a pipe directly.
- OpenSeekable resolves "file argument or piped stdin" to one seekable
handle, reporting ErrNoInput for the shared "nothing to read" case.
- OutputFile writes to a temporary sibling of the destination and
renames it
into place on Commit, so a failed run leaves no partial output. The temp
file is a sibling so the rename stays atomic rather than degrading to a
cross-filesystem copy.
Per review feedback on #3921:
- readPipedStdin now delegates its detection to streamio.PipeReader
rather
than answering "is there piped input?" a second way. Its read is still
unbounded; the callers that must stop buffering are changed separately.
- pkg/cli/pipe.go is deprecated rather than deleted, since the package
is
exported and may have callers outside this repository. Worth noting that
ReadFromFile has no size cap at all -- not even the 10 GB the tdf
commands
apply -- which is its own argument for the notice.
InspectTDF takes an io.ReadSeeker instead of a byte slice. GetTdfType
already
rewinds to the start, so the reader is positioned for LoadTDF. Because
cli.ExitWithError calls os.Exit and skips deferred functions, inspectRun
invokes cleanup explicitly on every exit path, including the successful
one:
piped input is spooled to disk and the temp file would otherwise
survive.
### Checklist
- [x] I have added or updated unit tests
- [ ] I have added or updated integration tests (if appropriate)
- [ ] I have added or updated documentation
### Testing Instructions
```
cd otdfctl && go test ./pkg/streamio/... ./cmd/... -race
```
`inspect` is the only command migrated in this PR; check it still reads
both a
file argument and piped stdin, and that no `otdfctl-spool-*` file
survives
either run.
<details>
<summary><b>The full DSPX-2604 stack — 20 PRs</b></summary>
| # | PR | Based on |
|---|----|----------|
| 01 | #3930 chore: bump go.work toolchain to go1.25.12 and simplify an
rt_test condition | `main` |
| 02 | #3931 feat(sdk): make the zipstream clock injectable for
deterministic ZIP output | `main` |
| 03 | #3932 fix(sdk): reject a zipstream write set that omits segment 0
| #3931 |
| 04 | #3933 fix(sdk): map ReadAt plaintext offsets from cumulative
segment sizes | `main` |
| 05 | #3934 chore(sdk): extract integrityAlgorithmString,
createPolicyBinding, signAssertions | `main` |
| 06 | #3935 chore(sdk): add direct tests for createKeyAccess,
encryptMetadata and tdfSalt | `main` |
| 07 | #3936 fix(sdk): fill each segment with io.ReadFull and size the
buffer to the input | `main` |
| 08 | #3937 chore(cli): move streaming IO helpers into pkg | `main` |
| 09 | #3938 fix(cli): stream encrypt instead of buffering the whole
payload | #3937 |
| 10 | #3939 fix(cli): stream decrypt and inspect instead of buffering |
#3938 |
| 11 | #3940 feat(sdk): add a chunked segment writer (experimental) |
`dspx-2604-base-11` = #3932 + #3934 + #3935 |
| 12 | #3941 fix(sdk): stop GetManifest from splitting the key under the
lock | #3940 |
| 13 | #3942 fix(sdk): reject a chunked split naming a KAS with no
resolved public key | #3941 |
| 14 | #3943 chore(sdk): alias experimental/tdf manifest and assertion
types | #3942 |
| 15 | #3944 fix(sdk): emit spec-compliant key access in
experimental/tdf and delegate Writer | #3943 |
| 16 | #3945 feat(sdk): accept io.Reader in CreateTDF and drop the 64 GB
payload cap | #3936 |
| 17 | #3946 chore(sdk): rewrite CreateTDF on top of the chunked writer
| `dspx-2604-base-17` = #3944 + #3945 |
| 18 | #3947 chore(sdk): drop dead TDFConfig fields and deprecate the
TDFFormat enum | #3946 |
| 19 | #3948 fix(cli): drop the encrypt-side stdin spool |
`dspx-2604-base-19` = #3947 + #3939 |
| 20 | #3949 feat(sdk): graduate the chunked writer to stable API |
#3948 |
**Reviewable in parallel right now**, since they sit directly on `main`
and depend on
nothing else: 01, 02, 04, 05, 06, 07, 08.
**Why three PRs have a `dspx-2604-base-*` base.** A GitHub PR takes one
base branch,
but 11, 17 and 19 each build on more than one parent. The `base-*`
branches are empty
merge commits that exist only to join those parents so the PR diff shows
exactly its
own change and nothing else. They contain no code, have no PR of their
own, and go
away once their parents land — retarget the child onto `main` at that
point.
**Wants a cross-SDK xtest run before merge:** 15, 17 (and therefore 20).
They touch
the KAS wire format.
**Red checks you may see are network flakes, not this stack.** Four
distinct ones hit
this batch and all clear on re-run: `golangci-lint config verify` timing
out on
`https://golangci-lint.run/.../golangci.v2.8.jsonschema.json` (fails the
whole `go
(<module>)` job and fail-fast cancels its siblings), the bats installer
getting a 403,
Docker Hub timing out on `keycloak/keycloak:26.4`, and `buf` reporting
"the server
hosted at that remote is unavailable" while the Java SDK generates
sources. The
`govulncheck` step also emits `##[error]` annotations against the
go1.25.11 stdlib, but
it is `continue-on-error: true` and never fails a job — 01 bumps the
toolchain and
clears those annotations.
</details>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added reliable support for inspecting TDF content from files, piped
input, and standard input.
- Added safer output handling that prevents incomplete files from
replacing existing results.
- Added clearer input errors when no content is provided or an input
cannot be opened.
- **Bug Fixes**
- Improved handling of large and non-seekable input streams.
- Preserved piped input correctly while processing and inspecting
content.
- Non-fatal inspection issues are now reported as warnings where
possible, allowing processing to continue.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>
Proposed Changes
sdk/experimental/tdfcarried its own copies of the manifest and assertiontypes --
Manifest,Segment,KeyAccess,Assertion,Statement,AssertionKeyand the rest -- structurally identical to the ones insdkbut distinct to the type system, so anything crossing the boundary needed
conversion. Two copies of the JWT signing and verification logic also had
to be kept in step by hand.
Replaces both files' definitions with type aliases.
sdkowns thedefinitions; this package re-exports them. Every exported name and every
method survives:
Assertion.Sign/Verify/GetHash,Statement.UnmarshalJSON,AssertionKey.IsEmpty/Algorithm,AssertionVerificationKeys.Get/IsEmptyand the fiveString()methodsall come along with the aliased types, so importers compile unchanged. A
manifest produced here can now be handed to the stable SDK without
conversion, which is what the follow-up delegation needs.
Two deliberate non-aliases:
Policy,PolicyBodyandPolicyAttributestay local.sdk.PolicyObjectdeclares
Bodyas an anonymous struct over an unexported element type, sothere is no nameable sdk equivalent to alias to. Exporting those in
sdkfirst would make the alias possible; that is a separate change.
IntegrityAlgorithmstays a distinctinttype.sdk.IntegrityAlgorithmis itself
= int, so no method can be attached to it, and aliasing wouldsilently drop
String()from this package's public API. The underlyingvalues match, so the two convert freely.
kSplitKeyType,kPolicyBindingAlg,kGMACPayloadLengthandcalculateSignatureare retained verbatim: this package still builds itsown manifests and they have callers in
writer.goandkey_access.go.The change that removes those callers removes these too.
No behavior change. The package's existing tests pass unmodified.
Checklist
Testing Instructions
No behavior change — the point is that the package's existing tests pass
unmodified against aliased types.
The full DSPX-2604 stack — 20 PRs
mainmainmainmainmainmainmaindspx-2604-base-11= #3932 + #3934 + #3935dspx-2604-base-17= #3944 + #3945dspx-2604-base-19= #3947 + #3939Reviewable in parallel right now, since they sit directly on
mainand depend onnothing else: 01, 02, 04, 05, 06, 07, 08.
Why three PRs have a
dspx-2604-base-*base. A GitHub PR takes one base branch,but 11, 17 and 19 each build on more than one parent. The
base-*branches are emptymerge commits that exist only to join those parents so the PR diff shows exactly its
own change and nothing else. They contain no code, have no PR of their own, and go
away once their parents land — retarget the child onto
mainat that point.Wants a cross-SDK xtest run before merge: 15, 17 (and therefore 20). They touch
the KAS wire format.
Red checks you may see are network flakes, not this stack. Four distinct ones hit
this batch and all clear on re-run:
golangci-lint config verifytiming out onhttps://golangci-lint.run/.../golangci.v2.8.jsonschema.json(fails the wholego (<module>)job and fail-fast cancels its siblings), the bats installer getting a 403,Docker Hub timing out on
keycloak/keycloak:26.4, andbufreporting "the serverhosted at that remote is unavailable" while the Java SDK generates sources. The
govulncheckstep also emits##[error]annotations against the go1.25.11 stdlib, butit is
continue-on-error: trueand never fails a job — 01 bumps the toolchain andclears those annotations.
Summary by CodeRabbit