Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #109 +/- ##
=======================================
Coverage 99.87% 99.87%
=======================================
Files 16 17 +1
Lines 788 806 +18
=======================================
+ Hits 787 805 +18
Misses 1 1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Blocking sinks in rill return as soon as their outcome is known, which can happen before the entire input is
consumed and processed. That's deliberate – the sooner we get control back, the sooner we can cancel the context
to prevent the current and future work.
Such a model gives promptness, but does not expose an "everything is done" event (also called
settlement), so any work happening after the early return stays unobservable. That event might be
necessary to free a shared resource, or to safely observe side effects the callbacks produced. Simply
put, users sometimes need
sync.WaitGroup.Wait(), but for pipelines.This PR adds a Scope API that combines both the first class Context integration
and a
Waitmethod that blocks until the pipeline settles.A scope exposes two methods,
WaitandCancel. Both cancel the associated context, butWaitalsoblocks until the pipeline is settled. At least one of them must be called to ensure the underlying
context does not leak.
Settlement is opt-in.
Waitgoes wherever the code first needs the pipeline fully stopped: right after the sink, after the error handling, or in a defer, so the function returns settled. IfWaitis omitted, a scope with a deferred Cancel behaves like a normal cancellable context, which is still useful for stopping the remaining context-aware work.How it works
We say a pipeline stage is settled once it won't do any more work: it has fully consumed its input, and
every callback it started has returned.
Most stages already expose this, they just don't call it that.
Map,Filter,FromSliceand friendsclose their output channel only after they've settled, so a closed output is the settlement signal. And
it composes backwards: a settled stage means the stage before it closed its output, and so on up to the
source. Sinks were the hole – no output channel, nothing to close, no signal.
A scope's
Waitblocks until every sink attached to it settles. By the inductive argument above, that meansthat all upstream stages, and hence the entire pipeline, have also settled.
Compared with errgroup
Most Go developers know errgroup, and the scope API was designed to be similar, while staying
pipeline friendly. The table below shows both similarities and differences.
g.GoWaitWaitWaitWaitcan be calledWaitcancels the contextWaitthat follows the sink call, or a deferredCancelCompatibility
Existing call sites keep working, since the options are variadic. But every sink's signature changed, so
code that assigns a sink to an exact function type has to be updated. Technically, this is a breaking
API change.