Conversation
…in memory Adds a Stateless Content Storage Location (INHERITED, CONTENT_REPOSITORY, IN_MEMORY) to Process Groups so that a Process Group running on the Stateless Execution Engine can buffer FlowFile content in memory instead of writing to the Content Repository. - Bump nifi-api to 2.12.0-SNAPSHOT (StatelessContentStorageLocation enum and VersionedProcessGroup.statelessContentStorageLocation). - Thread the value through the ProcessGroup interface, StandardProcessGroup (get/set/resolve/verify), DTO, DtoFactory, DAO, flow mapping and flow synchronization, mirroring executionEngine. - Defer the Stateless engine's ContentRepository selection until the group starts via DeferredStatelessContentRepository, so the configured location is honored (the group node is created before configuration is applied). When resolved to IN_MEMORY, a ByteArrayContentRepository buffers content in memory; otherwise the NiFi Content Repository is used. - Restrict IN_MEMORY to Process Groups that are disconnected from all other components so content never crosses repositories: reject setting IN_MEMORY when the group has incoming/outgoing connections, and reject connecting into or out of a group configured to buffer content in memory. - Add the FlowFile Content Storage selector to the edit-process-group dialog, shown when the Execution Engine is Stateless. - Add StandardProcessGroup unit tests and StatelessInMemoryContentIT system tests (in-memory processing with an on-disk Content Repository scan proving nothing is written to disk, plus the connection and inheritance constraints). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
NIFI-16271
Allows a Stateless Process Group to buffer FlowFile content in memory up to a data-size cap and/or a heap-percentage cap, then spill to the Content Repository. Content that leaves the group is written to on-disk claims before the FlowFile Repository is updated.
0%heap and an unset data size, which uses no in-memory content.Draft: this PR depends on
nifi-api2.12.0-SNAPSHOTfrom apache/nifi-api#108. It cannot merge until that API change is released.Test plan
StatelessInMemoryContentITfor under-budget in-memory content, spill-over, and rejecting limit changes while running./mvnw clean install -P contrib-check(JDK 21 / JDK 25) after the API snapshot is available in CI