Skip to content

feat(config, conversation, cli): Compact only oversized items - #994

Open
JeanMertz wants to merge 5 commits into
mainfrom
jp-c-compact-events
Open

feat(config, conversation, cli): Compact only oversized items#994
JeanMertz wants to merge 5 commits into
mainfrom
jp-c-compact-events

Conversation

@JeanMertz

Copy link
Copy Markdown
Collaborator

A compaction rule reaches every item in its turn range. That is the
right default for old turns, but it is the wrong tool for the case that
actually breaks a conversation: one fs_read_file that returned a 10 MB
log and now dominates the context window. Cutting it today means either
compacting a whole range of turns you wanted to keep, or editing
events.json by hand.

The reasoning and tool-call policies now accept an over size
threshold, so a rule reaches only the items large enough to matter:

jp conversation compact --tools=sres --over 1mb
jp query -k 'r,over=16kb+t=sres,over=1mb:..-3'

[[conversation.compaction.rules]]
tool_calls = { policy = "strip-responses", over = "1MB" }

Both surfaces compose with the existing range flags, so a threshold
narrows what a rule touches rather than replacing how it is scoped. This
also enables a tiered rule set: drop anything genuinely huge wherever it
sits, and strip everything older than the last few turns regardless of
size.

The threshold is stored in the compaction event and evaluated during
projection rather than resolved into a fixed set of matched items at
creation. Size comparison is cheap and deterministic, so it is a lazy
policy in RFD 064's terms, and projection stays a pure function of the
stored stream.

Sizes are judged per half for Strip, so a call with a short request
and a huge response loses only the response. Omit removes whole pairs,
so it is judged on the two halves combined and can never leave an
orphaned request behind. Measurement uses decoded content (tool
arguments are base64 at rest) against the raw stream, so re-running a
rule reaches the same items. summary takes no threshold: it replaces
its whole range rather than selecting from it.

Because a threshold reaches an unpredictable subset of its range, the
timeline names what it caught instead of leaving the range line to imply
everything was compacted:

Would have compacted turns 2..14 (13 total, tool responses over 1MB).
  turn 4  fs_read_file (response)  10.3 MB
  turn 9  cargo_test (response)  2.1 MB

A rule without a threshold does not itemize, and a threshold that
matched nothing says so.

Nothing changes for a rule that sets no threshold, on disk or in
behavior: a policy without one serializes as the bare string it always
was, and the built-in defaults carry no threshold.

BREAKING CHANGE: reasoning in jp conversation show -F json

The reasoning field of a compaction entry was a boolean; it is now the
serialized policy (null, "strip", or {"policy": "strip", "over": "16KB"}), matching how tool_calls already reported itself. Without
this a consumer could read tool_calls.over but had no way to see a
reasoning threshold.

Scripts testing truthiness need updating: .reasoning == true becomes
.reasoning != null.

Two building blocks for narrowing a compaction policy to the large items
in its range, neither of which has a consumer yet.

`ByteSize` is a size in bytes written as a human-readable string (`1MB`,
`512 KB`, `4GiB`) or a bare byte count. Unit suffixes are binary, so
`1KB` is 1024 bytes. Its `Display` picks the largest unit that divides
the value evenly and falls back to the raw count otherwise, which makes
the rendered form always parse back to the same value and therefore safe
to use as the serialized form. A separate `human()` gives the lossy
one-decimal rendering for terminal output.

`PolicySpec<P>` pairs a policy with the options qualifying which items it
reaches, today just an `over` size threshold. Declaring it once as a
generic wrapper keeps the option off each individual policy enum, so a
future policy cannot forget to honor it.

The serialized shape is the part worth attention: a spec without options
serializes as the bare policy, so adding this changes no existing config
file or event stream on disk. Only a spec that actually carries a
threshold takes the table form, where a map-shaped policy gains `over`
as a sibling key and a string-shaped one is promoted to
`{"policy": "...", "over": "..."}`.

Deserialization tries the whole map against `P` before falling back to
the promoted string. The order is load-bearing: a tagged unit variant
such as `{"policy": "omit"}` is indistinguishable from a promotion by
shape alone, and only the map reading is correct for it.

`FromStr` accepts `POLICY,option=value` so a `--cfg` assignment and the
inline compaction DSL can share one option syntax.

Signed-off-by: Jean Mertz <git@jeanmertz.com>
A compaction rule reaches every item in its turn range. That is the
right default for old turns, but it is the wrong tool for the case that
actually breaks a conversation: one `fs_read_file` that returned a 10 MB
log and now dominates the context window. Cutting it today means either
compacting a whole range of turns you wanted to keep, or editing
`events.json` by hand.

The reasoning and tool-call policies now accept an `over` size
threshold, so a rule reaches only the items large enough to matter:

    jp conversation compact --tools=sres --over 1mb
    jp query -k 'r,over=16kb+t=sres,over=1mb:..-3'

    [[conversation.compaction.rules]]
    tool_calls = { policy = "strip-responses", over = "1MB" }

Both surfaces compose with the existing range flags, so a threshold
narrows what a rule touches rather than replacing how it is scoped. This
also enables a tiered rule set: drop anything genuinely huge wherever it
sits, and strip everything older than the last few turns regardless of
size.

The threshold is stored in the compaction event and evaluated during
projection rather than resolved into a fixed set of matched items at
creation. Size comparison is cheap and deterministic, so it is a lazy
policy in RFD 064's terms, and projection stays a pure function of the
stored stream.

Sizes are judged per half for `Strip`, so a call with a short request
and a huge response loses only the response. `Omit` removes whole pairs,
so it is judged on the two halves combined and can never leave an
orphaned request behind. Measurement uses decoded content (tool
arguments are base64 at rest) against the raw stream, so re-running a
rule reaches the same items. `summary` takes no threshold: it replaces
its whole range rather than selecting from it.

Because a threshold reaches an unpredictable subset of its range, the
timeline names what it caught instead of leaving the range line to imply
everything was compacted:

    Would have compacted turns 2..14 (13 total, tool responses over 1MB).
      turn 4  fs_read_file (response)  10.3 MB
      turn 9  cargo_test (response)  2.1 MB

A rule without a threshold does not itemize, and a threshold that
matched nothing says so.

Nothing changes for a rule that sets no threshold, on disk or in
behavior: a policy without one serializes as the bare string it always
was, and the built-in defaults carry no threshold.

BREAKING CHANGE: `reasoning` in `jp conversation show -F json`

The `reasoning` field of a compaction entry was a boolean; it is now the
serialized policy (`null`, `"strip"`, or `{"policy": "strip", "over":
"16KB"}`), matching how `tool_calls` already reported itself. Without
this a consumer could read `tool_calls.over` but had no way to see a
reasoning threshold.

Scripts testing truthiness need updating: `.reasoning == true` becomes
`.reasoning != null`.

Signed-off-by: Jean Mertz <git@jeanmertz.com>
Review follow-ups on the compaction size threshold, the first two of
which are real defects.

A tool call ID is not unique. `TurnMut::build` documents that providers
like Google reuse one synthetic ID across streaming cycles within a
turn, and validates responses by count for exactly that reason. Keying
the size map by ID gave every occurrence of an ID the last one's sizes,
so `--tools=sres --over 1mb` on a turn holding a 4 MB `tc1` response
followed by a 2-byte `tc1` response kept the 4 MB one and reported no
match. The oversized payload the threshold exists to remove then reached
every later query, silently.

Sizes are now keyed by stream position, and a response pairs with the
oldest request in its turn carrying the same ID that has no response
yet, mirroring the rule `TurnMut::build` already applies. That also
fixes the tool name on a stripped response for a reused ID, which had
the same weakness.

A misspelled option key was silently dropped. `tool_calls = { policy =
"strip-responses", oer = "1MB" }` deserialized to an unthresholded rule
and stripped every response in range, the opposite of what the line
asks for. Schematic emits `deny_unknown_fields` for the surrounding
config types, so `PolicySpec` was the one place in this tree accepting
junk. The promoted-string form now rejects any leftover key, naming it.
A policy that legitimately serializes as a map with fields of its own is
untouched, since it consumes the map itself.

A summary rule no longer itemizes. `-k 's+r,over=1kb'` parses, and the
threshold rides along on the rule, but a summary replaces its whole
range and projection never consults the threshold. The timeline listed
items nothing had selected, under a label that read `summary`.

The generated JSON Schema describes the table form field by field rather
than as an unconstrained value, so a schema consumer still validates
`policy` and still rejects an unknown key. The `tool_calls` and
`reasoning` doc comments list the modes they accept and say what
omitting them does, which is what a reader scanning a generated
`config.toml` needs.

A test pins the stacking interaction the threshold makes visible: a
later thresholded overlay replaces an earlier policy for its whole turn,
so an item below the threshold is left raw rather than falling back to
the earlier, broader policy. That follows from latest-wins resolution
and is unchanged by this feature, but it was untested at item
granularity.

Signed-off-by: Jean Mertz <git@jeanmertz.com>
Signed-off-by: Jean Mertz <git@jeanmertz.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant