Skip to content

Grouped integer SUM doesn't count its per-group state in the aggregate's memory reservation #6252

Description

@andygrove

Describe the bug

The grouped accumulators behind integer SUM report the size of their struct instead of the state they hold. SumIntGroupsAccumulatorLegacy, SumIntGroupsAccumulatorAnsi and SumIntGroupsAccumulatorTry all return std::mem::size_of_val(self) from GroupsAccumulator::size() (sum_int.rs#L534, #L687, #L895). That is the Vec header, not the sums: Vec<Option<i64>> it points to, so each one leaves out 16 bytes per group. The Try variant also leaves out has_all_nulls.

DataFusion sizes a hash aggregate's reservation from its accumulators' size() plus the group values (AggregateHashTable::memory_size in datafusion-physical-plan 55.1.0). A grouped aggregate with integer sums therefore reserves less than it holds, spills later than it should, and can take the executor past spark.memory.offHeap.size without the pool noticing. Every SUM over Int8, Int16, Int32 or Int64 goes through these accumulators (planner.rs#L2869-L2873). For example, four integer sums over 10M groups hold about 640 MB that the aggregate never reserves.

The decimal accumulator already gets this right (sum_decimal.rs#L618-L621).

Steps to reproduce

Run update_batch on one of these accumulators over 1M distinct group indices, then call size(). It stays at the struct size instead of growing past 16 MB. I found this by reading the code and haven't measured the effect on a query.

Expected behavior

size() includes the capacity of sums, and of has_all_nulls for the Try variant, the way SumDecimalGroupsAccumulator::size() does.

Additional context

These accumulators date from #2600 and #3054, so 1.0.0 and 1.1.0 are affected.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:aggregationHash aggregates, aggregate expressionsarea:memoryMemory pools, reservations, OOM handlingbugSomething isn't workingrequires-triage

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions