Describe the bug
The executor's memory usage log prints allocated and reserved, and the tuning guide tells users to read the difference as native memory that no pool tracks. The container warning computes the same difference and adds Spark's off-heap usage to it (CometExecIterator.scala#L557-L575). reserved is the sum of the pools' reserved(). Since #6128 that total also includes bytes a grow recorded without a grant from Spark, the overcommit (spark_memory.rs#L55-L67). Those bytes are real allocations, but they get subtracted as if Spark were accounting for them, and they aren't in Spark's off-heap usage either. So while any pool is overcommitted, the untracked figure and the container warning both come out lower by the amount of the overcommit.
Expected behavior
The log subtracts only what Spark has actually granted, or reports the overcommit on its own line, so memory recorded without a grant still counts toward the container footprint.
Additional context
The overcommit is usually small and short-lived, so this is minor. #6250 changes the same formula, so the two should be reconciled.
Describe the bug
The executor's memory usage log prints
allocatedandreserved, and the tuning guide tells users to read the difference as native memory that no pool tracks. The container warning computes the same difference and adds Spark's off-heap usage to it (CometExecIterator.scala#L557-L575).reservedis the sum of the pools'reserved(). Since #6128 that total also includes bytes agrowrecorded without a grant from Spark, the overcommit (spark_memory.rs#L55-L67). Those bytes are real allocations, but they get subtracted as if Spark were accounting for them, and they aren't in Spark's off-heap usage either. So while any pool is overcommitted, the untracked figure and the container warning both come out lower by the amount of the overcommit.Expected behavior
The log subtracts only what Spark has actually granted, or reports the overcommit on its own line, so memory recorded without a grant still counts toward the container footprint.
Additional context
The overcommit is usually small and short-lived, so this is minor. #6250 changes the same formula, so the two should be reconciled.