What is the problem the feature request solves?
Native operators ask for memory with try_grow until they are refused, and then they spill. A partial grant from Spark is the normal spill signal, not an error. But CometTaskMemoryManager.acquireMemory logs a warning and calls TaskMemoryManager.showMemoryUsage() every time Spark grants less than was asked (CometTaskMemoryManager.java#L74-L85), and showMemoryUsage() logs another line per consumer at INFO. In a four-task aggregate with 96 MB of off-heap memory, that was 257 warnings in about five seconds. A query that spills heavily keeps producing them for its whole run. That buries real warnings and makes a healthy spill look like a failure.
Describe the potential solution
Log the partial grant at DEBUG, or at most once per task at INFO. Nothing is lost when a reservation really does fail: the error the pool returns already says how much was granted and lists the top consumers.
Additional context
No response
What is the problem the feature request solves?
Native operators ask for memory with
try_growuntil they are refused, and then they spill. A partial grant from Spark is the normal spill signal, not an error. ButCometTaskMemoryManager.acquireMemorylogs a warning and callsTaskMemoryManager.showMemoryUsage()every time Spark grants less than was asked (CometTaskMemoryManager.java#L74-L85), andshowMemoryUsage()logs another line per consumer at INFO. In a four-task aggregate with 96 MB of off-heap memory, that was 257 warnings in about five seconds. A query that spills heavily keeps producing them for its whole run. That buries real warnings and makes a healthy spill look like a failure.Describe the potential solution
Log the partial grant at DEBUG, or at most once per task at INFO. Nothing is lost when a reservation really does fail: the error the pool returns already says how much was granted and lists the top consumers.
Additional context
No response