Skip to content

CometTaskMemoryManager logs a warning and a memory dump every time a native reservation is refused #6257

Description

@andygrove

What is the problem the feature request solves?

Native operators ask for memory with try_grow until they are refused, and then they spill. A partial grant from Spark is the normal spill signal, not an error. But CometTaskMemoryManager.acquireMemory logs a warning and calls TaskMemoryManager.showMemoryUsage() every time Spark grants less than was asked (CometTaskMemoryManager.java#L74-L85), and showMemoryUsage() logs another line per consumer at INFO. In a four-task aggregate with 96 MB of off-heap memory, that was 257 warnings in about five seconds. A query that spills heavily keeps producing them for its whole run. That buries real warnings and makes a healthy spill look like a failure.

Describe the potential solution

Log the partial grant at DEBUG, or at most once per task at INFO. Nothing is lost when a reservation really does fail: the error the pool returns already says how much was granted and lists the top consumers.

Additional context

No response

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions