Skip to content

The native memory usage log understates untracked memory while a pool is overcommitted #6260

Description

@andygrove

Describe the bug

The executor's memory usage log prints allocated and reserved, and the tuning guide tells users to read the difference as native memory that no pool tracks. The container warning computes the same difference and adds Spark's off-heap usage to it (CometExecIterator.scala#L557-L575). reserved is the sum of the pools' reserved(). Since #6128 that total also includes bytes a grow recorded without a grant from Spark, the overcommit (spark_memory.rs#L55-L67). Those bytes are real allocations, but they get subtracted as if Spark were accounting for them, and they aren't in Spark's off-heap usage either. So while any pool is overcommitted, the untracked figure and the container warning both come out lower by the amount of the overcommit.

Expected behavior

The log subtracts only what Spark has actually granted, or reports the overcommit on its own line, so memory recorded without a grant still counts toward the container footprint.

Additional context

The overcommit is usually small and short-lived, so this is minor. #6250 changes the same formula, so the two should be reconciled.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:memoryMemory pools, reservations, OOM handlingbugSomething isn't workingrequires-triage

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions