aws.lambda.enhanced.out_of_memory not emitted when Lambda reaches memory limit but AWS reports as timeout

[Hamdi-Hassan-seccl]

Hi Team

We are investigating an incident where a Lambda reached its memory limit but our Datadog monitor based on aws.lambda.enhanced.out_of_memory did not trigger.

We tried reproducing a similar case in sandbox Lambda. Enhanced metrics are working generally as expected as we can see that aws.lambda.enhanced.invocations and aws.lambda.enhanced.max_memory_used.
However, the `out_of_memory one remains no-data even when Lambda memory usage reaches the configured limit. ( see screenshot attached below)

Example CloudWatch output we've observed in when reproduced the issue:

INIT_REPORT Init Duration: 180000.45 ms Phase: invoke Status: timeout
REPORT RequestId: 514789fd-5e24-49c5-a41c-24c049a36e2a
Duration: 180000.00 ms
Billed Duration: 180000 ms
Memory Size: 192 MB
Max Memory Used: 192 MB
Status: timeout

We also saw similar tests where memory was very close to the limit:

Memory Size: 128 MB
Max Memory Used: 127 MB
Status: timeout
and:
Memory Size: 256 MB
Max Memory Used: 252–254 MB

In these cases, Datadog emitted aws.lambda.enhanced.max_memory_used, but did not emit aws.lambda.enhanced.out_of_memory.
The Datadog extension appears to be running and subscribed to telemetry:

TELEMETRY Name: datadog-agent State: Already subscribed Types: [Platform, Extension, Function]
EXTENSION Name: datadog-agent State: Ready Events: [INVOKE, SHUTDOWN]

### Question
Is aws.lambda.enhanced.out_of_memory metric expected to emit only for explicit runtime OOM signatures, such as:

Runtime exited with error: signal: killed
Runtime.ExitError
OutOfMemoryError
JavaScript heap out of memory

Or should it also emit when the Lambda REPORT line shows memory saturation, for example:

Memory Size: 192 MB
Max Memory Used: 192 MB
Status: timeout

From an operational perspective, this looks like memory exhaustion leading to timeout, but the OOM metric does not emit.
Could you confirm whether this is expected behaviour?
Environment variables used:

Runtime: nodejs22.x
Datadog Lambda Extension version: Datadog-Extension-ARM v92
Architecture: arm64
Region: eu-west-1

[lym953]

Thanks for the report!

OOMs are tricky to capture, and here's how we currently capture OOMs for each runtime:

• Node: runtime-specific log message:
  • JavaScript heap out of memory, or
  • Runtime exited with error: signal: killed
• Java: runtime-specific log message:
  • java.lang.OutOfMemoryError
• .NET: runtime-specific log message:
  • OutOfMemoryException
• Python and Ruby: Runtime.OutOfMemory in PlatformReport event from Lambda telemetry API
• Go: max memory used == memory size

For Node, we are not also using max memory used == memory size (in addition to runtime-specific log message) to avoid double counting.

However, in your case, seems JavaScript heap out of memory or Runtime exited with error: signal: killed logs were not emitted for your function. Do you see any other symptom of OOM besides max memory used == memory size? If yes, then we can consider using that to capture OOM as well.

[Hamdi-Hassan-seccl]

Hi,
Unfortunately, no. I was also hoping to see something along the lines of signal: killed, but the only thing we can see is a timeout.

That said, your documentation suggests there may be some inconsistencies with this specific metric, and recommends reaching out here if we encounter this kind of case?