HeadNode-ClustermgtdHeartbeat alarm stays in ALARM state when HeadNode OS timezone is not UTC

[Kohei-Kuniyoshi-AGC]

Describe the issue

When the HeadNode's OS timezone is set to a non-UTC timezone (e.g., Asia/Tokyo / JST), the HeadNode-ClustermgtdHeartbeat CloudWatch alarm constantly remains in the ALARM (or INSUFFICIENT_DATA) state.

To Reproduce

1. Create a cluster with AWS ParallelCluster 3.15.1.
2. Set the OS timezone of the HeadNode to Asia/Tokyo (JST, +09:00).
3. Check the HeadNode-ClustermgtdHeartbeat alarm in the CloudWatch Console. It will stay in the ALARM state.

Root Cause Analysis

The clustermgtd.events log outputs timestamps in UTC with a colon in the offset, for example:
"datetime": "2026-08-18T09:30:47.592+00:00"

However, the CloudWatch Agent configuration (file_cloudwatch_agent_config.json) generated by ParallelCluster defines the timestamp format without a timezone specification:
"timestamp_format": "%Y-%m-%dT%H:%M:%S.%f"

Because the HeadNode OS is in JST, the CloudWatch Agent parses the time 09:30:47 as JST (local time) instead of UTC. This causes the metric data to be ingested with a timestamp 9 hours in the past. Since the CloudWatch Alarm evaluates only recent data points, it fails to find them and triggers the alarm.

Note: Appending %z to the timestamp_format does not work because the CloudWatch Agent only supports the %z format without a colon (e.g., +0900), and it fails to parse +00:00.

Proposed Solution

Add "timezone": "UTC" to the _events log stream definitions (e.g., clustermgtd_events, slurm_resume_events) in the CloudWatch Agent configuration template to force the agent to interpret the parsed timestamp as UTC.

{
    "log_stream_name": "<hostname>.{instance_id}.clustermgtd_events",
    "file_path": "/var/log/parallelcluster/clustermgtd.events",
    "timestamp_format": "%Y-%m-%dT%H:%M:%S.%f",
    "timezone": "UTC", 
    "log_group_name": "/aws/parallelcluster/..."
}

I have verified with AWS Support that adding "timezone": "UTC" resolves the issue successfully.

Environment:

AWS ParallelCluster Version: 3.15.1

[hanwen-cluster]

Hi @Kohei-Kuniyoshi-AGC

Thank you for reporting the issue. We have also been notified by AWS Support.

Changing the head node OS timezone away from UTC is not a tested by ParallelCluster

The impact is on observability only. It does not affect job execution and compute node lifecycle.

Workaround

In short, I confirm your suggestion of adding "timezone": "UTC".
In amazon-cloudwatch-agent.json, keep timestamp_format as is and add "timezone": "UTC" to clustermgtd.events and slurm_resume.events:

{
  "log_stream_name": "<keep the existing generated value>",
  "file_path": "/var/log/parallelcluster/clustermgtd.events",
  "timestamp_format": "%Y-%m-%dT%H:%M:%S.%f",
  "log_group_name": "<keep the existing generated value>",
  "timezone": "UTC"
}

Then reload the agent the same way ParallelCluster does:

/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
  -a append-config -m ec2 -s \
  -c file:/etc/parallelcluster/amazon-cloudwatch-agent/amazon-cloudwatch-agent.json

Note: this file is overwritten by ParallelCluster on pcluster update-cluster.

I will keep you updated on whether we plan an improvement here

[Kohei-Kuniyoshi-AGC]

Hi @hanwen-cluster

Thank you for your prompt response and for confirming the workaround.

I am glad to hear that adding "timezone": "UTC" is the correct approach and that this issue does not impact job execution or the compute node lifecycle. I will apply this workaround to our environment for now.

I really appreciate your support and would be very grateful if you could keep me updated on any future improvements regarding this behavior.

Thank you again, and I look forward to working with you in the future.