Skip to content

feat(cloudformation): Add DevOps Agent alarm investigations template - #38

Open
LorenzoBoccaccia wants to merge 1 commit into
aws:mainfrom
LorenzoBoccaccia:add-alarm-investigations-cfn
Open

feat(cloudformation): Add DevOps Agent alarm investigations template#38
LorenzoBoccaccia wants to merge 1 commit into
aws:mainfrom
LorenzoBoccaccia:add-alarm-investigations-cfn

Conversation

@LorenzoBoccaccia

@LorenzoBoccaccia LorenzoBoccaccia commented Aug 3, 2026

Copy link
Copy Markdown

Description

Forwards one CloudWatch alarm to a DevOps Agent generic webhook so the alarm opens an
investigation. Also moves the existing skill-policies template into its own directory
and adds a README for each.

No Lambda and no code. An EventBridge rule scoped to a single alarm ARN does the
filtering, its input transformer builds the incident payload, and an API destination
posts it through a connection holding the webhook API key as
Authorization: Bearer <key>. An IAM role grants the rule
events:InvokeApiDestination on that one destination.

incidentId is the EventBridge event id, which is stable across retries, so
redeliveries dedup instead of opening duplicate investigations.

API key rather than HMAC because EventBridge connections support only Basic, API key
and OAuth and cannot sign per request; HMAC would need a signing Lambda. The README
covers the trade-off.

Type of change

  • New skill
  • New custom agent
  • Update to an existing skill or agent
  • Documentation or infrastructure change

Testing

Not a skill, so Agent Skill Eval does not apply.

  • validate-template: 5 parameters, CAPABILITY_IAM.
  • Deployed twice, one stack per alarm. Each creates four resources: connection, API
    destination, rule, IAM role.
  • Forced an alarm to ALARM. Rule metrics in AWS/Events: TriggeredRules 1,
    InvocationAttempts 1, SuccessfulInvocationAttempts 1, RetryInvocationAttempts 0,
    FailedInvocations 0.
  • The event created an INVESTIGATION task in the Agent Space, IN_PROGRESS, carrying
    the alarm ARN. Its reference.referenceId is the EventBridge event id, which is the
    dedup path.
  • EventBridgeRuleTag: deployed with a value, which tags the rule and the role, and
    empty, which falls back to the stack name.
  • Test resources were tagged and deleted afterwards.

One behaviour to know about: an alarm ARN that already appears in an earlier incident
arrives LINKED to it, whether that incident is open or closed. Only a
first-of-its-kind ARN starts its own investigation. That is DevOps Agent correlation
rather than template behaviour, but it means the same alarm will not yield a fresh
investigation on every fire.

Trimmed reply for the review thread:

Renamed to EventBridgeRuleTag and made it optional.

Default: !Ref AWS::StackName is not valid — CloudFormation rejects intrinsics in a
parameter Default ("Every Default member must be a string"). Default is '' and a
condition resolves the stack-name fallback at the two use sites

@LorenzoBoccaccia
LorenzoBoccaccia force-pushed the add-alarm-investigations-cfn branch from d636396 to 6189121 Compare August 18, 2026 09:08
AllowedPattern: '^arn:aws[a-zA-Z-]*:secretsmanager:.+'
ConstraintDescription: Must be a Secrets Manager secret ARN.

AgentName:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: AgentName parameter: rename + make optional

AgentName is required but purely cosmetic — it only sets the DevOpsAgent= tag and, per its own description, "does not affect routing." Two small issues:

Misleading name. "Agent name" reads as if it selects/routes to a specific agent, so a deployer may think it must match their real agent or that a wrong value breaks delivery. Suggest AgentNameTag to signal it's metadata.
Needlessly required. Forcing a value that has no functional effect adds friction.
Proposed:

Rename AgentName → AgentNameTag (param, the four !Ref tag blocks, and the ParameterLabels/ParameterGroups entries).
Make it optional: Default: !Ref AWS::StackName, drop MinLength: 1.
Trim description to: "Value applied as the DevOpsAgent tag on created resources (identification / cost allocation only; does not affect routing)."

@LorenzoBoccaccia
LorenzoBoccaccia force-pushed the add-alarm-investigations-cfn branch from 6189121 to 90b32a0 Compare August 18, 2026 15:03
Forward a single CloudWatch alarm to a DevOps Agent generic webhook so
the alarm opens an investigation. One stack per alarm: an EventBridge
rule scoped to the alarm ARN, an input transformer that builds the
incident payload, an API destination and connection that POST it with
the webhook API key, and a role limited to invoking that destination.
No Lambda function and no code.

incidentId is the EventBridge event id, which is constant across
retries, so redeliveries deduplicate instead of opening a second
investigation. The retry policy caps delivery at 32 attempts over eight
hours and stops retrying events older than that.

Give each template its own directory named after the template file,
holding the template and a README.md, and add a README for the existing
skill-policies template. Update the references to the skill-policies
template path that the move would otherwise break, and correct the
repository name in the two contributor-facing convention docs.
@LorenzoBoccaccia
LorenzoBoccaccia force-pushed the add-alarm-investigations-cfn branch from 90b32a0 to 9239bc0 Compare August 19, 2026 07:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants