diff --git a/mcp/aws-eks-node-diagnostics-mcp/README.md b/mcp/aws-eks-node-diagnostics-mcp/README.md index f9c8a2a..1406515 100644 --- a/mcp/aws-eks-node-diagnostics-mcp/README.md +++ b/mcp/aws-eks-node-diagnostics-mcp/README.md @@ -180,11 +180,15 @@ EKS_NODE_ROLE_ARNS=arn:aws:iam::123456789012:role/eks-node-role \ PRESIGNED_URL_EXPIRATION=120 \ PER_CALLER_RATE_LIMIT_PER_MINUTE=30 \ TOOL_AUTHORIZATION="collect:client-soc;batch_collect:client-emergency" \ +APPROVAL_APPROVER_ARNS=arn:aws:iam::123456789012:role/OnCallOperator \ +APPROVAL_NOTIFICATION_EMAILS=oncall@example.com \ MCP_VPC_ID=vpc-0123456789abcdef0 \ MCP_VPC_SUBNET_IDS=subnet-aaa,subnet-bbb \ ./deploy.sh ``` +> `APPROVAL_APPROVER_ARNS` defaults to the IAM principal running `deploy.sh` when unset. Approvers need `ssm:SendAutomationSignal` (plus Systems Manager console access) to click Approve/Deny. + | Env var | What it restricts | Default | |---------|-------------------|---------| | `ALLOWED_REGIONS` | IAM resource ARNs + Lambda region scanning | Stack region | @@ -194,9 +198,10 @@ MCP_VPC_SUBNET_IDS=subnet-aaa,subnet-bbb \ | `EKS_NODE_ROLE_ARNS` | S3 PutObject + KMS Encrypt principals | Account root | | `PRESIGNED_URL_EXPIRATION` | Log artifact presigned URL lifetime (max 900 s) | 300 s | | `ALLOW_SELF_MANAGED_NODES` | Accept nodes with only the user-settable `kubernetes.io/cluster/*` tag (cross-checked via EKS API) | `false` | -| `REQUIRE_COLLECTION_APPROVAL` | Require human approval before `collect`/`batch_collect` run SSM | `true` | +| `REQUIRE_COLLECTION_APPROVAL` | Require human approval (native SSM `aws:approve`) before `collect`/`batch_collect` run | `true` | +| `APPROVAL_APPROVER_ARNS` | IAM users/roles allowed to approve collections (**required** when approval is on — synth fails without it) | (none — fail-closed) | | `APPROVAL_NOTIFICATION_EMAILS` | Comma-separated emails subscribed to the approval SNS topic | Empty | -| `APPROVAL_TTL_SECONDS` | How long a pending approval stays valid | `900` | +| `APPROVAL_TTL_SECONDS` | How long the `aws:approve` step waits for a decision before timing out | `900` | | `TOOL_AUTHORIZATION` | Per-tool client-id ACL (`tool:client_a,client_b;…`) | Empty (open) | | `PER_CALLER_RATE_LIMIT_PER_MINUTE` | Rate limit per caller (`0` disables) | 60 | | `MCP_VPC_ID` / `MCP_VPC_SUBNET_IDS` | Run Lambda in VPC + create S3/KMS endpoints | None | @@ -210,9 +215,8 @@ MCP_VPC_SUBNET_IDS=subnet-aaa,subnet-bbb \ | Lambda (SSM Automation) | Handles all 19 MCP tool invocations | | Lambda (Unzip) | Auto-extracts uploaded archives | | Lambda (Findings Indexer) | Pre-indexes errors for fast retrieval | -| Lambda (Collection Approval) + Function URL | Human approve/deny endpoint for `collect`/`batch_collect` | -| DynamoDB Table | Stores pending/approved collection requests (TTL-expired) | -| SNS Topic | Notifies approvers with the approve/deny link | +| SSM Documents (approval wrappers) | `aws:approve`-gated wrappers for `collect` (single) and `batch_collect` (fan-out) | +| SNS Topic | Notifies approvers with the SSM console approval link | | SSM Automation Role | Runs log collection on EC2 instances | | Cognito User Pool | OAuth2 authentication for MCP Gateway | | BedrockAgentCore Gateway | MCP protocol endpoint | @@ -231,7 +235,7 @@ All security controls are enforced by default. The construct fails synth unless | **Region restriction** | Stack region only | `ALLOWED_REGIONS` env var | | **Cluster restriction** | **Fail-closed** — must set `ALLOWED_CLUSTER_NAMES` or `ALLOW_ANY_CLUSTER_NAME=true` | `ALLOWED_CLUSTER_NAMES`, `ALLOW_ANY_CLUSTER_NAME` | | **SSM document restriction** | `AWS-RunShellScript` only | `ALLOWED_SSM_DOCUMENTS` env var | -| **Collection approval (human-in-the-loop)** | `collect`/`batch_collect` require out-of-band human approval before SSM runs | `REQUIRE_COLLECTION_APPROVAL` env var | +| **Collection approval (human-in-the-loop)** | `collect`/`batch_collect` pause at a native SSM `aws:approve` step until a designated approver approves in the Systems Manager console | `REQUIRE_COLLECTION_APPROVAL`, `APPROVAL_APPROVER_ARNS` env vars | | **`batch_collect` dry-run** | Defaults to dry-run; real execution needs explicit `dryRun=false` | tool parameter | | **Cluster allowlist (Lambda)** | Enforced when `ALLOWED_CLUSTER_NAMES` is set | `ALLOWED_CLUSTER_NAMES` env var | | **Presigned URL expiry (logs)** | 300 s, max 900 s | `PRESIGNED_URL_EXPIRATION` env var | @@ -281,13 +285,13 @@ Every tool that targets an instance validates that it belongs to an EKS cluster ### Collection Approval (Human-in-the-Loop) -`collect` and `batch_collect` are the only tools that *mutate* — they start SSM Automation (the AWS-managed `AWSSupport-CollectEKSInstanceLogs` document) on nodes. To stop a compromised/poisoned agent from triggering collection on its own, these tools are gated by an out-of-band human approval (on by default; disable with `REQUIRE_COLLECTION_APPROVAL=false`): +`collect` and `batch_collect` are the only tools that *mutate* — they start SSM Automation (the AWS-managed `AWSSupport-CollectEKSInstanceLogs` document) on nodes. To stop a compromised/poisoned agent from triggering collection on its own, these tools use SSM's **native `aws:approve` action** (on by default; disable with `REQUIRE_COLLECTION_APPROVAL=false`): -1. The agent calls `collect` (or `batch_collect` with `dryRun=false`). The Lambda does **not** call SSM. It writes a `PENDING` record to a DynamoDB table, publishes an approve/deny link to an SNS topic, and returns `status: "pending_approval"` with an `approvalId`. -2. A human opens the link (delivered via SNS to the subscribed approvers) and approves or denies. The link is a **capability URL** carrying a one-time, high-entropy secret token; only the SHA-256 of the token is stored server-side, and the token is **never** returned to the agent — so the agent cannot approve its own request. -3. The agent re-calls `collect` with the same `instanceId` plus the `approvalId`. The Lambda verifies the record is `APPROVED`, atomically marks it `CONSUMED` (single-use), and only then starts the SSM Automation. +1. The agent calls `collect` (or `batch_collect` with `dryRun=false`). The Lambda starts a **wrapper Automation document** whose first step is `aws:approve` — the execution immediately pauses inside SSM. The response is `status: "pending_approval"` with an `approvalConsoleUrl` deep link. +2. A designated approver (an IAM principal listed in `APPROVAL_APPROVER_ARNS`) opens the link — the Systems Manager console execution page — reviews the request, and clicks **Approve** or **Deny**. Approvers are also notified via SNS. The decision is IAM-authenticated and CloudTrail-audited; no secret tokens or custom endpoints are involved. +3. On approval, the document proceeds to the collection step **automatically** — the agent never re-calls `collect`; it just polls `status(executionId)`, which reports the approval state (`pending` / `approved` / `denied_or_expired`) and then the collection progress. -The approval endpoint is a separate Lambda (Function URL) with **no** SSM or collection permissions — approving only flips a DynamoDB flag. Requests auto-expire via DynamoDB TTL (`APPROVAL_TTL_SECONDS`, default 15 min). For a batch, one approval authorizes the whole batch; the per-node collections it fans out to are covered by that single approval. +The agent cannot approve its own request: the MCP Lambda has **no** `ssm:SendAutomationSignal` permission, and the approver list is fixed at deploy time (it is not a tool parameter). Pending requests time out after `APPROVAL_TTL_SECONDS` (default 15 min). For a batch, a single approval authorizes the whole batch — the wrapper document's fan-out step then starts one collection per sampled node. Note: because the wrapper documents are regional SSM documents deployed with the stack, approval-gated collection runs in the stack region only. ### Response Redaction @@ -382,12 +386,12 @@ For a detailed walkthrough of the architecture, data flows, tool design, cross-r | 3 — Cluster | `cluster_health`, `compare_nodes`, `batch_collect`†, `batch_status`, `network_diagnostics`, `storage_diagnostics` | Multi-node operations | | 4 — SOPs | `list_sops`, `get_sop` | 41 structured runbooks | -† `collect` and `batch_collect` are **mutating** (they start SSM Automation on nodes). By default they require **human-in-the-loop approval**: the first call returns `status: "pending_approval"` with an `approvalId`, a human approves via the SNS link, and the agent re-calls with the same arguments plus that `approvalId`. See [Security Model](#security-model). +† `collect` and `batch_collect` are **mutating** (they start SSM Automation on nodes). By default they require **human-in-the-loop approval** via SSM's native `aws:approve` action: the call returns `status: "pending_approval"` with an `approvalConsoleUrl`, a designated approver clicks Approve in the Systems Manager console, and collection proceeds automatically — the agent just keeps polling `status`. See [Security Model](#security-model). ### Agent Workflow ``` -collect → (human approves) → collect(approvalId) → status (poll) → validate → errors → search → correlate → read → summarize +collect → (human approves in SSM console) → status (poll) → validate → errors → search → correlate → read → summarize ``` > Set `REQUIRE_COLLECTION_APPROVAL=false` for a fully supervised/test deployment to skip the approval step. @@ -455,6 +459,10 @@ general triage, and follow whichever runbook matches. | Symptom | Cause | Fix | |---------|-------|-----| | `cdk synth` fails with "must set either `allowedClusterNames` …" | Cluster scope wasn't chosen | Set `ALLOWED_CLUSTER_NAMES=…` (preferred) or `ALLOW_ANY_CLUSTER_NAME=true` and re-run `./deploy.sh` | +| `cdk synth` fails with "`approvalApproverArns` is empty" | Approval is on but no approvers were designated | Set `APPROVAL_APPROVER_ARNS=…` (deploy.sh defaults it to the deploying principal) or `REQUIRE_COLLECTION_APPROVAL=false` for test deployments | +| `collect` stuck in `pending_approval` | No approver has acted in the SSM console | Open the `approvalConsoleUrl` from the response as a designated approver and click Approve; the request times out after `APPROVAL_TTL_SECONDS` | +| Approve button fails in the console | The signed-in principal isn't in `APPROVAL_APPROVER_ARNS` or lacks `ssm:SendAutomationSignal` | Sign in as a designated approver, or add the principal and redeploy | +| `status` shows `humanApproval: denied_or_expired` | Approver denied the request, or it timed out | Re-call `collect` to request a fresh approval if still needed | | Tool returns 403 "Caller is not permitted to invoke '…'" | Per-tool ACL doesn't include this client | Add the client to the matching `TOOL_AUTHORIZATION` entry | | Tool returns 429 "Rate limit exceeded" | Caller exceeded `PER_CALLER_RATE_LIMIT_PER_MINUTE` | Wait the `retryAfterSeconds` in the response, or raise the limit | | `collect` returns "document not found" | SSM document not in target region | Use a supported region or pass `region` explicitly | diff --git a/mcp/aws-eks-node-diagnostics-mcp/bin/app.ts b/mcp/aws-eks-node-diagnostics-mcp/bin/app.ts index 50c5214..8f00627 100644 --- a/mcp/aws-eks-node-diagnostics-mcp/bin/app.ts +++ b/mcp/aws-eks-node-diagnostics-mcp/bin/app.ts @@ -79,27 +79,51 @@ new EksNodeLogMcpStack(app, 'EksNodeLogMcpStack', { : undefined, // Human-in-the-loop approval for the mutating collection tools (collect, - // batch_collect). On by default (security review M1/M2): the agent's call - // creates a pending approval and notifies approvers via SNS; the SSM run only - // happens after a human approves via the approval link. Set - // REQUIRE_COLLECTION_APPROVAL=false only for a fully supervised/test deployment. + // batch_collect). On by default (security review M1/M2): collection runs via + // a wrapper SSM Automation document whose first step is the native + // aws:approve action — the execution pauses in SSM until a designated human + // approves it in the Systems Manager console, then collection proceeds + // automatically. Set REQUIRE_COLLECTION_APPROVAL=false only for a fully + // supervised/test deployment. requireCollectionApproval: process.env.REQUIRE_COLLECTION_APPROVAL ? !['0', 'false', 'no'].includes(process.env.REQUIRE_COLLECTION_APPROVAL.toLowerCase()) : undefined, - // Opt-in public Function URL for one-click approve/deny links. Default off: - // account guardrails (e.g. mitigation services that strip public Lambda - // policies) silently break public URLs. When off, the approval email - // contains an IAM-authenticated `aws lambda invoke` command instead. - approvalViaPublicUrl: process.env.APPROVAL_VIA_PUBLIC_URL === 'true', + // IAM principals allowed to approve collections (user/role ARNs, comma + // separated). REQUIRED when approval is enabled — synth fails without it. + // Approvers also need ssm:SendAutomationSignal to click Approve/Deny. + approvalApproverArns: process.env.APPROVAL_APPROVER_ARNS + ? process.env.APPROVAL_APPROVER_ARNS.split(',').filter(Boolean) + : undefined, - // Emails to subscribe to the approval SNS topic (each gets the approve/deny action). + // Emails to subscribe to the approval SNS topic (each gets the SSM console + // approval link when a collection is requested). approvalNotificationEmails: process.env.APPROVAL_NOTIFICATION_EMAILS ? process.env.APPROVAL_NOTIFICATION_EMAILS.split(',').filter(Boolean) : undefined, - // How long a pending approval stays valid (seconds). + // How long the aws:approve step waits for a human decision (seconds). approvalTtlSeconds: process.env.APPROVAL_TTL_SECONDS ? parseInt(process.env.APPROVAL_TTL_SECONDS, 10) : undefined, + + // Restricted tools opt-in (network packet capture). Absent from the MCP tool + // surface unless listed. tcpdump_capture is ADDITIONALLY approval-gated: every + // capture pauses at a native SSM aws:approve step until a designated human + // approves it in the Systems Manager console (M3). + // Format: ENABLED_RESTRICTED_TOOLS="tcpdump_capture,tcpdump_analyze" + enableRestrictedTools: process.env.ENABLED_RESTRICTED_TOOLS + ? process.env.ENABLED_RESTRICTED_TOOLS.split(',').map(s => s.trim()).filter(Boolean) + : undefined, + + // Presigned URL expiry for pcap downloads (tighter than log artifacts — + // captures may contain credentials in transit). Max 300s. + pcapPresignedUrlExpirationSeconds: process.env.PCAP_PRESIGNED_URL_EXPIRATION + ? parseInt(process.env.PCAP_PRESIGNED_URL_EXPIRATION, 10) + : undefined, + + // Size above which a completed pcap is flagged as oversized (advisory). + maxPcapBytes: process.env.MAX_PCAP_BYTES + ? parseInt(process.env.MAX_PCAP_BYTES, 10) + : undefined, }); diff --git a/mcp/aws-eks-node-diagnostics-mcp/deploy.sh b/mcp/aws-eks-node-diagnostics-mcp/deploy.sh index 35b660e..9d821da 100755 --- a/mcp/aws-eks-node-diagnostics-mcp/deploy.sh +++ b/mcp/aws-eks-node-diagnostics-mcp/deploy.sh @@ -378,6 +378,46 @@ if [ -z "$ALLOWED_CLUSTER_NAMES" ]; then export ALLOWED_CLUSTER_NAMES fi +# --- APPROVAL_APPROVER_ARNS: required when collection approval is on (default) --- +# collect/batch_collect pause at a native SSM aws:approve step until one of these +# IAM principals approves in the Systems Manager console. Default: the IAM +# principal running this deploy (assumed-role sessions map to the role ARN). +if [ "${REQUIRE_COLLECTION_APPROVAL:-true}" != "false" ] && [ -z "$APPROVAL_APPROVER_ARNS" ]; then + CALLER_ARN=$(aws sts get-caller-identity --query Arn --output text 2>/dev/null || echo "") + case "$CALLER_ARN" in + arn:*:sts::*:assumed-role/*) + CALLER_ACCOUNT="${CALLER_ARN#arn:*:sts::}"; CALLER_ACCOUNT="${CALLER_ACCOUNT%%:*}" + ROLE_NAME="${CALLER_ARN#*:assumed-role/}"; ROLE_NAME="${ROLE_NAME%%/*}" + APPROVAL_APPROVER_ARNS="arn:aws:iam::${CALLER_ACCOUNT}:role/${ROLE_NAME}" + ;; + arn:*) + APPROVAL_APPROVER_ARNS="$CALLER_ARN" + ;; + esac + if [ -z "$APPROVAL_APPROVER_ARNS" ]; then + echo "ERROR: Collection approval is enabled but APPROVAL_APPROVER_ARNS is not set" + echo "and the caller identity could not be detected. Set APPROVAL_APPROVER_ARNS" + echo "to the IAM user/role ARN(s) allowed to approve collections, or set" + echo "REQUIRE_COLLECTION_APPROVAL=false for a supervised/test deployment." + exit 1 + fi + echo "Collection approvers (defaulted to deploying principal): $APPROVAL_APPROVER_ARNS" + export APPROVAL_APPROVER_ARNS +elif [ -n "$APPROVAL_APPROVER_ARNS" ]; then + echo "Collection approvers: $APPROVAL_APPROVER_ARNS" + export APPROVAL_APPROVER_ARNS +fi + +# --- ENABLED_RESTRICTED_TOOLS: opt-in network packet capture tools --- +# tcpdump_capture/tcpdump_analyze are absent from the MCP tool surface unless +# listed here. tcpdump_capture is ADDITIONALLY approval-gated: every capture +# pauses at a native SSM aws:approve step until an approver approves it in the +# Systems Manager console (same approvers as collection). +if [ -n "${ENABLED_RESTRICTED_TOOLS:-}" ]; then + echo "Restricted tools enabled: $ENABLED_RESTRICTED_TOOLS (tcpdump_capture requires human approval per capture)" + export ENABLED_RESTRICTED_TOOLS +fi + echo "" echo "Deploying CDK stack..." npx cdk deploy "$STACK_NAME" --require-approval never --outputs-file cdk-outputs.json diff --git a/mcp/aws-eks-node-diagnostics-mcp/sops/runbooks/D4-mtu-fragmentation.md b/mcp/aws-eks-node-diagnostics-mcp/sops/runbooks/D4-mtu-fragmentation.md index 0928e31..4ce6957 100644 --- a/mcp/aws-eks-node-diagnostics-mcp/sops/runbooks/D4-mtu-fragmentation.md +++ b/mcp/aws-eks-node-diagnostics-mcp/sops/runbooks/D4-mtu-fragmentation.md @@ -27,8 +27,8 @@ SHOULD: - Use `search` tool with query=`AWS_VPC_MTU_OVERRIDE|MTU` to check CNI MTU configuration MAY: -- Manually capture fragmentation events on the node (via SSM Session Manager) if deeper analysis is needed, e.g. `sudo tcpdump -i any -nn 'ip[6:2] & 0x3fff != 0' -w /tmp/frag.pcap` -- Review the capture manually with `sudo tcpdump -nn -r /tmp/frag.pcap` to inspect MTU/fragmentation behavior +- Use `tcpdump_capture` tool with instanceId to capture fragmentation events (if needed for deeper analysis) +- Use `tcpdump_analyze` tool to analyze captured packets for MTU issues ## Phase 2 — Enrich @@ -66,6 +66,7 @@ escalation_conditions: safety_ratings: - "Log collection (collect), search, errors, network_diagnostics: GREEN (read-only)" + - "tcpdump_capture: YELLOW — human-approved packet capture (pauses at a native SSM aws:approve step until a designated approver approves in the Systems Manager console)" - "Modify CNI MTU config: YELLOW — operator action, not available via MCP tools" ## Common Issues diff --git a/mcp/aws-eks-node-diagnostics-mcp/sops/runbooks/D7-network-performance-degradation.md b/mcp/aws-eks-node-diagnostics-mcp/sops/runbooks/D7-network-performance-degradation.md index 823f9aa..3e865e8 100644 --- a/mcp/aws-eks-node-diagnostics-mcp/sops/runbooks/D7-network-performance-degradation.md +++ b/mcp/aws-eks-node-diagnostics-mcp/sops/runbooks/D7-network-performance-degradation.md @@ -69,10 +69,10 @@ MUST: - Use `correlate` tool with instanceId and pivotEvent set to the most prominent error pattern (e.g., `retransmit` or `rx_errors`) to build a timeline SHOULD: -- Manually capture live traffic on the node (via SSM Session Manager) if the issue is intermittent and log evidence is insufficient - - For latency: capture on the affected pod interface or eth0, e.g. `sudo tcpdump -i eth0 -nn -w /tmp/cap.pcap` - - For packet loss: capture with a filter matching the affected traffic flow, e.g. `sudo tcpdump -i eth0 -nn 'tcp' -w /tmp/cap.pcap` - - Review the capture manually with `sudo tcpdump -nn -r /tmp/cap.pcap` for retransmissions, resets, and latency patterns +- Use `tcpdump_capture` tool with instanceId to capture live traffic if the issue is intermittent and log evidence is insufficient + - For latency: capture on the affected pod interface or eth0 + - For packet loss: capture with a filter matching the affected traffic flow +- Use `tcpdump_analyze` tool to analyze the capture for retransmissions, resets, and latency patterns - Use `search` tool with query=`nf_conntrack_count|nf_conntrack_max` to rule out conntrack pressure (even if not full, high utilization can cause slowness) MAY: @@ -113,7 +113,8 @@ escalation_conditions: safety_ratings: - "Log collection (collect), search, errors, network_diagnostics, correlate: GREEN (read-only)" - - "Manual tcpdump on the node (via SSM Session Manager): YELLOW — operator action, not available via MCP tools" + - "tcpdump_capture: YELLOW — human-approved packet capture (pauses at a native SSM aws:approve step until a designated approver approves in the Systems Manager console)" + - "tcpdump_analyze: GREEN (read-only analysis of completed captures)" - "Modify TCP sysctl parameters: YELLOW — operator action, not available via MCP tools" - "Modify security groups / NACLs: YELLOW — operator action, not available via MCP tools" - "Replace instance (hardware errors): RED — operator action, requires approval" @@ -125,7 +126,7 @@ safety_ratings: resolution: "Operator action: update ENA driver to latest. If errors persist, replace instance (possible hardware issue)." - symptoms: "search returns high TCPRetransSegs or TCPTimeouts from /proc/net/snmp" - diagnosis: "TCP retransmissions indicate packet loss in the network path. Manually run tcpdump on the node to identify where loss occurs." + diagnosis: "TCP retransmissions indicate packet loss in the network path. Use tcpdump_capture to identify where loss occurs." resolution: "If loss is on-node: check iptables DROP rules via network_diagnostics. If loss is off-node: escalate as VPC/upstream issue." - symptoms: "network_diagnostics iptables section shows DROP rules on FORWARD chain" @@ -144,7 +145,7 @@ safety_ratings: diagnosis: "Routing issue — traffic to certain pod CIDRs has no valid next hop." resolution: "Operator action: check VPC route tables and CNI routing. May need to restart aws-node DaemonSet." -- symptoms: "a manual tcpdump capture shows retransmissions only for traffic leaving the VPC (cross-AZ or internet)" +- symptoms: "tcpdump_analyze shows retransmissions only for traffic leaving the VPC (cross-AZ or internet)" diagnosis: "Loss in the upstream path, not on the node. Node-level fixes will not help." resolution: "Escalate: check VPC peering, TGW, NAT gateway, or internet gateway health." @@ -178,9 +179,9 @@ search(instanceId="i-0abc123def456", query="DROP.*INPUT|DROP.*FORWARD|REJECT") # Step 7: Check IRQ distribution search(instanceId="i-0abc123def456", query="softirq.*NET_RX|ksoftirqd|irqbalance") -# Step 8: Capture traffic manually on the node if still inconclusive (via SSM Session Manager) -# sudo tcpdump -i eth0 -nn -c 5000 -w /tmp/cap.pcap -# sudo tcpdump -nn -r /tmp/cap.pcap # review retransmissions / resets +# Step 8: Capture traffic if still inconclusive +tcpdump_capture(instanceId="i-0abc123def456", interface="eth0", duration=30, filter="tcp") +tcpdump_analyze(instanceId="i-0abc123def456", commandId="") # Step 9: Correlate timeline correlate(instanceId="i-0abc123def456", pivotEvent="retransmit", timeWindow=300) @@ -199,8 +200,8 @@ evidence: content: "" - type: search content: "" - - type: manual_tcpdump - content: "" + - type: tcpdump_analyze + content: "" - type: correlate content: "" severity: HIGH diff --git a/mcp/aws-eks-node-diagnostics-mcp/sops/runbooks/D9-pod-to-pod-connectivity.md b/mcp/aws-eks-node-diagnostics-mcp/sops/runbooks/D9-pod-to-pod-connectivity.md index c7cc411..10baeaa 100644 --- a/mcp/aws-eks-node-diagnostics-mcp/sops/runbooks/D9-pod-to-pod-connectivity.md +++ b/mcp/aws-eks-node-diagnostics-mcp/sops/runbooks/D9-pod-to-pod-connectivity.md @@ -23,8 +23,8 @@ context: > routing table. Traffic between pods on different nodes goes through the node's eth0, VPC routing, and the destination node's ENI. Failures can occur at any layer: veth misconfiguration, iptables/eBPF NetworkPolicy enforcement dropping traffic, missing routes for pod CIDRs, CNI plugin bugs, security group rules blocking - inter-node traffic, or NACL restrictions. This SOP uses a manual `tcpdump` capture (run on the node via SSM - Session Manager) on both the pod's veth interface and the node's eth0 to pinpoint exactly where packets are lost. Cross-references D1 (IP allocation), D3 + inter-node traffic, or NACL restrictions. This SOP uses tcpdump_capture on both the pod's veth interface + and the node's eth0 to pinpoint exactly where packets are lost. Cross-references D1 (IP allocation), D3 (conntrack), D5 (DNS), D7 (general network perf), D8 (kube-proxy/service connectivity). --- @@ -157,23 +157,23 @@ SHOULD: ### 2F — Packet Capture (tcpdump) -MUST (if the issue is not identified from log analysis above). Run these manually on the node via SSM Session Manager — packet capture is not available as an MCP tool: -- On the SOURCE node, capture traffic on the pod's veth interface: - - `sudo tcpdump -i -nn 'host ' -w /tmp/src-veth.pcap` - - Duration: ~30 seconds while reproducing the connectivity failure +MUST (if the issue is not identified from log analysis above): +- Use `tcpdump_capture` tool on the SOURCE node to capture traffic on the pod's veth interface: + - Filter for traffic to/from the destination pod IP + - Duration: 30 seconds while reproducing the connectivity failure - This shows if packets LEAVE the source pod -- On the SOURCE node, capture on eth0: - - `sudo tcpdump -i eth0 -nn 'host ' -w /tmp/src-eth0.pcap` +- Use `tcpdump_capture` tool on the SOURCE node to capture on eth0: + - Same filter for destination pod IP - This shows if packets reach the node's outbound interface (cross-node) or are dropped before -- Review both captures with `sudo tcpdump -nn -r `: +- Use `tcpdump_analyze` tool to analyze both captures: - Packets on veth but NOT on eth0: dropped by iptables/eBPF/routing on the source node - Packets on eth0 of source but not arriving at destination: dropped in VPC (SG, NACL, routing) - Packets arriving at destination eth0 but not on destination veth: dropped on destination node SHOULD: -- If cross-node: also capture on the DESTINATION node's eth0 and the destination pod's veth +- If cross-node: also use `tcpdump_capture` on the DESTINATION node's eth0 and the destination pod's veth - This gives the full 4-point trace: src-veth → src-eth0 → dst-eth0 → dst-veth -- Review the captures for: +- Use `tcpdump_analyze` to check for: - TCP RST (connection refused — something is actively rejecting) - TCP SYN with no SYN-ACK (packets silently dropped) - ICMP unreachable messages (routing or firewall rejection) @@ -231,7 +231,8 @@ escalation_conditions: safety_ratings: - "Log collection (collect), search, errors, network_diagnostics, correlate, compare_nodes: GREEN (read-only)" - - "Manual tcpdump on the node (via SSM Session Manager): YELLOW — operator action, not available via MCP tools" + - "tcpdump_capture: YELLOW — human-approved packet capture (pauses at a native SSM aws:approve step until a designated approver approves in the Systems Manager console)" + - "tcpdump_analyze: GREEN (read-only analysis of completed captures)" - "Modify NetworkPolicy: YELLOW — operator action, not available via MCP tools" - "Modify iptables FORWARD policy: YELLOW — operator action, affects all pod traffic on node" - "Modify security groups: YELLOW — operator action, affects network access" @@ -332,17 +333,20 @@ search(instanceId="i-0abc123def456", query="blackhole|no route|missing.*route|po # Step 8: Check security groups (cross-node) search(instanceId="i-0abc123def456", query="security group|sg-|SecurityGroupIds") -# Step 9-11: Capture packets MANUALLY on the node(s) via SSM Session Manager -# (packet capture is not an MCP tool). While reproducing the issue: -# # source pod veth: -# sudo tcpdump -i -nn 'host ' -w /tmp/src-veth.pcap -# # source node eth0: -# sudo tcpdump -i eth0 -nn 'host ' -w /tmp/src-eth0.pcap -# # if cross-node, on the DESTINATION node (i-0dest789ghi012): -# sudo tcpdump -i eth0 -nn 'host ' -w /tmp/dst-eth0.pcap -# sudo tcpdump -i -nn 'host ' -w /tmp/dst-veth.pcap -# # review each with: sudo tcpdump -nn -r -collect(instanceId="i-0dest789ghi012") # collect logs from the destination node too +# Step 9: Capture on source pod veth (while reproducing the issue) +tcpdump_capture(instanceId="i-0abc123def456", interface="", duration=30, filter="host ") +tcpdump_analyze(instanceId="i-0abc123def456", commandId="") + +# Step 10: Capture on source node eth0 +tcpdump_capture(instanceId="i-0abc123def456", interface="eth0", duration=30, filter="host ") +tcpdump_analyze(instanceId="i-0abc123def456", commandId="") + +# Step 11: If cross-node — capture on DESTINATION node eth0 and veth +collect(instanceId="i-0dest789ghi012") +tcpdump_capture(instanceId="i-0dest789ghi012", interface="eth0", duration=30, filter="host ") +tcpdump_analyze(instanceId="i-0dest789ghi012", commandId="") +tcpdump_capture(instanceId="i-0dest789ghi012", interface="", duration=30, filter="host ") +tcpdump_analyze(instanceId="i-0dest789ghi012", commandId="") # Step 12: Correlate timeline correlate(instanceId="i-0abc123def456", pivotEvent="connection refused|timed out|DENY", timeWindow=300) @@ -367,8 +371,8 @@ evidence: content: "" - type: search content: "" - - type: manual_tcpdump - content: "" + - type: tcpdump_analyze + content: "" - type: correlate content: "" severity: HIGH diff --git a/mcp/aws-eks-node-diagnostics-mcp/sops/runbooks/Z1-general-troubleshooting.md b/mcp/aws-eks-node-diagnostics-mcp/sops/runbooks/Z1-general-troubleshooting.md index f3dc0c8..3a03ed8 100644 --- a/mcp/aws-eks-node-diagnostics-mcp/sops/runbooks/Z1-general-troubleshooting.md +++ b/mcp/aws-eks-node-diagnostics-mcp/sops/runbooks/Z1-general-troubleshooting.md @@ -89,7 +89,7 @@ SHOULD: MAY: - Use `compare_nodes` tool with instanceIds of affected + healthy node to find what differs -- Manually run `tcpdump` on the node (via SSM Session Manager) if networking is suspected but network_diagnostics is inconclusive, e.g. `sudo tcpdump -i any -nn -c 200` +- Use `tcpdump_capture` tool if networking is suspected but network_diagnostics is inconclusive - Use EKS MCP `get_cloudwatch_logs` with clusterName, resource_type="cluster", log_type="control-plane", filter_pattern="error" to check kube-audit logs for recent API errors, denied requests, or failed mutations that may correlate with the issue - Use EKS MCP `get_cloudwatch_logs` with clusterName, resource_type="cluster", log_type="control-plane", filter_pattern="Forbidden" to check for RBAC denials in the audit log that may indicate permission issues diff --git a/mcp/aws-eks-node-diagnostics-mcp/src/lambda/ssm-automation-enhanced.py b/mcp/aws-eks-node-diagnostics-mcp/src/lambda/ssm-automation-enhanced.py index 3d9e919..9cb13b7 100644 --- a/mcp/aws-eks-node-diagnostics-mcp/src/lambda/ssm-automation-enhanced.py +++ b/mcp/aws-eks-node-diagnostics-mcp/src/lambda/ssm-automation-enhanced.py @@ -19,8 +19,6 @@ import re import hashlib import time -import uuid -import secrets import signal import threading from contextlib import contextmanager @@ -45,7 +43,6 @@ ec2_client = boto3.client('ec2') cloudwatch_client = boto3.client('cloudwatch') sns_client = boto3.client('sns') -dynamodb_client = boto3.client('dynamodb') # Regional client cache to avoid re-creating clients per invocation _regional_clients: Dict[str, Dict[str, Any]] = {} @@ -59,20 +56,29 @@ # ── Human-in-the-loop approval for mutating collection tools (M1/M2) ── # collect/batch_collect start SSM Automation on nodes. Rather than let an # autonomous (potentially poisoned) agent trigger that directly, the Lambda -# creates a pending approval, notifies humans via SNS, and only runs the SSM -# call after a human approves out-of-band. The approval is NOT a tool parameter -# the agent can set — approval requires a secret token delivered only to humans. -APPROVAL_TABLE_NAME = os.environ.get('APPROVAL_TABLE_NAME', '') +# starts a wrapper SSM Automation document whose FIRST step is the native +# `aws:approve` action. The execution pauses there until a designated human +# approves it in the AWS Systems Manager console (or via +# `ssm:SendAutomationSignal`) — only then does the document proceed to run the +# actual log collection. Approval is NOT a tool parameter the agent can set: +# the Lambda has no ssm:SendAutomationSignal permission, and the approvers are +# fixed IAM principals baked in at deploy time. APPROVAL_TOPIC_ARN = os.environ.get('APPROVAL_TOPIC_ARN', '') -APPROVAL_BASE_URL = os.environ.get('APPROVAL_BASE_URL', '') # approve Function URL (opt-in; empty in CLI mode) -APPROVAL_FUNCTION_NAME = os.environ.get('APPROVAL_FUNCTION_NAME', '') # approval handler for direct IAM-auth invoke +COLLECT_APPROVAL_DOCUMENT = os.environ.get('COLLECT_APPROVAL_DOCUMENT', '') +BATCH_APPROVAL_DOCUMENT = os.environ.get('BATCH_APPROVAL_DOCUMENT', '') +TCPDUMP_APPROVAL_DOCUMENT = os.environ.get('TCPDUMP_APPROVAL_DOCUMENT', '') +APPROVAL_APPROVERS = [ + a.strip() for a in os.environ.get('APPROVAL_APPROVERS', '').split(',') if a.strip() +] +# True only when email subscriptions were created at deploy time — used to +# phrase the pending-approval message honestly (an SNS publish to a topic with +# zero subscribers "succeeds" but nobody is notified). +APPROVAL_EMAILS_CONFIGURED = os.environ.get( + 'APPROVAL_EMAILS_CONFIGURED', '' +).strip().lower() in ('1', 'true', 'yes') REQUIRE_COLLECTION_APPROVAL = os.environ.get( 'REQUIRE_COLLECTION_APPROVAL', 'true' ).strip().lower() in ('1', 'true', 'yes') -try: - APPROVAL_TTL_SECONDS = int(os.environ.get('APPROVAL_TTL_SECONDS', '900')) -except (ValueError, TypeError): - APPROVAL_TTL_SECONDS = 900 def emit_metric(metric_name: str, value: float = 1.0, unit: str = 'Count', @@ -208,6 +214,40 @@ def _parse_presigned_url_expiration() -> int: PRESIGNED_URL_EXPIRATION = _parse_presigned_url_expiration() +def _parse_pcap_presigned_url_expiration() -> int: + """ + Parse PCAP_PRESIGNED_URL_EXPIRATION_SECONDS env var. Network captures may + contain credentials in transit and other sensitive payloads — they get a + much shorter window than ordinary log artifacts. Default 60s, max 300s. + """ + raw = os.environ.get('PCAP_PRESIGNED_URL_EXPIRATION_SECONDS', '') + try: + val = int(raw) + if val > 0: + return min(val, 300) + except (ValueError, TypeError): + pass + return 60 + + +PCAP_PRESIGNED_URL_EXPIRATION = _parse_pcap_presigned_url_expiration() + + +def _parse_max_pcap_bytes() -> int: + """Cap at which a pcap upload is flagged as oversized.""" + raw = os.environ.get('MAX_PCAP_BYTES', '') + try: + val = int(raw) + if val > 0: + return val + except (ValueError, TypeError): + pass + return 200 * 1024 * 1024 # 200 MiB + + +MAX_PCAP_BYTES = _parse_max_pcap_bytes() + + # ============================================================================= # ALLOWED REGIONS — configurable via env var (T9, T11 mitigation) # ============================================================================= @@ -247,13 +287,20 @@ def resolve_and_validate_region(arguments: Dict, instance_id: str = None) -> tup # EKS clusters this deployment is permitted to act on. Populated from the # ALLOWED_CLUSTER_NAMES env var (the CDK also enforces it at the IAM layer). -# When empty, no cluster-name allowlist is enforced at the Lambda layer — -# region and EKS-tag validation still apply. ALLOWED_CLUSTER_NAMES = set( c.strip() for c in os.environ.get('ALLOWED_CLUSTER_NAMES', '').split(',') if c.strip() ) +# Fail-closed companion to ALLOWED_CLUSTER_NAMES (E2): an empty allowlist only +# permits all clusters when the operator explicitly acknowledged the broader +# scope at deploy time (the CDK `allowAnyClusterName: true` flag). Without the +# acknowledgment, an empty allowlist rejects every cluster instead of allowing +# every cluster. +ALLOW_ANY_CLUSTER_NAME = os.environ.get( + 'ALLOW_ANY_CLUSTER_NAME', '' +).strip().lower() in ('1', 'true', 'yes') + # Whether to accept nodes that carry ONLY the user-settable # kubernetes.io/cluster/* tag (self-managed node groups). The EKS-managed # eks:cluster-name tag cannot be set through standard EC2 tag APIs, so it is @@ -267,12 +314,13 @@ def resolve_and_validate_region(arguments: Dict, instance_id: str = None) -> tup def cluster_name_allowed(cluster_name: Optional[str]) -> bool: """ - True if the cluster is permitted by the Lambda-level allowlist. When the - allowlist is empty, all clusters are permitted (region + tag validation - still apply). + True if the cluster is permitted by the Lambda-level allowlist (E2). + Fail-closed: an empty allowlist permits clusters only when the operator + explicitly acknowledged any-cluster scope at deploy time + (ALLOW_ANY_CLUSTER_NAME=true). Region + tag validation always still apply. """ if not ALLOWED_CLUSTER_NAMES: - return True + return ALLOW_ANY_CLUSTER_NAME return bool(cluster_name) and cluster_name in ALLOWED_CLUSTER_NAMES @@ -282,10 +330,14 @@ def validate_cluster_name(cluster_name: str) -> Optional[Dict]: (E2 mitigation). Returns None if allowed, or an error_response dict. """ if not cluster_name_allowed(cluster_name): + allowed = ', '.join(sorted(ALLOWED_CLUSTER_NAMES)) or ( + 'none — deployment has no cluster allowlist and any-cluster scope ' + 'was not acknowledged (ALLOW_ANY_CLUSTER_NAME)' + ) return error_response( 403, f"Cluster '{cluster_name}' is not permitted by this deployment. " - f"Allowed clusters: {', '.join(sorted(ALLOWED_CLUSTER_NAMES))}" + f"Allowed clusters: {allowed}" ) return None @@ -400,10 +452,13 @@ def _handler(signum, frame): # ============================================================================= # Tools that require explicit opt-in via the ENABLED_RESTRICTED_TOOLS env var. -# No restricted tools are currently defined — the invasive tcpdump capture/analyze -# tools were removed entirely. This set is retained so the authorization gate -# below (and any future restricted tool) keeps working without further changes. -RESTRICTED_TOOLS: set = set() +# These tools perform invasive operations (network captures, namespace entry) +# and are completely removed from the routing table by default. They do not +# appear in available_tools and cannot be invoked unless enabled. +RESTRICTED_TOOLS = { + 'tcpdump_capture', + 'tcpdump_analyze', +} # Parse enabled restricted tools from env var ENABLED_RESTRICTED_TOOLS = set( @@ -411,9 +466,12 @@ def _handler(signum, frame): if t.strip() ) -# NOTE: collect/batch_collect (mutating tools) are gated at runtime by the -# human-in-the-loop approval workflow (see enforce_collection_approval), not by -# hiding them from the tool surface. +# NOTE: mutating tools are gated at runtime by the human-in-the-loop approval +# workflow (a native SSM aws:approve step), not only by hiding them from the +# tool surface: collect/batch_collect are always visible but approval-gated; +# tcpdump_capture is BOTH opt-in (ENABLED_RESTRICTED_TOOLS) and approval-gated +# — every capture pauses at aws:approve until a designated human approves it +# in the Systems Manager console. def _parse_tool_authorization() -> Dict[str, set]: @@ -534,7 +592,7 @@ def validate_tool_authorization(tool_name: str, caller: Optional[Dict] = None) - (a) Restricted-tool opt-in (ENABLED_RESTRICTED_TOOLS). (b) Per-tool ACL keyed on Cognito client_id (TOOL_AUTHORIZATION). Mutating tools (collect/batch_collect) are gated separately at runtime by the - human approval workflow (enforce_collection_approval), not here. + human approval workflow (a native SSM aws:approve step), not here. Returns None if authorized, or an error_response dict if denied. """ if tool_name in RESTRICTED_TOOLS and tool_name not in ENABLED_RESTRICTED_TOOLS: @@ -561,6 +619,123 @@ def validate_tool_authorization(tool_name: str, caller: Optional[Dict] = None) - return None +# ============================================================================= +# BPF FILTER VALIDATION — allowlist-based (T1 mitigation) +# ============================================================================= + +# Allowlist of safe BPF filter tokens. This is intentionally restrictive. +# BPF filters are a mini-language; we only allow known-safe primitives. +_BPF_ALLOWED_KEYWORDS = frozenset({ + # Protocols + 'tcp', 'udp', 'icmp', 'arp', 'ip', 'ip6', 'ether', 'vlan', 'stp', + # Directions + 'src', 'dst', + # Qualifiers + 'host', 'net', 'port', 'portrange', 'proto', + # Logical operators + 'and', 'or', 'not', + # TCP flags (used in bracket expressions) + 'tcp-syn', 'tcp-ack', 'tcp-fin', 'tcp-rst', 'tcp-push', 'tcp-urg', + # Misc + 'greater', 'less', 'len', +}) + +# Pattern for valid BPF tokens: keywords, IPs, CIDRs, numbers, and bracket expressions +_BPF_TOKEN_PATTERN = re.compile( + r'^(' + r'\d{1,5}' # port numbers + r'|\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}(/\d{1,2})?' # IPv4 addresses and CIDRs + r'|[0-9a-f:]+(/\d{1,3})?' # IPv6 addresses and CIDRs + r'|\d+-\d+' # port ranges (e.g., 80-443) + r')$', + re.IGNORECASE +) + +# Bracket expressions like tcp[tcpflags], tcp[13], udp[0:2] +_BPF_BRACKET_PATTERN = re.compile( + r'^(tcp|udp|icmp|ip|ip6|ether)\[' + r'[a-z0-9:]+\]' + r'(\s*[&|!=<>]+\s*' + r'(\(?(tcp-syn|tcp-ack|tcp-fin|tcp-rst|tcp-push|tcp-urg|0x[0-9a-f]+|\d+)\)?)' + r')?$', + re.IGNORECASE +) + + +def validate_bpf_filter(bpf_filter: str) -> Optional[str]: + """ + Validate a BPF filter expression using allowlist-based validation. + + Returns None if valid, or an error message string if invalid. + + Security: This replaces the previous denylist approach which missed + backticks, newlines, and other injection vectors. The allowlist approach + only permits known-safe BPF primitives. + """ + if not bpf_filter: + return None + + # Hard reject: any control characters, backticks, or shell metacharacters + # This catches \n, \r, \t, backticks, $, etc. + if re.search(r'[\x00-\x1f\x7f`$\\;{}<>!~^]', bpf_filter): + return 'BPF filter contains forbidden characters (control chars, backticks, shell metacharacters)' + + # Reject excessively long filters + if len(bpf_filter) > 256: + return 'BPF filter too long (max 256 characters)' + + # Reject parentheses used for subshells — BPF uses them for grouping but + # we handle them carefully + # Allow balanced parentheses only + depth = 0 + for ch in bpf_filter: + if ch == '(': + depth += 1 + elif ch == ')': + depth -= 1 + if depth < 0: + return 'BPF filter has unbalanced parentheses' + if depth != 0: + return 'BPF filter has unbalanced parentheses' + + # Strip parentheses for token validation (BPF uses them for grouping) + stripped = bpf_filter.replace('(', ' ').replace(')', ' ') + + # Tokenize and validate each token + tokens = stripped.split() + if not tokens: + return 'BPF filter is empty after parsing' + + for token in tokens: + token_lower = token.lower().strip() + if not token_lower: + continue + + # Check against known keywords + if token_lower in _BPF_ALLOWED_KEYWORDS: + continue + + # Check against token pattern (IPs, ports, numbers) + if _BPF_TOKEN_PATTERN.match(token_lower): + continue + + # Check bracket expressions (e.g., tcp[tcpflags]) + if _BPF_BRACKET_PATTERN.match(token_lower): + continue + + # Comparison operators + if token_lower in ('!=', '==', '>=', '<=', '>', '<', '=', '&'): + continue + + # Hex values (used in flag comparisons) + if re.match(r'^0x[0-9a-f]+$', token_lower): + continue + + return f"BPF filter contains disallowed token: '{token}'. Only standard BPF primitives are permitted." + + return None + + # ============================================================================= # EKS INSTANCE VALIDATION — verify target is an EKS node (T4, T13 mitigation) # ============================================================================= @@ -1914,9 +2089,15 @@ def find_execution_by_idempotency_token(instance_id: str, token: str) -> Optiona response = regional_ssm.get_automation_execution( AutomationExecutionId=execution_id ) + execution = response['AutomationExecution'] return { 'executionId': execution_id, - 'status': response['AutomationExecution']['AutomationExecutionStatus'] + 'status': execution['AutomationExecutionStatus'], + 'documentName': execution.get('DocumentName', ''), + 'region': exec_region, + # Full execution snapshot so the idempotent-replay path can + # run the same wrapper-status augmentation status() uses. + '_execution': execution, } except Exception: return None @@ -3452,7 +3633,7 @@ def lambda_handler(event: Dict, context: Any) -> Dict: # Tool routing. collect/batch_collect are mutating (they start SSM Automation # on nodes); rather than hide them, they are gated at runtime by a human - # approval (M1/M2) — see enforce_collection_approval. Read/analysis tools run + # approval (M1/M2) — a native SSM aws:approve step. Read/analysis tools run # directly. tools = { # Core Operations (Tier 1) @@ -3481,19 +3662,23 @@ def lambda_handler(event: Dict, context: Any) -> Dict: 'get_sop': get_sop, } - # Strip any caller-supplied server-only fields (approval bypass, injected - # caller identity) so an agent cannot forge them, then inject the trusted - # caller identity extracted from the JWT for use by the approval workflow. + # Strip any caller-supplied server-only fields (injected caller identity) + # so an agent cannot forge them, then inject the trusted caller identity + # extracted from the JWT for use by the approval notifications. if isinstance(event, dict): for _k in [k for k in list(event.keys()) if isinstance(k, str) and k.startswith('_')]: event.pop(_k, None) event['_caller_client_id'] = caller.get('client_id') event['_caller_sub'] = caller.get('sub') - # Restricted tools are only registered when explicitly enabled. There are - # currently no restricted tools defined — the invasive tcpdump capture/analyze - # tools were removed. New restricted tools can be added to this map. - _restricted_tool_map: Dict = {} + # Restricted tools are only registered when explicitly enabled + # (ENABLED_RESTRICTED_TOOLS). tcpdump_capture is ADDITIONALLY gated by the + # human-in-the-loop approval workflow — enabling it here only exposes the + # tool; every capture still pauses at a native SSM aws:approve step. + _restricted_tool_map: Dict = { + 'tcpdump_capture': tcpdump_capture, + 'tcpdump_analyze': tcpdump_analyze, + } for rt_name, rt_func in _restricted_tool_map.items(): if rt_name in ENABLED_RESTRICTED_TOOLS: tools[rt_name] = rt_func @@ -3628,203 +3813,306 @@ def error_response(status_code: int, message: str, details: Dict = None) -> Dict def _approval_configured() -> bool: - """True when the approval workflow infrastructure is wired up.""" - return bool(APPROVAL_TABLE_NAME) + """True when the SSM-native approval workflow is wired up.""" + return bool(COLLECT_APPROVAL_DOCUMENT and APPROVAL_APPROVERS) + + +def console_automation_url(region: str, execution_id: str) -> str: + """Deep link to the SSM console page where approvers Approve/Deny an execution.""" + return ( + f'https://{region}.console.aws.amazon.com/systems-manager/automation/' + f'execution/{execution_id}?region={region}' + ) -def create_collection_approval(tool_name: str, target: str, region: str, arguments: Dict) -> Dict: +def notify_approvers(tool_name: str, target: str, region: str, execution_id: str, + arguments: Dict) -> None: """ - Create a PENDING approval, notify approvers via SNS, and return a - 'pending_approval' response for the agent. The approve/deny link carries a - one-time secret token that is delivered ONLY to humans (via SNS) — it is - never returned to the caller, so the agent cannot approve its own request. + Publish a rich notification with the SSM console deep link. This complements + the bare-bones notification the `aws:approve` step itself sends to the same + topic. Best-effort — the console approval card exists regardless. """ - approval_id = uuid.uuid4().hex - token = secrets.token_urlsafe(32) - token_hash = hashlib.sha256(token.encode('utf-8')).hexdigest() - now = int(time.time()) - ttl = now + APPROVAL_TTL_SECONDS + if not APPROVAL_TOPIC_ARN: + return requested_by = str( arguments.get('_caller_client_id') or arguments.get('_caller_sub') or 'unknown' ) - - dynamodb_client.put_item( - TableName=APPROVAL_TABLE_NAME, - Item={ - 'approvalId': {'S': approval_id}, - 'tokenHash': {'S': token_hash}, - 'tool': {'S': tool_name}, - 'target': {'S': target}, - 'region': {'S': region}, - 'status': {'S': 'PENDING'}, - 'requestedBy': {'S': requested_by}, - 'createdAt': {'N': str(now)}, - 'ttl': {'N': str(ttl)}, - }, - ) - - # Two delivery modes for the approve/deny action. The public Function URL - # is opt-in only: account guardrails can strip public Lambda policies and - # silently break the links, so the default is an IAM-authenticated direct - # invoke of the approval handler — no public endpoint involved. - if APPROVAL_BASE_URL: - base = APPROVAL_BASE_URL.rstrip('/') - approve_action = f"{base}/?approvalId={approval_id}&token={token}&decision=approve" - deny_action = f"{base}/?approvalId={approval_id}&token={token}&decision=deny" - action_hint = "The link contains a one-time secret — do not forward it." - elif APPROVAL_FUNCTION_NAME: - def _invoke_cmd(decision: str) -> str: - payload = json.dumps({ - 'approvalId': approval_id, 'token': token, 'decision': decision, - }) - return ( - f"aws lambda invoke --function-name {APPROVAL_FUNCTION_NAME} " - f"--region {DEFAULT_REGION} --cli-binary-format raw-in-base64-out " - f"--payload '{payload}' /dev/stdout" - ) - approve_action = _invoke_cmd('approve') - deny_action = _invoke_cmd('deny') - action_hint = ( - "Run the command with YOUR AWS credentials (requires lambda:InvokeFunction " - "on the approval handler). The payload contains a one-time secret — do not forward it." + try: + sns_client.publish( + TopicArn=APPROVAL_TOPIC_ARN, + Subject=f'[EKS Diag MCP] Approval needed: {tool_name} on {target}'[:100], + Message=( + f"An agent requested '{tool_name}', which starts SSM log collection on " + f"{target} (region {region}).\n\n" + f"Requested by client: {requested_by}\n" + f"Execution ID: {execution_id}\n\n" + f"Approve or deny in the AWS Systems Manager console:\n" + f"{console_automation_url(region, execution_id)}\n\n" + f"(You must be signed in as one of the designated approvers and have " + f"ssm:SendAutomationSignal permission. The execution stays paused at the " + f"approval step until you decide; it times out if nobody responds.)" + ), ) - else: - approve_action = deny_action = '(approval endpoint not configured)' - action_hint = '' + except Exception as e: + logger.error(f'Failed to publish approval notification: {e}') - if APPROVAL_TOPIC_ARN: - try: - sns_client.publish( - TopicArn=APPROVAL_TOPIC_ARN, - Subject=f'[EKS Diag MCP] Approval needed: {tool_name} on {target}'[:100], - Message=( - f"An agent requested '{tool_name}', which starts SSM log collection on " - f"{target} (region {region}).\n\n" - f"Requested by client: {requested_by}\n" - f"Approval ID: {approval_id}\n\n" - f"APPROVE:\n{approve_action}\n\n" - f"DENY:\n{deny_action}\n\n" - f"This request expires in {APPROVAL_TTL_SECONDS // 60} minutes. " - f"{action_hint}" - ), - ) - except Exception as e: - logger.error(f'Failed to publish approval notification: {e}') - return success_response({ +def _pending_approval_response(tool_name: str, target: str, region: str, + execution_id: str, extra: Optional[Dict] = None) -> Dict: + """Standard 'pending human approval' response for approval-gated executions.""" + url = console_automation_url(region, execution_id) + if APPROVAL_EMAILS_CONFIGURED: + notify_note = 'Approvers were also notified by email via SNS.' + else: + notify_note = ( + 'No email subscriptions are configured on the approval SNS topic, so ' + 'nobody is notified automatically — share the console link with an ' + 'approver directly.' + ) + payload = { 'status': 'pending_approval', - 'approvalId': approval_id, + 'executionId': execution_id, 'tool': tool_name, 'target': target, 'region': region, + 'approvalConsoleUrl': url, 'message': ( - f"'{tool_name}' requires human approval before it runs SSM on {target}. " - f"An approval request was sent to the operators. Once a human approves it, " - f"re-call {tool_name} with the SAME arguments plus approvalId=\"{approval_id}\"." + f"'{tool_name}' requires human approval before SSM log collection runs on " + f"{target}. The SSM Automation execution has started and is PAUSED at a " + f"native aws:approve step. A designated approver must approve it in the " + f"AWS Systems Manager console: {url} — {notify_note} " + f"Once approved, collection proceeds automatically." ), - 'expiresInSeconds': APPROVAL_TTL_SECONDS, + 'humanApproval': { + 'state': 'pending', + 'consoleUrl': url, + 'howToApprove': ( + 'Open the console link, review the request, and choose Approve or Deny ' + 'on the waitForHumanApproval step (or run: aws ssm send-automation-signal ' + f'--automation-execution-id {execution_id} --signal-type Approve ' + f'--region {region}).' + ), + }, + 'suggestedPollIntervalSeconds': 30, + 'polling': { + 'intervalSeconds': 30, + 'maxAttempts': 10, + 'serverSideWaitSeconds': APPROVAL_WAIT_SECONDS, + 'onExhausted': 'stop polling and ask the user to get the request approved', + }, 'nextStep': ( - f'Wait for a human to approve, then call {tool_name}(..., ' - f'approvalId="{approval_id}"). Re-calling before approval returns pending.' + f'Share the console link with an approver, then call ' + f'status(executionId="{execution_id}") repeatedly, up to 10 times. Each call ' + f'waits up to {APPROVAL_WAIT_SECONDS}s server-side while approval is pending, ' + f'so just call again immediately after each response — do NOT stop and wait ' + f'for the user to confirm approval. Collection continues automatically once ' + f'approved (no re-call of {tool_name} is needed). If humanApproval.state is ' + f'still "pending" after 10 calls, stop and ask the user to get it approved.' ), - }) + 'task': { + 'taskId': execution_id, + 'state': 'running', + 'message': 'Waiting for human approval in the AWS Systems Manager console', + 'progress': 0, + }, + } + if extra: + payload.update(extra) + return success_response(payload) -def consume_collection_approval(approval_id: str, tool_name: str, target: str) -> Optional[Dict]: +def enforce_approval_preconditions(target_region: str) -> Optional[Dict]: """ - Validate and atomically consume an approval. Returns None when the request - is approved and may proceed, otherwise a response dict (pending / denied / - expired / mismatched / already-used) to return to the caller. + Fail-closed checks for the approval-gated path (M1/M2). Returns None when the + wrapper document can be started, or an error response. """ - try: - resp = dynamodb_client.get_item( - TableName=APPROVAL_TABLE_NAME, - Key={'approvalId': {'S': approval_id}}, - ) - except Exception as e: - return error_response(500, f'Failed to look up approval: {e}') - - item = resp.get('Item') - if not item: + if not _approval_configured(): return error_response( - 403, - f'Unknown or expired approvalId. Request a new approval by calling ' - f'{tool_name} without an approvalId.', + 503, + 'Human approval is required for collection, but the approval workflow is not ' + 'configured (COLLECT_APPROVAL_DOCUMENT/APPROVAL_APPROVERS unset). ' + 'Contact the operator.', ) - - if item.get('tool', {}).get('S') != tool_name or item.get('target', {}).get('S') != target: + if target_region != DEFAULT_REGION: return error_response( - 403, - 'approvalId does not match this tool and target. Request a fresh approval.', + 400, + f'Approval-gated collection is only available in {DEFAULT_REGION}: the ' + f'approval wrapper document is a regional SSM document deployed with this ' + f'stack. Requested region: {target_region}. Deploy the stack in that region, ' + f'or (test deployments only) set REQUIRE_COLLECTION_APPROVAL=false.', ) + return None - now = int(time.time()) - ttl = int(item.get('ttl', {}).get('N', '0') or 0) - if ttl and now > ttl: - return error_response(403, 'This approval has expired. Request a new one.') - status = item.get('status', {}).get('S', '') - if status == 'PENDING': - return success_response({ - 'status': 'pending_approval', - 'approvalId': approval_id, - 'message': 'Approval is still pending. Ask an approver to use the link, then retry.', - 'nextStep': f'Retry {tool_name}(..., approvalId="{approval_id}") after approval.', - }) - if status == 'DENIED': - return error_response(403, 'This request was denied by an approver.') - if status == 'CONSUMED': - return error_response(403, 'This approval was already used (approvals are single-use). Request a new one.') - if status != 'APPROVED': - return error_response(403, f'Approval is not usable (status={status}).') - - # Atomically flip APPROVED -> CONSUMED so an approval can be used only once. +def start_collection_with_approval(instance_id: str, target_region: str, + arguments: Dict) -> Dict: + """ + Start the approval-gated wrapper automation for a single instance. The + wrapper pauses at aws:approve until a human approves in the SSM console, + then runs AWSSupport-CollectEKSInstanceLogs automatically. + """ + regional_ssm = get_regional_client('ssm', target_region) try: - dynamodb_client.update_item( - TableName=APPROVAL_TABLE_NAME, - Key={'approvalId': {'S': approval_id}}, - UpdateExpression='SET #s = :consumed', - ConditionExpression='#s = :approved', - ExpressionAttributeNames={'#s': 'status'}, - ExpressionAttributeValues={ - ':consumed': {'S': 'CONSUMED'}, - ':approved': {'S': 'APPROVED'}, + response = regional_ssm.start_automation_execution( + DocumentName=COLLECT_APPROVAL_DOCUMENT, + Parameters={ + 'EKSInstanceId': [instance_id], + 'LogDestination': [LOGS_BUCKET], + 'AutomationAssumeRole': [SSM_AUTOMATION_ROLE_ARN], + 'Approvers': APPROVAL_APPROVERS, + 'SNSTopicArn': [APPROVAL_TOPIC_ARN], }, ) - except dynamodb_client.exceptions.ConditionalCheckFailedException: - return error_response(403, 'This approval was already used or changed state. Request a new one.') except Exception as e: - return error_response(500, f'Failed to consume approval: {e}') + return error_response(500, f'Failed to start approval-gated collection: {str(e)}') + + execution_id = response['AutomationExecutionId'] + + idempotency_token = arguments.get('idempotencyToken') + if idempotency_token: + store_idempotency_mapping(instance_id, idempotency_token, execution_id) + store_execution_region(execution_id, target_region) + + notify_approvers('collect', instance_id, target_region, execution_id, arguments) + return _pending_approval_response( + 'collect', instance_id, target_region, execution_id, + extra={'instanceId': instance_id, 's3Bucket': LOGS_BUCKET}, + ) + + +# ── Wrapper-execution status helpers (used by status/batch_status) ── + +APPROVAL_STEP_NAME = 'waitForHumanApproval' - return None # approved and consumed — caller may proceed +# Server-side long-poll budget while an approval is pending. Agents generally +# cannot sleep between tool calls, so the status tools hold the request open +# for up to this long (checking SSM every APPROVAL_WAIT_CHECK_SECONDS) before +# responding — back-to-back agent polls are then naturally paced ~30s apart, +# and the response returns early the moment a human decides. +APPROVAL_WAIT_SECONDS = 25 +APPROVAL_WAIT_CHECK_SECONDS = 5 -def enforce_collection_approval(tool_name: str, target: str, region: str, arguments: Dict) -> Optional[Dict]: +def _approval_step_pending(execution: Dict) -> bool: + """True while the wrapper execution is paused at the aws:approve step.""" + for step in execution.get('StepExecutions', []) or []: + if step.get('StepName') == APPROVAL_STEP_NAME: + return step.get('StepStatus') in ('Pending', 'InProgress', 'Waiting') + return False + + +def wait_for_approval_decision(regional_ssm, execution_id: str, execution: Dict) -> Dict: """ - Approval gate for mutating collection tools (M1/M2). Returns None when the - call may proceed (approval disabled, or a valid approval was consumed), or a - response dict (pending / denied / error) that the caller must return as-is. + Long-poll SSM while the approval is pending, up to APPROVAL_WAIT_SECONDS. + Returns the most recent execution snapshot (early when a human decides). """ - # Internal bypass: set ONLY by server-side code (e.g. batch_collect after it - # obtained one approval for the whole batch). Caller-supplied '_'-prefixed - # keys are stripped by the handler, so an agent cannot forge this. - if arguments.get('_approval_bypass') is True: - return None - if not REQUIRE_COLLECTION_APPROVAL: - return None - if not _approval_configured(): - # Fail closed: approval is required but the workflow isn't configured. - return error_response( - 503, - 'Human approval is required for collection, but the approval workflow is not ' - 'configured (APPROVAL_TABLE_NAME/APPROVAL_TOPIC_ARN unset). Contact the operator.', + deadline = time.time() + APPROVAL_WAIT_SECONDS + latest = execution + while _approval_step_pending(latest) and time.time() < deadline: + time.sleep(APPROVAL_WAIT_CHECK_SECONDS) + try: + latest = regional_ssm.get_automation_execution( + AutomationExecutionId=execution_id + )['AutomationExecution'] + except Exception: + break # transient read error — return what we have + return latest + + +def _is_approval_wrapper(document_name: str) -> bool: + """True when an execution was started from one of the approval wrapper docs.""" + if not document_name: + return False + return document_name in ( + COLLECT_APPROVAL_DOCUMENT, BATCH_APPROVAL_DOCUMENT, TCPDUMP_APPROVAL_DOCUMENT, + ) + + +def _step_output_values(step: Dict, key: str) -> List[str]: + """Extract a list-valued output from a StepExecution, defensively.""" + outputs = step.get('Outputs', {}) or {} + values = outputs.get(key, []) + return [v for v in values if isinstance(v, str)] + + +def augment_wrapper_status(execution: Dict, result: Dict, target_region: str) -> None: + """ + Enrich a status result for an approval-wrapper execution: expose the human + approval state (pending / approved / denied-or-expired), the SSM console + deep link, and the child collection execution id once the approval clears. + Mutates `result` in place. + """ + execution_id = execution.get('AutomationExecutionId', result.get('executionId', '')) + url = console_automation_url(target_region, execution_id) + steps = execution.get('StepExecutions', []) or [] + approve_step = next((s for s in steps if s.get('StepName') == APPROVAL_STEP_NAME), None) + if approve_step is None: + return + + approve_status = approve_step.get('StepStatus', '') + + if approve_status in ('Pending', 'InProgress', 'Waiting'): + result['humanApproval'] = { + 'state': 'pending', + 'consoleUrl': url, + 'message': 'Waiting for a human to approve in the AWS Systems Manager console.', + } + result['suggestedPollIntervalSeconds'] = 30 + result['polling'] = { + 'intervalSeconds': 30, + 'maxAttempts': 10, + 'serverSideWaitSeconds': APPROVAL_WAIT_SECONDS, + 'onExhausted': 'stop polling and ask the user to get the request approved', + } + result['nextStep'] = ( + f'A human must approve in the SSM console ({url}). ' + f'Call status again immediately (each call already waits up to ' + f'{APPROVAL_WAIT_SECONDS}s server-side while pending), up to 10 calls total ' + f'without waiting for the user; if still pending after that, stop and ask ' + f'the user to get the request approved.' ) - approval_id = arguments.get('approvalId') - if not approval_id: - return create_collection_approval(tool_name, target, region, arguments) - return consume_collection_approval(approval_id, tool_name, target) + if 'task' in result: + result['task']['message'] = 'Waiting for human approval in the SSM console' + return + + if approve_status in ('Failed', 'TimedOut', 'Cancelled'): + result['humanApproval'] = { + 'state': 'denied_or_expired', + 'consoleUrl': url, + 'message': ( + 'The approval was denied by an approver or timed out without a decision. ' + 'No collection ran.' + ), + } + result['nextStep'] = 'Re-call collect to request a fresh approval if still needed.' + if 'task' in result: + result['task']['state'] = 'failed' + result['task']['message'] = 'Human approval denied or expired — collection did not run' + return + + # Approved — surface child execution id(s) from the post-approval step(s) + result['humanApproval'] = {'state': 'approved', 'consoleUrl': url} + for step in steps: + if step.get('StepName') == APPROVAL_STEP_NAME: + continue + child_ids = _step_output_values(step, 'ExecutionId') + if child_ids: + result['childExecutionId'] = child_ids[0] + batch_children = _step_output_values(step, 'Executions') + if batch_children: + result['childExecutions'] = _parse_batch_children(batch_children) + + +def _parse_batch_children(raw_entries: List[str]) -> List[Dict]: + """Parse 'instanceId|executionId' entries emitted by the batch fan-out step.""" + children = [] + for entry in raw_entries: + if '|' in entry: + iid, _, eid = entry.partition('|') + children.append({'instanceId': iid.strip(), 'executionId': eid.strip()}) + return children def start_log_collection(arguments: Dict) -> Dict: @@ -3885,20 +4173,33 @@ def start_log_collection(arguments: Dict) -> Dict: if idempotency_token: existing = find_execution_by_idempotency_token(instance_id, idempotency_token) if existing: - return success_response({ + existing_region = existing.get('region', target_region) + response_data = { 'message': 'Returning existing execution (idempotent)', 'executionId': existing['executionId'], 'status': existing['status'], 'instanceId': instance_id, - 'region': target_region, - 'idempotent': True - }) - - # Human-in-the-loop approval gate (M1). Blocks the SSM call until a human - # approves out-of-band. Returns a 'pending_approval' response on first call. - approval_response = enforce_collection_approval('collect', instance_id, target_region, arguments) - if approval_response is not None: - return approval_response + 'region': existing_region, + 'idempotent': True, + } + # Mirror the augmentation get_collection_status applies — without + # it, a retried collect() with the same token while the wrapper is + # paused at aws:approve reports a bare "InProgress" and the agent + # cannot tell the run is waiting on a human approval. + if _is_approval_wrapper(existing.get('documentName', '')): + augment_wrapper_status(existing['_execution'], response_data, existing_region) + return success_response(response_data) + + # Human-in-the-loop approval gate (M1). When enabled, collection runs via a + # wrapper document whose FIRST step is the native aws:approve action — the + # execution pauses inside SSM until a designated human approves it in the + # Systems Manager console, then the collection step runs automatically. + # The agent only needs to poll status(); no second collect call is required. + if REQUIRE_COLLECTION_APPROVAL: + precondition_error = enforce_approval_preconditions(target_region) + if precondition_error is not None: + return precondition_error + return start_collection_with_approval(instance_id, target_region, arguments) try: # Start SSM Automation in the target region @@ -3988,7 +4289,14 @@ def get_collection_status(arguments: Dict) -> Dict: AutomationExecutionId=execution_id ) execution = response['AutomationExecution'] - + + # Approval-wrapper executions: hold the request open (server-side + # long-poll) while the aws:approve step is pending, so agent polls are + # paced ~30s apart even when the agent cannot sleep between calls. + if (_is_approval_wrapper(execution.get('DocumentName', '')) + and _approval_step_pending(execution)): + execution = wait_for_approval_decision(regional_ssm, execution_id, execution) + status = execution['AutomationExecutionStatus'] result = { @@ -4053,7 +4361,12 @@ def get_collection_status(arguments: Dict) -> Dict: 'message': result.get('failureReason', f'SSM status: {status}'), 'progress': result.get('progress', 0), } - + + # Approval-wrapper executions: expose the human-approval state (pending / + # approved / denied), the SSM console deep link, and child execution ids. + if _is_approval_wrapper(result.get('documentName', '')): + augment_wrapper_status(execution, result, target_region) + return success_response({'automation': result}) except regional_ssm.exceptions.AutomationExecutionNotFoundException: @@ -4459,9 +4772,17 @@ def read_log_chunk(arguments: Dict) -> Dict: line_count = arguments.get('lineCount', DEFAULT_LINE_COUNT) # E4: restrict to log-bundle keys; blocks arbitrary reads and path traversal. - # When instanceId is supplied, the key must belong to that instance (rejects - # reading another instance's bundle). - key_err = validate_log_key(log_key, expected_instance_id=arguments.get('instanceId')) + # instanceId is mandatory so the key is always scoped to the instance under + # investigation — omitting it would otherwise allow lateral reads of other + # instances' bundles. + instance_id = arguments.get('instanceId') + if not instance_id: + return error_response( + 400, + 'instanceId is required: read() only returns log content for the ' + 'instance under investigation.' + ) + key_err = validate_log_key(log_key, expected_instance_id=instance_id) if key_err: return key_err @@ -5236,8 +5557,16 @@ def get_artifact_reference(arguments: Dict) -> Dict: expiration_seconds = min(arguments.get('expirationMinutes', 0) * 60 or PRESIGNED_URL_EXPIRATION, PRESIGNED_URL_EXPIRATION) # E4: restrict to log-bundle keys; blocks arbitrary reads and path traversal. - # When instanceId is supplied, the key must belong to that instance. - key_err = validate_log_key(log_key, expected_instance_id=arguments.get('instanceId')) + # instanceId is mandatory so presigned URLs are always scoped to the + # instance under investigation. + instance_id = arguments.get('instanceId') + if not instance_id: + return error_response( + 400, + 'instanceId is required: artifact() only returns URLs for the ' + 'instance under investigation.' + ) + key_err = validate_log_key(log_key, expected_instance_id=instance_id) if key_err: return key_err @@ -6560,26 +6889,90 @@ def batch_collect(arguments: Dict) -> Dict: 'message': f'{len(filtered_nodes)} nodes grouped into {len(bucket_list)} buckets. Will collect from {total_planned} representative nodes. Re-run with dryRun=false to proceed.', }) - # M2: a single human approval authorizes the whole batch (target = the - # cluster). Only reached for real execution — dry runs returned above. - approval_response = enforce_collection_approval('batch_collect', cluster_name, target_region, arguments) - if approval_response is not None: - return approval_response - - # 7. Execute collections + # 7. Real execution — dry runs returned above. batch_id = hashlib.sha256(f"{cluster_name}-{datetime.utcnow().isoformat()}".encode(), usedforsecurity=False).hexdigest()[:12] + sampled_ids = [iid for bucket in bucket_list for iid in bucket['sampleNodes']] + + # M2: when approval is required, start the batch wrapper document ONCE. + # Its first step is the native aws:approve action — a single human + # approval in the SSM console authorizes the whole batch — then an + # aws:executeScript step fans out one collection per sampled node. + if REQUIRE_COLLECTION_APPROVAL: + precondition_error = enforce_approval_preconditions(target_region) + if precondition_error is not None: + return precondition_error + if not BATCH_APPROVAL_DOCUMENT: + return error_response( + 503, + 'Human approval is required but the batch approval document is not ' + 'configured (BATCH_APPROVAL_DOCUMENT unset). Contact the operator.', + ) + try: + wrapper_resp = regional_ssm.start_automation_execution( + DocumentName=BATCH_APPROVAL_DOCUMENT, + Parameters={ + 'InstanceIds': sampled_ids, + 'LogDestination': [LOGS_BUCKET], + 'AutomationAssumeRole': [SSM_AUTOMATION_ROLE_ARN], + 'Approvers': APPROVAL_APPROVERS, + 'SNSTopicArn': [APPROVAL_TOPIC_ARN], + }, + ) + except Exception as e: + return error_response(500, f'Failed to start approval-gated batch collection: {str(e)}') + + batch_execution_id = wrapper_resp['AutomationExecutionId'] + store_execution_region(batch_execution_id, target_region) + try: + s3_client.put_object( + Bucket=LOGS_BUCKET, + Key=f"batches/{batch_id}/metadata.json", + Body=json.dumps({ + 'batchId': batch_id, + 'clusterName': cluster_name, + 'region': target_region, + 'createdAt': datetime.utcnow().isoformat(), + 'approvalExecutionId': batch_execution_id, + 'plannedInstanceIds': sampled_ids, + 'buckets': bucket_list, + 'executions': [], + }, default=str), + ContentType='application/json', + ) + except Exception: + pass + + notify_approvers('batch_collect', cluster_name, target_region, batch_execution_id, arguments) + return _pending_approval_response( + 'batch_collect', cluster_name, target_region, batch_execution_id, + extra={ + 'batchId': batch_id, + 'clusterName': cluster_name, + 'plannedCollections': total_planned, + 'plannedInstanceIds': sampled_ids, + 'buckets': bucket_list, + 'nextStep': ( + f'A single human approval in the SSM console authorizes the whole ' + f'batch ({total_planned} nodes). Poll batch_status(batchId=' + f'"{batch_id}") every 30 seconds, up to 10 attempts, without ' + f'waiting for the user — the fan-out happens automatically after ' + f'approval. If still pending after 10 attempts, stop polling and ' + f'ask the user to get the request approved.' + ), + }, + ) + + # Approval disabled (supervised/test deployments): fan out directly. executions = [] for bucket in bucket_list: for iid in bucket['sampleNodes']: try: - # Reuse existing collect logic. The batch already carries one - # human approval, so bypass the per-instance approval gate. + # Reuse existing collect logic. collect_args = { 'instanceId': iid, 'region': target_region, 'idempotencyToken': f"batch-{batch_id}-{iid}", - '_approval_bypass': True, } result = start_log_collection(collect_args) result_body = json.loads(result.get('body', '{}')) @@ -6667,6 +7060,7 @@ def batch_status(arguments: Dict) -> Dict: batch_id = arguments.get('batchId') # If batchId provided, load execution IDs from stored metadata + meta = None if batch_id and not execution_ids: try: meta_result = safe_s3_read(f"batches/{batch_id}/metadata.json") @@ -6679,6 +7073,79 @@ def batch_status(arguments: Dict) -> Dict: except Exception: pass + # Approval-gated batches: a wrapper execution owns the approval + fan-out. + # Resolve its state — pending approval, denied, or fan-out child executions. + if not execution_ids and meta and meta.get('approvalExecutionId'): + wrapper_id = meta['approvalExecutionId'] + wrapper_region = meta.get('region', DEFAULT_REGION) + try: + wrapper_ssm = get_regional_client('ssm', wrapper_region) + wrapper_exec = wrapper_ssm.get_automation_execution( + AutomationExecutionId=wrapper_id + )['AutomationExecution'] + except Exception as e: + return error_response(500, f'Failed to look up batch approval execution: {str(e)}') + + # Server-side long-poll while the batch approval is pending (see + # wait_for_approval_decision) so agent polls are paced ~30s apart. + if _approval_step_pending(wrapper_exec): + wrapper_exec = wait_for_approval_decision(wrapper_ssm, wrapper_id, wrapper_exec) + + probe: Dict = {'executionId': wrapper_id} + augment_wrapper_status(wrapper_exec, probe, wrapper_region) + approval = probe.get('humanApproval', {}) + + if approval.get('state') == 'pending': + return success_response({ + 'allComplete': False, + 'batchId': batch_id, + 'status': 'pending_approval', + 'approvalExecutionId': wrapper_id, + 'humanApproval': approval, + 'approvalConsoleUrl': approval.get('consoleUrl'), + 'suggestedPollIntervalSeconds': 30, + 'polling': { + 'intervalSeconds': 30, + 'maxAttempts': 10, + 'serverSideWaitSeconds': APPROVAL_WAIT_SECONDS, + 'onExhausted': 'stop polling and ask the user to get the request approved', + }, + 'nextStep': ( + 'A human must approve the batch in the AWS Systems Manager console ' + f"({approval.get('consoleUrl')}). Call batch_status again immediately " + f'(each call already waits up to {APPROVAL_WAIT_SECONDS}s server-side ' + f'while pending), up to 10 calls total without waiting for the user; ' + f'if still pending after that, stop and ask the user to get the ' + f'request approved.' + ), + }) + if approval.get('state') == 'denied_or_expired': + return success_response({ + 'allComplete': True, + 'batchId': batch_id, + 'status': 'denied_or_expired', + 'approvalExecutionId': wrapper_id, + 'humanApproval': approval, + 'nextStep': 'Approval was denied or expired — no collections ran. ' + 'Re-run batch_collect if still needed.', + }) + + execution_ids = [ + c['executionId'] for c in probe.get('childExecutions', []) + if c.get('executionId') + ] + if not execution_ids: + return success_response({ + 'allComplete': False, + 'batchId': batch_id, + 'status': wrapper_exec.get('AutomationExecutionStatus', 'InProgress'), + 'approvalExecutionId': wrapper_id, + 'humanApproval': approval, + 'message': 'Approved — the fan-out step is starting child collections.', + 'suggestedPollIntervalSeconds': 15, + 'nextStep': 'Poll batch_status again in 15 seconds.', + }) + if not execution_ids: return error_response(400, 'executionIds list or batchId is required') @@ -6697,7 +7164,7 @@ def _poll(eid): status = execution['AutomationExecutionStatus'] # Extract instanceId from parameters params = execution.get('Parameters', {}) - instance_id = params.get('InstanceId', [None])[0] if params.get('InstanceId') else None + instance_id = (params.get('EKSInstanceId') or params.get('InstanceId') or [None])[0] return { 'executionId': eid, 'instanceId': instance_id, @@ -8123,6 +8590,15 @@ def _network_assessment(issues: List[Dict]) -> str: if critical: sections = set(i['section'] for i in critical) + # Proactive tcpdump escalation for non-kube-proxy critical issues + if not kube_proxy_down: + return ( + f"CRITICAL — {len(critical)} critical networking issues in: {', '.join(sections)}. Immediate investigation needed. " + "ESCALATION: If log analysis is inconclusive, use tcpdump_capture (requires the tool to be " + "enabled and a human approval in the SSM console) to capture live traffic " + "on the affected node. Target port 53 for DNS issues, port 443/6443 for API server issues, " + "or the pod IP for pod-to-pod connectivity problems." + ) return f"CRITICAL — {len(critical)} critical networking issues in: {', '.join(sections)}. Immediate investigation needed." return f"WARNING — {len(issues)} non-critical networking issues found. Review recommended." @@ -8840,3 +9316,2482 @@ def _storage_assessment(issues: List[Dict]) -> str: return f"CRITICAL — {len(critical)} critical storage issues in: {', '.join(sections)}. Immediate investigation needed." return f"WARNING — {len(issues)} non-critical storage issues found. Review recommended." + +# ============================================================================= +# TCPDUMP — restricted network packet capture tools (opt-in + approval-gated) +# ============================================================================= +# +# tcpdump_capture is the most invasive tool this server exposes: it runs +# tcpdump as root on a worker node (optionally inside a pod's network +# namespace via nsenter). Two independent gates apply: +# 1. Opt-in: both tools are RESTRICTED_TOOLS — absent from the tool surface +# unless ENABLED_RESTRICTED_TOOLS includes them (deploy-time choice). +# 2. Human approval (M3): when REQUIRE_COLLECTION_APPROVAL is true (the +# default), every capture starts a wrapper SSM Automation document whose +# FIRST step is the native aws:approve action. The execution pauses +# inside SSM until a designated approver approves it in the Systems +# Manager console; only then does the runTcpdump step send the capture +# script to the node. The Lambda has no ssm:SendAutomationSignal (it +# cannot approve its own requests) and, in approval mode, no +# ssm:SendCommand (it cannot bypass the wrapper). +# tcpdump_analyze is read-only (S3 reads of completed captures) and needs no +# approval. + +TCPDUMP_RUN_STEP_NAME = 'runTcpdump' + + +def _tcpdump_approval_configured() -> bool: + """True when the tcpdump approval wrapper is wired up.""" + return bool(TCPDUMP_APPROVAL_DOCUMENT and APPROVAL_APPROVERS) + + +def enforce_tcpdump_approval_preconditions(target_region: str) -> Optional[Dict]: + """ + Fail-closed checks for the approval-gated tcpdump path (M3). Returns None + when the wrapper document can be started, or an error response. + """ + if not _tcpdump_approval_configured(): + return error_response( + 503, + 'Human approval is required for packet capture, but the approval workflow ' + 'is not configured (TCPDUMP_APPROVAL_DOCUMENT/APPROVAL_APPROVERS unset). ' + 'Contact the operator.', + ) + if target_region != DEFAULT_REGION: + return error_response( + 400, + f'Approval-gated packet capture is only available in {DEFAULT_REGION}: the ' + f'approval wrapper document is a regional SSM document deployed with this ' + f'stack. Requested region: {target_region}. Deploy the stack in that region, ' + f'or (test deployments only) set REQUIRE_COLLECTION_APPROVAL=false.', + ) + return None + + +def _tcpdump_execution_metadata_key(execution_id: str) -> str: + return f'tcpdump-executions/{execution_id}.json' + + +def _start_tcpdump_with_approval(regional_ssm, instance_id: str, target_region: str, + script: str, duration: int, ns_label: str, + capture_metadata: Dict, arguments: Dict) -> Dict: + """ + Start the approval-gated tcpdump wrapper automation. The wrapper pauses at + aws:approve until a human approves in the SSM console, then its runTcpdump + step sends the capture script to the node via AWS-RunShellScript. + """ + scope = capture_metadata.get('networkNamespace', 'host') + try: + response = regional_ssm.start_automation_execution( + DocumentName=TCPDUMP_APPROVAL_DOCUMENT, + Parameters={ + 'InstanceId': [instance_id], + 'Commands': [script], + 'ExecutionTimeoutSeconds': [str(duration + 120)], + 'DurationSeconds': [str(duration)], + 'Interface': [capture_metadata.get('interface', 'any')], + 'BpfFilter': [capture_metadata.get('filter') or 'none'], + 'CaptureScope': [scope], + 'AutomationAssumeRole': [SSM_AUTOMATION_ROLE_ARN], + 'Approvers': APPROVAL_APPROVERS, + 'SNSTopicArn': [APPROVAL_TOPIC_ARN], + }, + ) + except Exception as e: + return error_response(500, f'Failed to start approval-gated tcpdump: {str(e)}') + + execution_id = response['AutomationExecutionId'] + store_execution_region(execution_id, target_region) + + # Persist capture metadata keyed by the wrapper execution id. Once a human + # approves and the runTcpdump step emits a CommandId, the poll path copies + # this to tcpdump-commands/{commandId}.json so tcpdump_analyze finds it. + try: + s3_client.put_object( + Bucket=LOGS_BUCKET, + Key=_tcpdump_execution_metadata_key(execution_id), + Body=json.dumps({**capture_metadata, 'executionId': execution_id}), + ) + except Exception: + pass # Non-fatal + + notify_approvers( + 'tcpdump_capture', + f'{instance_id} ({scope})', + target_region, execution_id, arguments, + ) + pending = _pending_approval_response( + 'tcpdump_capture', instance_id, target_region, execution_id, + extra={ + 'instanceId': instance_id, + 'durationSeconds': duration, + 'interface': capture_metadata.get('interface'), + 'filter': capture_metadata.get('filter') or 'none', + 'captureScope': capture_metadata.get('captureScope'), + 'networkNamespace': scope, + 's3Key': capture_metadata.get('s3Key'), + 's3Bucket': LOGS_BUCKET, + }, + ) + # The generic pending response points at status(); tcpdump polls go through + # tcpdump_capture(executionId=...) instead, which resolves the CommandId + # after approval and then tracks the capture itself. + try: + body = json.loads(pending['body']) + body['message'] = ( + f"'tcpdump_capture' requires human approval before the capture runs on " + f"{instance_id}{ns_label}. The SSM Automation execution has started and is " + f"PAUSED at a native aws:approve step. A designated approver must approve " + f"it in the AWS Systems Manager console: {body.get('approvalConsoleUrl')}" + ) + body['nextStep'] = ( + f'Share the console link with an approver, then poll with ' + f'tcpdump_capture(executionId="{execution_id}", instanceId="{instance_id}", ' + f'confirmCapture=true) repeatedly, up to 10 times. Each call waits up to ' + f'{APPROVAL_WAIT_SECONDS}s server-side while approval is pending — do NOT ' + f'stop and wait for the user to confirm approval. The capture starts ' + f'automatically once approved (no new tcpdump_capture request is needed). ' + f'If still pending after 10 calls, stop and ask the user to get it approved.' + ) + pending['body'] = json.dumps(body, default=str) + except Exception: + pass + return pending + + +def _poll_tcpdump_wrapper(execution_id: str, instance_id: str, arguments: Dict) -> Dict: + """ + Poll an approval-gated tcpdump wrapper execution. While the aws:approve + step is pending this long-polls SSM (like status() does for collections); + once approved it resolves the runTcpdump step's CommandId, upgrades the + stored capture metadata, and delegates to the Run Command poller. + """ + target_region = get_execution_region(execution_id) or resolve_region(arguments, instance_id) + try: + regional_ssm = get_regional_client('ssm', target_region) + execution = regional_ssm.get_automation_execution( + AutomationExecutionId=execution_id + )['AutomationExecution'] + except Exception as e: + return error_response(500, f'Failed to read tcpdump execution {execution_id}: {str(e)}') + + if _approval_step_pending(execution): + execution = wait_for_approval_decision(regional_ssm, execution_id, execution) + + url = console_automation_url(target_region, execution_id) + steps = execution.get('StepExecutions', []) or [] + approve_step = next( + (s for s in steps if s.get('StepName') == APPROVAL_STEP_NAME), None) + approve_status = (approve_step or {}).get('StepStatus', '') + + if approve_status in ('Pending', 'InProgress', 'Waiting'): + return success_response({ + 'status': 'pending_approval', + 'executionId': execution_id, + 'instanceId': instance_id, + 'region': target_region, + 'approvalConsoleUrl': url, + 'humanApproval': { + 'state': 'pending', + 'consoleUrl': url, + 'message': 'Waiting for a human to approve in the AWS Systems Manager console.', + }, + 'suggestedPollIntervalSeconds': 30, + 'polling': { + 'intervalSeconds': 30, + 'maxAttempts': 10, + 'serverSideWaitSeconds': APPROVAL_WAIT_SECONDS, + 'onExhausted': 'stop polling and ask the user to get the capture approved', + }, + 'nextStep': ( + f'A human must approve in the SSM console ({url}). Call ' + f'tcpdump_capture(executionId="{execution_id}", instanceId="{instance_id}", ' + f'confirmCapture=true) again immediately (each call already waits up to ' + f'{APPROVAL_WAIT_SECONDS}s server-side), up to 10 calls total; if still ' + f'pending after that, stop and ask the user to get it approved.' + ), + 'task': { + 'taskId': execution_id, + 'state': 'running', + 'message': 'Waiting for human approval in the AWS Systems Manager console', + 'progress': 0, + }, + }) + + if approve_status in ('Failed', 'TimedOut', 'Cancelled'): + return error_response(403, 'tcpdump capture approval was denied or expired', { + 'executionId': execution_id, + 'instanceId': instance_id, + 'humanApproval': { + 'state': 'denied_or_expired', + 'consoleUrl': url, + 'message': ( + 'The approval was denied by an approver or timed out without a ' + 'decision. No packet capture ran.' + ), + }, + 'nextStep': 'Re-call tcpdump_capture to request a fresh approval if still needed.', + 'task': { + 'taskId': execution_id, + 'state': 'failed', + 'message': 'Human approval denied or expired — packet capture did not run', + 'progress': 0, + }, + }) + + # Approved — resolve the CommandId emitted by the runTcpdump step. + run_step = next( + (s for s in steps if s.get('StepName') == TCPDUMP_RUN_STEP_NAME), None) + command_ids = _step_output_values(run_step or {}, 'CommandId') + if not command_ids: + exec_status = execution.get('AutomationExecutionStatus', 'InProgress') + if exec_status in ('Failed', 'TimedOut', 'Cancelled'): + return error_response(500, f'tcpdump wrapper execution {exec_status}', { + 'executionId': execution_id, + 'failureReason': parse_failure_reason(execution), + 'consoleUrl': url, + }) + return success_response({ + 'status': 'in_progress', + 'executionId': execution_id, + 'instanceId': instance_id, + 'humanApproval': {'state': 'approved', 'consoleUrl': url}, + 'message': 'Approved — capture command is being dispatched to the node.', + 'nextStep': ( + f'Poll again in ~15s with tcpdump_capture(executionId="{execution_id}", ' + f'instanceId="{instance_id}", confirmCapture=true).' + ), + 'task': { + 'taskId': execution_id, + 'state': 'running', + 'message': 'Approved — dispatching tcpdump to the node', + 'progress': 5, + }, + }) + + command_id = command_ids[0] + + # Upgrade the execution-keyed metadata to command-keyed metadata (the shape + # _poll_tcpdump_status and tcpdump_analyze expect). Refresh startedAt so + # staleness/elapsed math reflects when the capture actually started, not + # when approval was requested. + try: + try: + s3_client.head_object( + Bucket=LOGS_BUCKET, Key=f'tcpdump-commands/{command_id}.json') + metadata_exists = True + except Exception: + metadata_exists = False + if not metadata_exists: + meta_resp = s3_client.get_object( + Bucket=LOGS_BUCKET, + Key=_tcpdump_execution_metadata_key(execution_id), + ) + metadata = json.loads(meta_resp['Body'].read().decode('utf-8')) + metadata['commandId'] = command_id + step_start = (run_step or {}).get('ExecutionStartTime') + if step_start is not None: + try: + metadata['startedAt'] = step_start.strftime('%Y%m%dT%H%M%SZ') + except Exception: + pass + s3_client.put_object( + Bucket=LOGS_BUCKET, + Key=f'tcpdump-commands/{command_id}.json', + Body=json.dumps(metadata), + ) + except Exception: + pass # Non-fatal — the Run Command poller falls back to stdout parsing + + result = _poll_tcpdump_status(command_id, instance_id, arguments) + try: + body = json.loads(result['body']) + if isinstance(body, dict): + body['executionId'] = execution_id + body['humanApproval'] = {'state': 'approved', 'consoleUrl': url} + result['body'] = json.dumps(body, default=str) + except Exception: + pass + return result + + +def tcpdump_capture(arguments: Dict) -> Dict: + """ + Run tcpdump on an EKS worker node via SSM Run Command for a specified duration, + then upload the pcap file to S3. + + Inputs: + instanceId: EC2 instance ID (required) + durationSeconds: Capture duration in seconds (default: 120, max: 300) + interface: Network interface to capture on (default: "any") + filter: BPF filter expression (e.g., "port 443", "host 10.0.0.1") (optional) + region: AWS region where the instance runs (optional, auto-detected) + + Returns: + commandId for async polling, or capture results if already complete + """ + instance_id = arguments.get('instanceId') + if not instance_id: + return error_response(400, 'instanceId is required') + + if not re.match(r'^i-[0-9a-f]{8,17}$', instance_id): + return error_response(400, f'Invalid instanceId format: {instance_id}') + + duration = int(arguments.get('durationSeconds', 120)) + if duration < 10 or duration > 300: + return error_response(400, 'durationSeconds must be between 10 and 300') + + interface = arguments.get('interface', 'any') + # Sanitize interface name to prevent injection — strict allowlist + if not re.match(r'^[a-zA-Z0-9\-\.]+$', interface): + return error_response(400, f'Invalid interface name: {interface}') + + bpf_filter = arguments.get('filter', '') + # Allowlist-based BPF filter validation (replaces denylist approach) + bpf_error = validate_bpf_filter(bpf_filter) + if bpf_error: + return error_response(400, f'Invalid BPF filter: {bpf_error}') + + # Container/pod namespace support + container_pid = arguments.get('containerPid', '') + if container_pid: + container_pid = str(container_pid).strip() + if not re.match(r'^\d+$', container_pid): + return error_response(400, f'Invalid containerPid — must be a numeric PID: {container_pid}') + + pod_name = arguments.get('podName', '').strip() + pod_namespace = arguments.get('podNamespace', 'default').strip() + if pod_name and not re.match(r'^[a-zA-Z0-9][a-zA-Z0-9\-\.]{0,252}$', pod_name): + return error_response(400, f'Invalid podName: {pod_name}') + if pod_namespace and not re.match(r'^[a-zA-Z0-9][a-zA-Z0-9\-]{0,62}$', pod_namespace): + return error_response(400, f'Invalid podNamespace: {pod_namespace}') + + # Can't specify both podName and containerPid + if pod_name and container_pid: + return error_response(400, 'Specify either podName or containerPid, not both') + + # Status polls (before the confirmation gate — a poll is not a new capture): + # executionId polls an approval-gated wrapper execution; commandId polls + # the underlying SSM Run Command directly. + execution_id = arguments.get('executionId') + if execution_id: + return _poll_tcpdump_wrapper(execution_id, instance_id, arguments) + command_id = arguments.get('commandId') + if command_id: + return _poll_tcpdump_status(command_id, instance_id, arguments) + + # ── Confirmation gate (T2 mitigation) ── + # tcpdump installs packages, enters container namespaces, and captures raw + # network traffic as root. Require explicit confirmation to proceed. + confirm = arguments.get('confirmCapture', False) + if confirm is not True and str(confirm).lower() != 'true': + scope_desc = ( + f'pod {pod_namespace}/{pod_name}' if pod_name + else f'container PID {container_pid}' if container_pid + else 'host network namespace' + ) + return error_response(400, + f'tcpdump_capture requires explicit confirmation. This tool will: ' + f'(1) run tcpdump as root on instance {instance_id} targeting {scope_desc}, ' + f'(2) capture raw network packets for {duration}s on interface {interface}, ' + f'(3) upload pcap to S3. ' + f'Set confirmCapture=true to proceed.', + { + 'requiresConfirmation': True, + 'instanceId': instance_id, + 'scope': scope_desc, + 'durationSeconds': duration, + 'interface': interface, + 'filter': bpf_filter or 'none', + } + ) + + # Resolve and validate region + target_region, region_error = resolve_and_validate_region(arguments, instance_id) + if region_error: + return region_error + + # Validate instance belongs to an EKS cluster + instance_error = validate_eks_instance(instance_id, target_region) + if instance_error: + return instance_error + + try: + regional_ssm = get_regional_client('ssm', target_region) + except Exception as e: + return error_response(500, f'Failed to create SSM client for region {target_region}: {str(e)}') + + # Build the shell script that runs tcpdump and uploads to S3 + timestamp = datetime.utcnow().strftime('%Y%m%dT%H%M%SZ') + s3_prefix = f"tcpdump/{instance_id}/{timestamp}" + s3_key = f"{s3_prefix}/capture.pcap" + s3_key_txt = f"{s3_prefix}/capture_summary.txt" + s3_key_stats = f"{s3_prefix}/capture_stats.json" + s3_uri = f"s3://{LOGS_BUCKET}/{s3_key}" + s3_uri_txt = f"s3://{LOGS_BUCKET}/{s3_key_txt}" + s3_uri_stats = f"s3://{LOGS_BUCKET}/{s3_key_stats}" + + filter_clause = f' {bpf_filter}' if bpf_filter else '' + + # Determine nsenter prefix based on pod or PID + use_nsenter = bool(container_pid or pod_name) + ns_label = '' + if container_pid: + ns_label = f' (container PID {container_pid} namespace)' + elif pod_name: + ns_label = f' (pod {pod_namespace}/{pod_name} namespace)' + + script = f"""#!/bin/bash +set -euo pipefail + +PCAP_FILE="/tmp/tcpdump_capture_{timestamp}.pcap" +TXT_FILE="/tmp/tcpdump_summary_{timestamp}.txt" +STATS_FILE="/tmp/tcpdump_stats_{timestamp}.json" + +# Check if tcpdump is available — do NOT auto-install packages (T2 mitigation) +if ! command -v tcpdump &>/dev/null; then + echo "FATAL: tcpdump is not installed on this node. Install it manually or use an AMI that includes tcpdump." + echo "For Amazon Linux 2: yum install -y tcpdump" + echo "For Ubuntu: apt-get install -y tcpdump" + exit 1 +fi + +NSENTER_PREFIX="" +""" + + # Add pod PID discovery when podName is provided + if pod_name: + script += f""" +# === Resolve pod "{pod_namespace}/{pod_name}" to container PID === +echo "Resolving pod {pod_namespace}/{pod_name} to container PID..." +TARGET_PID="" + +# Ensure PATH includes common binary locations (SSM may have minimal PATH) +export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$PATH" + +# Set containerd endpoint for crictl (EKS standard) +export CONTAINER_RUNTIME_ENDPOINT="unix:///run/containerd/containerd.sock" +export CONTAINERD_ADDRESS="/run/containerd/containerd.sock" + +# Find crictl binary (may not be in default SSM PATH) +CRICTL="" +for p in /usr/local/bin/crictl /usr/bin/crictl /opt/bin/crictl $(which crictl 2>/dev/null); do + if [ -x "$p" ]; then CRICTL="$p"; break; fi +done + +# Method 1: crictl (containerd/CRI-O — standard on EKS AL2023 / 1.24+) +if [ -n "$CRICTL" ]; then + echo "Using crictl ($CRICTL) to find pod..." + # crictl pods --name does substring match, so filter precisely + POD_ID=$($CRICTL pods --namespace '{pod_namespace}' -q 2>/dev/null | while read pid; do + PNAME=$($CRICTL inspectp --output json "$pid" 2>/dev/null | python3 -c "import sys,json; print(json.load(sys.stdin).get('status',{{}}).get('metadata',{{}}).get('name',''))" 2>/dev/null || true) + if [ "$PNAME" = "{pod_name}" ]; then echo "$pid"; break; fi + done) + if [ -z "$POD_ID" ]; then + # Fallback: simple name match (works when pod name is unique enough) + POD_ID=$($CRICTL pods --name '{pod_name}' --namespace '{pod_namespace}' -q 2>/dev/null | head -1) + fi + if [ -n "$POD_ID" ]; then + echo "Found pod ID: $POD_ID" + CONTAINER_ID=$($CRICTL ps --pod "$POD_ID" -q 2>/dev/null | head -1) + if [ -n "$CONTAINER_ID" ]; then + echo "Found container ID: $CONTAINER_ID" + # Extract PID using JSON parsing — try multiple paths (containerd versions differ) + TARGET_PID=$($CRICTL inspect --output json "$CONTAINER_ID" 2>/dev/null | python3 -c " +import sys,json +d=json.load(sys.stdin) +# Try info.pid (containerd 1.x), then status.pid, then info.runtimeSpec.linux.namespaces +pid = d.get('info',{{}}).get('pid',0) +if not pid: + pid = d.get('status',{{}}).get('pid',0) +if not pid: + # Last resort: look for any 'pid' key recursively + def find_pid(obj): + if isinstance(obj, dict): + if 'pid' in obj and isinstance(obj['pid'], int) and obj['pid'] > 0: + return obj['pid'] + for v in obj.values(): + r = find_pid(v) + if r: return r + return 0 + pid = find_pid(d) +print(pid) +" 2>/dev/null || true) + echo "crictl: pod=$POD_ID container=$CONTAINER_ID pid=$TARGET_PID" + # Validate PID immediately; if invalid, try the pause (sandbox) container instead + if [ -n "$TARGET_PID" ] && [ "$TARGET_PID" != "0" ] && [ ! -e "/proc/$TARGET_PID/ns/net" ]; then + echo "WARNING: container PID $TARGET_PID has no /proc entry or ns/net — trying sandbox (pause) container..." + SANDBOX_PID=$($CRICTL inspectp --output json "$POD_ID" 2>/dev/null | python3 -c " +import sys,json +d=json.load(sys.stdin) +pid = d.get('info',{{}}).get('pid',0) +if not pid: + pid = d.get('status',{{}}).get('pid',0) +if not pid: + def find_pid(obj): + if isinstance(obj, dict): + if 'pid' in obj and isinstance(obj['pid'], int) and obj['pid'] > 0: + return obj['pid'] + for v in obj.values(): + r = find_pid(v) + if r: return r + return 0 + pid = find_pid(d) +print(pid) +" 2>/dev/null || true) + if [ -n "$SANDBOX_PID" ] && [ "$SANDBOX_PID" != "0" ] && [ -e "/proc/$SANDBOX_PID/ns/net" ]; then + echo "Using sandbox (pause) container PID $SANDBOX_PID instead" + TARGET_PID="$SANDBOX_PID" + else + echo "Sandbox PID $SANDBOX_PID also invalid" + TARGET_PID="" + fi + fi + else + echo "crictl: pod found but no running containers in pod $POD_ID" + fi + else + echo "crictl: no pod matching name='{pod_name}' namespace='{pod_namespace}'" + echo "Available pods on this node:" + $CRICTL pods 2>/dev/null | head -10 || true + fi +else + echo "crictl not found on this node" +fi + +# Method 2: ctr (containerd native CLI — usually present even when crictl is not) +if [ -z "$TARGET_PID" ] || [ "$TARGET_PID" = "0" ]; then + CTR="" + for p in /usr/local/bin/ctr /usr/bin/ctr $(which ctr 2>/dev/null); do + if [ -x "$p" ]; then CTR="$p"; break; fi + done + # Find containerd socket + CTR_ADDR="" + for sock in /run/containerd/containerd.sock /var/run/containerd/containerd.sock; do + if [ -S "$sock" ]; then CTR_ADDR="$sock"; break; fi + done + if [ -n "$CTR" ] && [ -n "$CTR_ADDR" ]; then + echo "Trying ctr ($CTR) with socket $CTR_ADDR to find pod container..." + CTR_CMD="$CTR --address $CTR_ADDR" + # containerd uses k8s.io namespace for Kubernetes containers + # ctr containers ls does NOT show pod names — must inspect each container's labels + # First find non-pause container, then fall back to pause (sandbox) container + SANDBOX_CID="" + for cid in $($CTR_CMD -n k8s.io containers ls -q 2>/dev/null); do + INFO=$($CTR_CMD -n k8s.io containers info "$cid" 2>/dev/null || true) + if echo "$INFO" | grep -q '"io.kubernetes.pod.name": "{pod_name}"'; then + if echo "$INFO" | grep -q '"io.kubernetes.pod.namespace": "{pod_namespace}"'; then + # Check if this is a pause/sandbox container + if echo "$INFO" | grep -q '"io.kubernetes.cri.container-type": "sandbox"'; then + SANDBOX_CID="$cid" + echo "ctr: found sandbox container $cid (saving as fallback)" + else + echo "ctr: found app container $cid" + # '|| true' guards the pipeline: under set -euo pipefail a + # no-match grep (e.g. container has no running task) would + # otherwise abort the whole script instead of falling + # through to the sandbox / cgroup-scan fallbacks. + CTR_PID=$($CTR_CMD -n k8s.io task ls 2>/dev/null | grep "$cid" | awk '{{print $2}}' || true) + if [ -n "$CTR_PID" ] && [ "$CTR_PID" != "0" ] && [ -e "/proc/$CTR_PID/ns/net" ]; then + TARGET_PID="$CTR_PID" + echo "ctr: resolved pid=$TARGET_PID" + break + fi + fi + fi + fi + done + # Fall back to sandbox (pause) container — shares the same network namespace + if ([ -z "$TARGET_PID" ] || [ "$TARGET_PID" = "0" ]) && [ -n "$SANDBOX_CID" ]; then + echo "ctr: using sandbox container $SANDBOX_CID" + CTR_PID=$($CTR_CMD -n k8s.io task ls 2>/dev/null | grep "$SANDBOX_CID" | awk '{{print $2}}' || true) + if [ -n "$CTR_PID" ] && [ "$CTR_PID" != "0" ] && [ -e "/proc/$CTR_PID/ns/net" ]; then + TARGET_PID="$CTR_PID" + echo "ctr: resolved pid=$TARGET_PID via sandbox" + fi + fi + [ -z "$TARGET_PID" ] && echo "ctr: could not resolve pod '{pod_name}' in namespace '{pod_namespace}'" + elif [ -n "$CTR" ]; then + echo "ctr found ($CTR) but no containerd socket found" + fi +fi + +# Method 3: docker (older EKS AMIs with dockershim) +if [ -z "$TARGET_PID" ] || [ "$TARGET_PID" = "0" ]; then + if command -v docker &>/dev/null; then + echo "Trying docker..." + DOCKER_ID=$(docker ps --filter "label=io.kubernetes.pod.name={pod_name}" --filter "label=io.kubernetes.pod.namespace={pod_namespace}" -q 2>/dev/null | head -1) + if [ -n "$DOCKER_ID" ]; then + # NOTE: parse JSON with python3 instead of a Go template. Literal + # double-curly-brace sequences in this script break the approval + # wrapper: SSM Automation re-resolves them in substituted document + # parameters and fails with ".State.Pid is not defined". + TARGET_PID=$(docker inspect "$DOCKER_ID" 2>/dev/null | python3 -c "import sys,json; print(json.load(sys.stdin)[0].get('State',{{}}).get('Pid',0))" 2>/dev/null || true) + echo "docker: container=$DOCKER_ID pid=$TARGET_PID" + fi + fi +fi + +# Method 4: search /proc cgroups for the pod name (works with containerd/CRI-O) +if [ -z "$TARGET_PID" ] || [ "$TARGET_PID" = "0" ]; then + echo "Trying /proc cgroup scan for pod name..." + # Container PIDs have cgroup entries containing the pod UID or pod name + for pid_dir in /proc/[0-9]*/cgroup; do + pid=$(echo "$pid_dir" | cut -d/ -f3) + if grep -q "{pod_name}" "$pid_dir" 2>/dev/null; then + # Verify it's a container process (not a host process) + if [ -e "/proc/$pid/ns/net" ] && [ "$(readlink /proc/$pid/ns/net)" != "$(readlink /proc/1/ns/net)" ]; then + TARGET_PID="$pid" + echo "cgroup scan: found pid=$TARGET_PID (cgroup matches pod name)" + break + fi + fi + done +fi + +# Method 5: fallback — search /proc for pause or main container process +if [ -z "$TARGET_PID" ] || [ "$TARGET_PID" = "0" ]; then + echo "Trying /proc process scan..." + # Look for any process whose network namespace differs from host and whose cgroup contains pod-related strings + for pid in $(ps -eo pid --no-headers 2>/dev/null | tr -d ' '); do + if [ -e "/proc/$pid/ns/net" ] && [ "$(readlink /proc/$pid/ns/net 2>/dev/null)" != "$(readlink /proc/1/ns/net 2>/dev/null)" ]; then + # Check if this PID's cmdline or environ references the pod + if grep -q "{pod_name}" /proc/$pid/cmdline 2>/dev/null || grep -q "{pod_name}" /proc/$pid/environ 2>/dev/null; then + TARGET_PID="$pid" + echo "proc scan: found pid=$TARGET_PID (cmdline/environ matches)" + break + fi + fi + done +fi + +if [ -z "$TARGET_PID" ] || [ "$TARGET_PID" = "0" ]; then + echo "FATAL: Could not resolve pod {pod_namespace}/{pod_name} to a container PID on this node." + echo "Ensure the pod is running on this specific worker node (instance {instance_id})." + echo "Use 'kubectl get pod -n {pod_namespace} {pod_name} -o wide' to verify the node." + echo "" + echo "Debug info:" + echo " crictl binary: ${{CRICTL:-not found}}" + echo " ctr binary: ${{CTR:-not found}}" + echo " containerd socket: ${{CTR_ADDR:-$(ls -la /run/containerd/containerd.sock 2>/dev/null || ls -la /var/run/containerd/containerd.sock 2>/dev/null || echo 'not found')}}" + echo " docker: $(which docker 2>/dev/null || echo 'not found')" + echo " Running containers:" + ${{CRICTL:-true}} ps 2>/dev/null | head -10 || ${{CTR:-true}} --address ${{CTR_ADDR:-/run/containerd/containerd.sock}} -n k8s.io task ls 2>/dev/null | head -10 || docker ps 2>/dev/null | head -10 || echo " (no container runtime accessible)" + exit 1 +fi + +echo "Resolved pod {pod_namespace}/{pod_name} -> PID $TARGET_PID" +# Validate PID: /proc//ns/net is a SYMLINK, not a directory — use -e (exists) not -d +if [ ! -e "/proc/$TARGET_PID/ns/net" ]; then + # Retry: PID might be a thread group leader; check if /proc/ exists at all + if [ ! -d "/proc/$TARGET_PID" ]; then + echo "FATAL: PID $TARGET_PID does not exist in /proc (process may have exited)" + else + echo "FATAL: PID $TARGET_PID exists but /proc/$TARGET_PID/ns/net is missing" + echo " /proc/$TARGET_PID/ns contents: $(ls -la /proc/$TARGET_PID/ns/ 2>/dev/null || echo 'cannot list')" + fi + exit 1 +fi +NSENTER_PREFIX="nsenter -n -t $TARGET_PID " +""" + elif container_pid: + script += f""" +# === Validate container PID {container_pid} === +if [ ! -e "/proc/{container_pid}/ns/net" ]; then + if [ ! -d "/proc/{container_pid}" ]; then + echo "FATAL: PID {container_pid} does not exist in /proc (process may have exited)" + else + echo "FATAL: PID {container_pid} exists but /proc/{container_pid}/ns/net is missing" + echo " /proc/{container_pid}/ns contents: $(ls -la /proc/{container_pid}/ns/ 2>/dev/null || echo 'cannot list')" + fi + exit 1 +fi +CONTAINER_COMM=$(cat /proc/{container_pid}/comm 2>/dev/null || echo "unknown") +echo "Targeting container process: PID {container_pid} ($CONTAINER_COMM)" +NSENTER_PREFIX="nsenter -n -t {container_pid} " +""" + + script += f""" +echo "Starting tcpdump{ns_label} on interface '{interface}' for {duration}s..." +echo "Filter: '{bpf_filter or 'none'}'" +echo "Output: $PCAP_FILE" + +# Run tcpdump with timeout (with optional nsenter) +timeout {duration} ${{NSENTER_PREFIX}}tcpdump -i {interface} -w "$PCAP_FILE" -c 100000{filter_clause} 2>&1 || true + +# Verify capture file exists and has data +if [ ! -f "$PCAP_FILE" ]; then + echo "FATAL: Capture file not created" + exit 1 +fi + +FILE_SIZE=$(stat -c%s "$PCAP_FILE" 2>/dev/null || stat -f%z "$PCAP_FILE" 2>/dev/null || echo "0") +echo "Capture complete. File size: $FILE_SIZE bytes" + +if [ "$FILE_SIZE" -eq 0 ]; then + echo "WARNING: Capture file is empty — no packets matched the filter" +fi + +# Decode pcap to human-readable text summary (first 5000 packets max) +echo "Decoding pcap to text summary..." +# Use -c 5000 instead of piping through head to avoid SIGPIPE under set -euo pipefail +tcpdump -nn -r "$PCAP_FILE" -c 5000 > "$TXT_FILE" 2>/dev/null || true +TXT_SIZE=$(stat -c%s "$TXT_FILE" 2>/dev/null || stat -f%z "$TXT_FILE" 2>/dev/null || echo "0") +PACKET_COUNT=$(wc -l < "$TXT_FILE" 2>/dev/null || echo "0") +echo "Decoded $PACKET_COUNT packets to text (txt_size=$TXT_SIZE)" + +# If decode produced empty output, log diagnostics and retry +if [ "$TXT_SIZE" -eq 0 ] || [ "$PACKET_COUNT" -eq 0 ]; then + echo "WARNING: Text decode produced empty output." + echo "Pcap file details:" + ls -la "$PCAP_FILE" 2>/dev/null || true + file "$PCAP_FILE" 2>/dev/null || true + # Retry with verbose stderr to diagnose + echo "Retry with stderr:" + tcpdump -nn -r "$PCAP_FILE" -c 10 2>&1 || true +fi + +# Generate stats JSON with protocol breakdown and top talkers (using Python for valid JSON) +echo "Generating capture statistics..." +if command -v python3 &>/dev/null; then + python3 - "$PCAP_FILE" "$STATS_FILE" << 'PYSTATS' +import subprocess, json, sys, re +from collections import Counter + +pcap, out = sys.argv[1], sys.argv[2] + +def tcpdump_count(extra_args=None): + cmd = ['tcpdump', '-nn', '-r', pcap] + (extra_args or []) + try: + r = subprocess.run(cmd, capture_output=True, text=True, timeout=30) + return len([l for l in r.stdout.strip().splitlines() if l]) + except Exception: + return 0 + +def tcpdump_lines(): + try: + r = subprocess.run(['tcpdump', '-nn', '-r', pcap], capture_output=True, text=True, timeout=60) + return [l for l in r.stdout.strip().splitlines() if l] + except Exception: + return [] + +lines = tcpdump_lines() +total = len(lines) + +# Extract IPs from tcpdump output: field 3 = src (IP.port), field 5 = dst (IP.port:) +ip_port_re = re.compile(r'^(\\d+\\.\\d+\\.\\d+\\.\\d+)\\.\\d+$') +src_counter, dst_counter = Counter(), Counter() +for line in lines: + parts = line.split() + if len(parts) >= 5: + # Source: field index 2 (0-based), format "IP.port" + m = ip_port_re.match(parts[2]) + if m: + src_counter[m.group(1)] += 1 + # Destination: field index 4, format "IP.port:" (trailing colon) + dst_raw = parts[4].rstrip(':') + m = ip_port_re.match(dst_raw) + if m: + dst_counter[m.group(1)] += 1 + +retrans = sum(1 for l in lines if 'retransmit' in l.lower() or 'retrans' in l.lower()) + +stats = {{ + "totalPackets": total, + "protocols": {{ + "tcp": tcpdump_count(['tcp']), + "udp": tcpdump_count(['udp']), + "icmp": tcpdump_count(['icmp']), + "arp": tcpdump_count(['arp']), + }}, + "ports": {{ + "dns_53": tcpdump_count(['port', '53']), + "http_80": tcpdump_count(['port', '80']), + "https_443": tcpdump_count(['port', '443']), + }}, + "tcpFlags": {{ + "syn": tcpdump_count(['tcp[tcpflags] & (tcp-syn) != 0']), + "rst": tcpdump_count(['tcp[tcpflags] & (tcp-rst) != 0']), + }}, + "possibleRetransmits": retrans, + "topSourceIPs": dict(src_counter.most_common(10)), + "topDestinationIPs": dict(dst_counter.most_common(10)), +}} + +with open(out, 'w') as f: + json.dump(stats, f, indent=2) +print(f"Stats generated: {{total}} packets") +PYSTATS +else + # Fallback: minimal stats without top talkers if python3 not available + TOTAL=$(tcpdump -nn -r "$PCAP_FILE" 2>/dev/null | wc -l) + TCP_COUNT=$(tcpdump -nn -r "$PCAP_FILE" tcp 2>/dev/null | wc -l) + UDP_COUNT=$(tcpdump -nn -r "$PCAP_FILE" udp 2>/dev/null | wc -l) + ICMP_COUNT=$(tcpdump -nn -r "$PCAP_FILE" icmp 2>/dev/null | wc -l) + echo '{{"totalPackets":'$TOTAL',"protocols":{{"tcp":'$TCP_COUNT',"udp":'$UDP_COUNT',"icmp":'$ICMP_COUNT'}},"topSourceIPs":{{}},"topDestinationIPs":{{}}}}' > "$STATS_FILE" +fi +if [ ! -f "$STATS_FILE" ] || [ ! -s "$STATS_FILE" ]; then + echo '{{"error":"stats generation failed"}}' > "$STATS_FILE" +fi + +# Upload all artifacts to S3 (non-fatal — node may lack S3 permissions) +# IMPORTANT: disable set -e for uploads — these are best-effort and must not kill the script +set +e +UPLOAD_FAILURES=0 + +echo "Uploading pcap to {s3_uri}..." +aws s3 cp "$PCAP_FILE" "{s3_uri}" --no-progress 2>&1 +if [ $? -eq 0 ]; then + echo "UPLOAD_PCAP=ok" +else + echo "WARNING: Failed to upload pcap to S3 (node IAM role may lack s3:PutObject permission)" + echo "UPLOAD_PCAP=failed" + UPLOAD_FAILURES=$((UPLOAD_FAILURES + 1)) +fi + +echo "Uploading text summary to {s3_uri_txt}..." +aws s3 cp "$TXT_FILE" "{s3_uri_txt}" --quiet 2>&1 +if [ $? -eq 0 ]; then + echo "UPLOAD_TXT=ok" +else + echo "WARNING: Failed to upload text summary to S3" + echo "UPLOAD_TXT=failed" + UPLOAD_FAILURES=$((UPLOAD_FAILURES + 1)) +fi + +echo "Uploading stats to {s3_uri_stats}..." +aws s3 cp "$STATS_FILE" "{s3_uri_stats}" --quiet 2>&1 +if [ $? -eq 0 ]; then + echo "UPLOAD_STATS=ok" +else + echo "WARNING: Failed to upload stats to S3" + echo "UPLOAD_STATS=failed" + UPLOAD_FAILURES=$((UPLOAD_FAILURES + 1)) +fi + +# Do NOT re-enable set -e — the inline output section and cleanup must not kill the script +# set -e is intentionally left off for the remainder + +if [ "$UPLOAD_FAILURES" -gt 0 ]; then + echo "WARNING: $UPLOAD_FAILURES of 3 uploads failed. Ensure the node IAM role has s3:PutObject permission to {LOGS_BUCKET}." + echo "See README — 'S3 Upload Permissions for Worker Nodes' section." +fi + +echo "S3_KEY={s3_key}" +echo "S3_KEY_TXT={s3_key_txt}" +echo "S3_KEY_STATS={s3_key_stats}" +echo "FILE_SIZE=$FILE_SIZE" +echo "PACKET_COUNT=$PACKET_COUNT" +echo "UPLOAD_FAILURES=$UPLOAD_FAILURES" + +# Inline the decoded text and stats in stdout so Lambda can parse them even if S3 upload failed +echo "===INLINE_STATS_BEGIN===" +cat "$STATS_FILE" 2>/dev/null || echo '{{"error":"stats file missing"}}' +echo "" +echo "===INLINE_STATS_END===" +echo "===INLINE_TXT_BEGIN===" +head -500 "$TXT_FILE" 2>/dev/null || echo "(no decoded text)" +echo "" +echo "===INLINE_TXT_END===" + +# Cleanup +rm -f "$PCAP_FILE" "$TXT_FILE" "$STATS_FILE" 2>/dev/null || true +echo "DONE" +exit 0 +""" + + capture_metadata = { + 'instanceId': instance_id, + 'region': target_region, + 's3Key': s3_key, + 's3KeyTxt': s3_key_txt, + 's3KeyStats': s3_key_stats, + 's3Prefix': s3_prefix, + 'durationSeconds': duration, + 'interface': interface, + 'filter': bpf_filter, + 'containerPid': container_pid or None, + 'podName': pod_name or None, + 'podNamespace': pod_namespace if pod_name else None, + 'startedAt': timestamp, + 'captureScope': 'podNamespace' if (pod_name or container_pid) else 'hostNamespace', + 'nsenterUsed': bool(pod_name or container_pid), + 'networkNamespace': ( + f'pod/{pod_namespace}/{pod_name}' if pod_name + else f'container/PID-{container_pid}' if container_pid + else 'host' + ), + } + + # Human-in-the-loop approval gate (M3). Packet capture is the most invasive + # tool this server exposes — when approval is required (the default), the + # capture runs through a wrapper SSM Automation document whose FIRST step + # is the native aws:approve action. The execution pauses inside SSM until a + # designated human approves it in the Systems Manager console; only then + # does the runTcpdump step send this script to the node. The direct + # send_command path below exists only for REQUIRE_COLLECTION_APPROVAL=false + # (supervised/test deployments). + if REQUIRE_COLLECTION_APPROVAL: + precondition_error = enforce_tcpdump_approval_preconditions(target_region) + if precondition_error is not None: + return precondition_error + return _start_tcpdump_with_approval( + regional_ssm, instance_id, target_region, script, duration, + ns_label, capture_metadata, arguments, + ) + + try: + response = regional_ssm.send_command( + InstanceIds=[instance_id], + DocumentName='AWS-RunShellScript', + Parameters={ + 'commands': [script], + 'executionTimeout': [str(duration + 120)], # extra buffer for install + upload + }, + TimeoutSeconds=duration + 180, + Comment=f'tcpdump capture for {instance_id} ({duration}s)', + ) + + cmd_id = response['Command']['CommandId'] + + # Store capture metadata for status polling and tcpdump_analyze + try: + s3_client.put_object( + Bucket=LOGS_BUCKET, + Key=f"tcpdump-commands/{cmd_id}.json", + Body=json.dumps({**capture_metadata, 'commandId': cmd_id}), + ) + except Exception: + pass # Non-fatal + + return success_response({ + 'message': f'tcpdump capture started ({duration}s){ns_label}', + 'commandId': cmd_id, + 'instanceId': instance_id, + 'region': target_region, + 'durationSeconds': duration, + 'interface': interface, + 'filter': bpf_filter or 'none', + 'containerPid': container_pid or None, + 'podName': pod_name or None, + 'podNamespace': pod_namespace if pod_name else None, + 'captureScope': 'podNamespace' if (pod_name or container_pid) else 'hostNamespace', + 'nsenterUsed': bool(pod_name or container_pid), + 'networkNamespace': ( + f'pod/{pod_namespace}/{pod_name}' if pod_name + else f'container/PID-{container_pid}' if container_pid + else 'host' + ), + 's3Key': s3_key, + 's3KeyTxt': s3_key_txt, + 's3KeyStats': s3_key_stats, + 's3Bucket': LOGS_BUCKET, + 'estimatedCompletionSeconds': duration + 30, + 'nextStep': f'Poll with tcpdump_capture(commandId="{cmd_id}", instanceId="{instance_id}") after ~{duration + 30}s. Once complete, use tcpdump_analyze(instanceId="{instance_id}", commandId="{cmd_id}") to read the decoded packet summary.', + 'task': { + 'taskId': cmd_id, + 'state': 'running', + 'message': f'tcpdump running for {duration}s on {interface}', + 'progress': 0, + }, + }) + + except Exception as e: + return error_response(500, f'Failed to start tcpdump: {str(e)}') + + +def _poll_tcpdump_status(command_id: str, instance_id: str, arguments: Dict) -> Dict: + """Poll the status of a tcpdump SSM Run Command.""" + + # Try to load stored metadata + metadata = {} + try: + meta_resp = s3_client.get_object( + Bucket=LOGS_BUCKET, + Key=f"tcpdump-commands/{command_id}.json", + ) + metadata = json.loads(meta_resp['Body'].read().decode('utf-8')) + except Exception: + pass + + target_region = metadata.get('region') or resolve_region(arguments, instance_id) + + try: + regional_ssm = get_regional_client('ssm', target_region) + result = regional_ssm.get_command_invocation( + CommandId=command_id, + InstanceId=instance_id, + ) + + status = result.get('Status', 'Unknown') + stdout = result.get('StandardOutputContent', '') + stderr = result.get('StandardErrorContent', '') + + # Parse output for S3 key and file size + s3_key = metadata.get('s3Key', '') + s3_key_txt = metadata.get('s3KeyTxt', '') + s3_key_stats = metadata.get('s3KeyStats', '') + file_size = 0 + packet_count = 0 + for line in stdout.split('\n'): + if line.startswith('S3_KEY='): + s3_key = line.split('=', 1)[1].strip() + if line.startswith('S3_KEY_TXT='): + s3_key_txt = line.split('=', 1)[1].strip() + if line.startswith('S3_KEY_STATS='): + s3_key_stats = line.split('=', 1)[1].strip() + if line.startswith('FILE_SIZE='): + try: + file_size = int(line.split('=', 1)[1].strip()) + except ValueError: + pass + if line.startswith('PACKET_COUNT='): + try: + packet_count = int(line.split('=', 1)[1].strip()) + except ValueError: + pass + + # Check if capture completed but S3 upload failed (script has inline data) + # Use multiple markers for robustness — SSM truncates StandardOutputContent at 24KB + # so early markers like "Capture complete." may be cut if inline stats/text are large. + # + # IMPORTANT: These markers are ONLY printed AFTER the capture succeeds. + # Genuine failures (tcpdump not found, pod PID not resolved, no capture file) + # exit with "FATAL:" before any of these markers are emitted, so + # capture_completed will correctly be False for real failures. + capture_completed = ( + 'Capture complete.' in stdout + or 'DONE' in stdout + or 'UPLOAD_PCAP=ok' in stdout + or 'UPLOAD_PCAP=failed' in stdout + or ('FILE_SIZE=' in stdout and 'S3_KEY=' in stdout) + ) + # Double-check: if stdout contains FATAL, the capture itself failed — never treat as success + if 'FATAL:' in stdout: + capture_completed = False + upload_failures = 0 + for line in stdout.split('\n'): + if line.startswith('UPLOAD_FAILURES='): + try: + upload_failures = int(line.split('=', 1)[1].strip()) + except ValueError: + pass + + # Extract inline stats and text from stdout (available even when S3 upload fails) + inline_stats = {} + inline_txt_lines = [] + if '===INLINE_STATS_BEGIN===' in stdout: + try: + stats_block = stdout.split('===INLINE_STATS_BEGIN===')[1].split('===INLINE_STATS_END===')[0].strip() + if stats_block: + inline_stats = json.loads(stats_block) + except (IndexError, json.JSONDecodeError): + pass + if '===INLINE_TXT_BEGIN===' in stdout: + try: + txt_block = stdout.split('===INLINE_TXT_BEGIN===')[1].split('===INLINE_TXT_END===')[0].strip() + if txt_block: + inline_txt_lines = txt_block.split('\n') + except IndexError: + pass + + # If capture completed (even if S3 upload failed), treat as success with warnings + if status in ('Success',) or (capture_completed and status == 'Failed'): + # Generate presigned URL for download (may fail if pcap wasn't uploaded). + # pcap captures use a tighter expiration than ordinary log artifacts + # because they may contain credentials in transit. + presigned_url = '' + try: + presigned_url = s3_client.generate_presigned_url( + 'get_object', + Params={'Bucket': LOGS_BUCKET, 'Key': s3_key}, + ExpiresIn=PCAP_PRESIGNED_URL_EXPIRATION, + ) + except Exception: + pass + + # If S3 uploads failed, store inline data to S3 from Lambda (Lambda has S3 permissions) + if (upload_failures > 0 or (status == 'Failed' and capture_completed)) and inline_stats: + try: + s3_client.put_object( + Bucket=LOGS_BUCKET, + Key=s3_key_stats, + Body=json.dumps(inline_stats, indent=2), + ContentType='application/json', + ) + except Exception: + pass + if (upload_failures > 0 or (status == 'Failed' and capture_completed)) and inline_txt_lines: + try: + s3_client.put_object( + Bucket=LOGS_BUCKET, + Key=s3_key_txt, + Body='\n'.join(inline_txt_lines), + ContentType='text/plain', + ) + except Exception: + pass + + warnings = [] + actual_failures = 0 + if upload_failures > 0 or (status == 'Failed' and capture_completed): + actual_failures = upload_failures if upload_failures > 0 else 3 # assume all failed if script died during upload + warnings.append(f'{actual_failures} of 3 S3 uploads failed from the node (node IAM role may lack s3:PutObject). Stats and text summary were recovered from stdout and uploaded by Lambda.') + if actual_failures == 3: + warnings.append('pcap file was NOT uploaded — it was too large to inline in stdout. Add S3 PutObject permission to the node IAM role to capture pcap files.') + + # Surface a warning when the pcap exceeds the configured upload cap. + pcap_oversized = False + if file_size and MAX_PCAP_BYTES and file_size > MAX_PCAP_BYTES: + pcap_oversized = True + warnings.append( + f'pcap exceeds MAX_PCAP_BYTES ({format_bytes(MAX_PCAP_BYTES)}); ' + f'capture is {format_bytes(file_size)}. Consider shorter durationSeconds ' + f'or a tighter BPF filter on future captures.' + ) + + response_data = { + 'commandId': command_id, + 'instanceId': instance_id, + 'status': 'completed' if not warnings else 'completed_with_warnings', + 's3Key': s3_key, + 's3KeyTxt': s3_key_txt, + 's3KeyStats': s3_key_stats, + 's3Bucket': LOGS_BUCKET, + 'fileSizeBytes': file_size, + 'fileSizeHuman': format_bytes(file_size), + 'packetCount': packet_count, + 'pcapOversized': pcap_oversized, + 'pcapMaxBytes': MAX_PCAP_BYTES, + 'presignedUrl': presigned_url, + 'presignedUrlExpiresIn': f'{PCAP_PRESIGNED_URL_EXPIRATION} seconds', + 'output': stdout[-2000:] if len(stdout) > 2000 else stdout, + 'nextStep': f'Use tcpdump_analyze(instanceId="{instance_id}", commandId="{command_id}") to read decoded packet data and statistics.', + 'task': { + 'taskId': command_id, + 'state': 'completed', + 'message': f'tcpdump capture completed' + (f' ({actual_failures} S3 uploads failed — recovered via Lambda)' if warnings else f' — uploaded to s3://{LOGS_BUCKET}/{s3_key}'), + 'progress': 100, + }, + } + if warnings: + response_data['warnings'] = warnings + if inline_stats: + response_data['inlineStats'] = inline_stats + + return success_response(response_data) + + elif status in ('InProgress', 'Pending', 'Delayed'): + elapsed = 0 + duration = metadata.get('durationSeconds', 120) + if metadata.get('startedAt'): + try: + start_dt = datetime.strptime(metadata['startedAt'], '%Y%m%dT%H%M%SZ') + elapsed = (datetime.utcnow() - start_dt).total_seconds() + except Exception: + pass + progress = min(95, int((elapsed / (duration + 30)) * 100)) if duration else 0 + + return success_response({ + 'commandId': command_id, + 'instanceId': instance_id, + 'status': 'in_progress', + 'elapsedSeconds': int(elapsed), + 'durationSeconds': duration, + 'nextStep': f'Poll again in 15-30 seconds', + 'task': { + 'taskId': command_id, + 'state': 'running', + 'message': f'tcpdump capture in progress ({int(elapsed)}s / {duration}s)', + 'progress': progress, + }, + }) + + else: + # Failed / TimedOut / Cancelled + return error_response(500, f'tcpdump command {status}', { + 'commandId': command_id, + 'status': status, + 'stdout': stdout[-2000:] if stdout else '', + 'stderr': stderr[-2000:] if stderr else '', + 'statusDetails': result.get('StatusDetails', ''), + 'task': { + 'taskId': command_id, + 'state': 'failed', + 'message': f'tcpdump command {status}: {stderr[:200] if stderr else "unknown error"}', + 'progress': 0, + }, + }) + + except Exception as e: + return error_response(500, f'Failed to poll tcpdump status: {str(e)}') + + +def _analyze_dns_packets(lines: list) -> Dict: + """Analyze DNS queries in decoded tcpdump lines. + Detects ndots search-domain expansion (normal), NXDomain storms, and real failures.""" + dns_re = re.compile( + r'>\s+\S+\.53:\s+\d+\+?\s+(A|AAAA|CNAME|MX|SRV|PTR|TXT|SOA|NS)\?\s+(\S+)', + re.IGNORECASE, + ) + nxdomain_re = re.compile(r'NXDomain', re.IGNORECASE) + k8s_svc_suffix = re.compile( + r'\.svc\.cluster\.local\..*\.svc\.cluster\.local', + re.IGNORECASE, + ) + + queries = [] + nxdomain_count = 0 + ndots_expansion_queries = [] + total_dns = 0 + + for line in lines: + m = dns_re.search(line) + if m: + total_dns += 1 + qtype = m.group(1).upper() + qname = m.group(2).rstrip('.') + queries.append({'type': qtype, 'name': qname}) + if k8s_svc_suffix.search(qname): + ndots_expansion_queries.append(qname) + if nxdomain_re.search(line): + nxdomain_count += 1 + + if total_dns == 0: + return {} + + result: Dict = { + 'totalDnsQueries': total_dns, + 'nxdomainResponses': nxdomain_count, + 'anomalies': [], + } + + if ndots_expansion_queries: + unique = list(set(ndots_expansion_queries))[:10] + result['ndotsSearchDomainExpansion'] = { + 'count': len(ndots_expansion_queries), + 'isNormalBehavior': True, + 'explanation': ( + 'Queries with doubled Kubernetes suffixes (e.g., ' + 'name.ns.svc.cluster.local.ns.svc.cluster.local) are NORMAL. ' + 'With the default ndots:5, the glibc resolver appends each ' + '/etc/resolv.conf search domain to names with fewer than 5 dots ' + 'before trying the name as-is. These NXDomain responses are ' + 'expected and harmless. To reduce them: use a trailing dot on ' + 'FQDNs, lower ndots to 2, or use short service names.' + ), + 'exampleQueries': unique, + } + result['anomalies'].append({ + 'type': 'ndots_search_expansion', + 'severity': 'info', + 'message': ( + f'{len(ndots_expansion_queries)} DNS queries show ndots:5 search-domain ' + f'expansion (doubled .svc.cluster.local suffixes). This is NORMAL ' + f'Kubernetes DNS behavior, not a misconfiguration. The queries get ' + f'NXDomain but the correct resolution succeeds afterward.' + ), + }) + elif nxdomain_count > total_dns * 0.5 and total_dns > 10: + result['anomalies'].append({ + 'type': 'high_nxdomain_rate', + 'severity': 'warning', + 'message': ( + f'{nxdomain_count} NXDomain responses out of {total_dns} DNS queries ' + f'({(nxdomain_count/total_dns)*100:.0f}%) — possible DNS misconfiguration ' + f'or queries for non-existent services.' + ), + }) + + return result + + +def _analyze_tcp_rst_patterns(lines: list) -> Dict: + """Detect TCP RST patterns that are normal in Kubernetes. + Health probes (liveness/readiness) open a TCP connection and immediately close it, + producing RST packets. kube-proxy DNAT race during pod termination also causes RSTs. + These are expected and should not be flagged as connection failures. + Ref: https://docs.aws.amazon.com/prescriptive-guidance/latest/ha-resiliency-amazon-eks-apps/probes-checks.html""" + rst_re = re.compile(r'Flags\s+\[R\.?\]|Flags\s+\[R\]', re.IGNORECASE) + syn_re = re.compile(r'Flags\s+\[S\]', re.IGNORECASE) + fin_re = re.compile(r'Flags\s+\[F\.?\]', re.IGNORECASE) + # Health probe pattern: SYN then RST within a few packets to same port, short-lived + # Detect port 10250 (kubelet), 10256 (kube-proxy health), 15021 (istio), common probe ports + probe_port_re = re.compile(r'\.\s*(10250|10256|10257|10259|15021|8080|8443|80|443)\s*[>:]') + + total_rst = 0 + probe_rst = 0 + short_lived_rst = 0 + rst_lines_sample = [] + + # Track connections: (src, dst, port) -> packet count + connections: Dict[str, int] = {} + + for line in lines: + if rst_re.search(line): + total_rst += 1 + if probe_port_re.search(line): + probe_rst += 1 + if len(rst_lines_sample) < 5: + rst_lines_sample.append(line.strip()[:200]) + + if total_rst == 0: + return {} + + result: Dict = {'totalRstPackets': total_rst, 'anomalies': []} + + if probe_rst > 0: + result['healthProbeRsts'] = { + 'count': probe_rst, + 'isNormalBehavior': True, + 'explanation': ( + 'TCP RST packets to kubelet (10250), kube-proxy health (10256), ' + 'or common HTTP ports after very short connections are NORMAL. ' + 'Kubernetes liveness/readiness probes open a TCP connection to verify ' + 'the port is listening, then close it immediately — producing a RST. ' + 'This is expected probe behavior, not a connection failure.' + ), + } + result['anomalies'].append({ + 'type': 'health_probe_rst', + 'severity': 'info', + 'isNormalBehavior': True, + 'message': ( + f'{probe_rst} TCP RST packets detected on health-check ports ' + f'(10250/10256/8080/etc). This is NORMAL Kubernetes health probe ' + f'behavior — probes open and immediately close TCP connections.' + ), + }) + + return result + + +def _analyze_kube_proxy_dnat(lines: list) -> Dict: + """Detect kube-proxy DNAT patterns where both ClusterIP and PodIP appear for the same flow. + When a pod connects to a ClusterIP Service, kube-proxy/iptables performs DNAT to rewrite + the destination to a backend PodIP. In tcpdump on the node, you see BOTH the original + ClusterIP destination AND the DNATed PodIP — this looks like duplicate or phantom traffic + but is completely normal kube-proxy behavior. + Ref: https://docs.aws.amazon.com/eks/latest/best-practices/hybrid-nodes-app-network-traffic.html""" + # Detect 10.x.x.x (typical ClusterIP range) and pod IPs in same capture + cluster_ip_re = re.compile(r'\b(10\.\d{1,3}\.\d{1,3}\.\d{1,3})\.\d+\b') + # Look for the same source talking to two different IPs on the same port + flow_re = re.compile( + r'(\d+\.\d+\.\d+\.\d+)\.(\d+)\s+>\s+(\d+\.\d+\.\d+\.\d+)\.(\d+)' + ) + + src_dst_pairs: Dict[str, set] = {} # src.port -> set of dst IPs + for line in lines: + m = flow_re.search(line) + if m: + src_ip, src_port, dst_ip, dst_port = m.groups() + key = f"{src_ip}:{src_port}->{dst_port}" + if key not in src_dst_pairs: + src_dst_pairs[key] = set() + src_dst_pairs[key].add(dst_ip) + + # Find flows where same src:port->dstPort talks to multiple dst IPs (DNAT indicator) + dnat_flows = {k: v for k, v in src_dst_pairs.items() if len(v) > 1} + + if not dnat_flows: + return {} + + examples = [] + for key, dsts in list(dnat_flows.items())[:5]: + examples.append(f"{key} -> {', '.join(sorted(dsts))}") + + return { + 'dnatFlowCount': len(dnat_flows), + 'isNormalBehavior': True, + 'explanation': ( + 'Flows where the same source connects to multiple destination IPs on the ' + 'same port are typically kube-proxy DNAT in action. When a pod connects to ' + 'a ClusterIP Service, iptables rewrites the destination to a backend pod IP. ' + 'tcpdump on the node captures BOTH the pre-DNAT (ClusterIP) and post-DNAT ' + '(PodIP) packets, making it look like duplicate traffic. This is normal.' + ), + 'exampleFlows': examples, + 'anomalies': [{ + 'type': 'kube_proxy_dnat', + 'severity': 'info', + 'isNormalBehavior': True, + 'message': ( + f'{len(dnat_flows)} flows show the same source connecting to multiple ' + f'destination IPs on the same port. This is likely kube-proxy DNAT — ' + f'ClusterIP being rewritten to backend PodIP. NORMAL behavior.' + ), + }], + } + + +def _analyze_vpc_cni_snat(lines: list) -> Dict: + """Detect VPC CNI SNAT where pod IP is translated to node IP for external traffic. + By default, VPC CNI SNATs pod traffic destined outside the VPC — the source IP changes + from the pod's IP to the node's primary ENI IP. In tcpdump you see outbound packets + with the node IP as source instead of the pod IP. This is expected. + Ref: https://docs.aws.amazon.com/eks/latest/userguide/external-snat.html""" + # Detect traffic to external IPs (non-RFC1918, non-cluster) + flow_re = re.compile( + r'(\d+\.\d+\.\d+\.\d+)\.(\d+)\s+>\s+(\d+\.\d+\.\d+\.\d+)\.(\d+)' + ) + private_re = re.compile(r'^(10\.|172\.(1[6-9]|2\d|3[01])\.|192\.168\.)') + + external_flows = 0 + for line in lines: + m = flow_re.search(line) + if m: + dst_ip = m.group(3) + if not private_re.match(dst_ip) and not dst_ip.startswith('127.'): + external_flows += 1 + + if external_flows < 5: + return {} + + return { + 'externalTrafficFlows': external_flows, + 'isNormalBehavior': True, + 'explanation': ( + 'Traffic to external (non-RFC1918) IPs undergoes SNAT by the VPC CNI plugin. ' + 'The pod source IP is translated to the node primary ENI IP before leaving ' + 'the VPC. In tcpdump on the node, outbound external packets show the node IP ' + 'as source, not the pod IP. This is default VPC CNI behavior ' + '(AWS_VPC_K8S_CNI_EXTERNALSNAT=false). Inbound responses are reverse-NATed ' + 'back to the pod IP.' + ), + 'anomalies': [{ + 'type': 'vpc_cni_snat', + 'severity': 'info', + 'isNormalBehavior': True, + 'message': ( + f'{external_flows} packets to external IPs detected. Source IP translation ' + f'(SNAT) from pod IP to node IP is NORMAL VPC CNI behavior for traffic ' + f'leaving the VPC.' + ), + }], + } + + +def _analyze_tcp_keepalives(lines: list) -> Dict: + """Detect TCP keepalive packets on idle connections. + Long-lived connections (e.g., gRPC, database pools, websockets) send periodic TCP + keepalive probes to prevent idle timeout by NAT gateways (350s), NLBs, or conntrack. + These appear as small packets with ack flag on established connections. Normal behavior. + Ref: https://aws.amazon.com/blogs/networking-and-content-delivery/implementing-long-running-tcp-connections-within-vpc-networking/""" + # Keepalives are typically: small ack-only packets, often with length 0 + keepalive_re = re.compile(r'Flags\s+\[\.?\].*length\s+0', re.IGNORECASE) + total_keepalive_candidates = 0 + + for line in lines: + if keepalive_re.search(line) and 'ack' in line.lower(): + total_keepalive_candidates += 1 + + if total_keepalive_candidates < 10: + return {} + + return { + 'keepaliveCandidates': total_keepalive_candidates, + 'isNormalBehavior': True, + 'explanation': ( + 'Zero-length ACK packets on established connections are typically TCP keepalive ' + 'probes. Applications and kernels send these to prevent idle connection timeout ' + 'by NAT Gateway (350s idle timeout), NLB, or conntrack table eviction. This is ' + 'expected for long-lived connections like gRPC streams, database connection pools, ' + 'and websockets.' + ), + 'anomalies': [{ + 'type': 'tcp_keepalive', + 'severity': 'info', + 'isNormalBehavior': True, + 'message': ( + f'{total_keepalive_candidates} zero-length ACK packets detected (likely TCP ' + f'keepalives). This is NORMAL for long-lived connections preventing idle ' + f'timeout by NAT Gateway/NLB/conntrack.' + ), + }], + } + + +def _analyze_icmp_expected(lines: list) -> Dict: + """Detect expected ICMP patterns in Kubernetes. + ICMP port-unreachable during rolling updates (old pod IP, new pod not yet ready), + ICMP fragmentation-needed for PMTUD, and ICMP redirect from VPC routing are all normal. + Ref: https://docs.aws.amazon.com/eks/latest/best-practices/vpc-cni.html""" + icmp_unreach_re = re.compile(r'ICMP.*unreachable', re.IGNORECASE) + icmp_frag_re = re.compile(r'ICMP.*frag.*needed|ICMP.*too\s+big', re.IGNORECASE) + icmp_redirect_re = re.compile(r'ICMP.*redirect', re.IGNORECASE) + + unreachable = 0 + frag_needed = 0 + redirect = 0 + + for line in lines: + if icmp_unreach_re.search(line): + unreachable += 1 + if icmp_frag_re.search(line): + frag_needed += 1 + if icmp_redirect_re.search(line): + redirect += 1 + + total = unreachable + frag_needed + redirect + if total == 0: + return {} + + result: Dict = {'anomalies': []} + + if frag_needed > 0: + result['pmtudFragNeeded'] = { + 'count': frag_needed, + 'isNormalBehavior': True, + 'explanation': ( + 'ICMP "fragmentation needed" (type 3 code 4) or "packet too big" messages ' + 'are part of Path MTU Discovery (PMTUD). This is the network telling the ' + 'sender to reduce packet size. Normal for VPC traffic crossing different MTU ' + 'boundaries (e.g., 9001 jumbo frames to 1500 standard).' + ), + } + result['anomalies'].append({ + 'type': 'pmtud_frag_needed', + 'severity': 'info', + 'isNormalBehavior': True, + 'message': ( + f'{frag_needed} ICMP fragmentation-needed packets detected. This is NORMAL ' + f'Path MTU Discovery behavior.' + ), + }) + + if unreachable > 0 and unreachable < 20: + result['icmpUnreachable'] = { + 'count': unreachable, + 'isNormalBehavior': True, + 'explanation': ( + 'A small number of ICMP port/host unreachable messages is normal during ' + 'rolling updates, pod termination, or when UDP services are briefly ' + 'unavailable. The VPC CNI 30-second IP cooldown cache means old pod IPs ' + 'may receive traffic briefly after pod deletion.' + ), + } + result['anomalies'].append({ + 'type': 'icmp_unreachable_transient', + 'severity': 'info', + 'isNormalBehavior': True, + 'message': ( + f'{unreachable} ICMP unreachable packets detected. Small numbers are NORMAL ' + f'during rolling updates or pod termination (VPC CNI 30s IP cooldown).' + ), + }) + elif unreachable >= 20: + result['anomalies'].append({ + 'type': 'icmp_unreachable_high', + 'severity': 'warning', + 'isNormalBehavior': False, + 'message': ( + f'{unreachable} ICMP unreachable packets detected — this is higher than ' + f'expected for normal rolling updates. Investigate for misconfigured ' + f'services, missing endpoints, or network policy blocks.' + ), + }) + + return result + + +def _analyze_coredns_transients(lines: list) -> Dict: + """Detect brief DNS failures that occur during CoreDNS scaling events. + When CoreDNS pods scale down, there is a propagation delay for kube-proxy to update + iptables rules. During this window, DNS queries may be sent to a terminating CoreDNS pod + and get SERVFAIL or timeout. Setting lameduck duration in CoreDNS mitigates this. + Ref: https://docs.aws.amazon.com/eks/latest/best-practices/scale-cluster-services.html""" + servfail_re = re.compile(r'SERVFAIL|ServFail', re.IGNORECASE) + dns_timeout_re = re.compile(r'>\s+\S+\.53:.*\[.*\].*no\s+response', re.IGNORECASE) + + servfail_count = 0 + for line in lines: + if servfail_re.search(line): + servfail_count += 1 + + if servfail_count == 0: + return {} + + if servfail_count < 10: + return { + 'servfailCount': servfail_count, + 'isNormalBehavior': True, + 'explanation': ( + 'A small number of SERVFAIL responses can occur during CoreDNS pod scaling ' + 'events. When a CoreDNS pod terminates, there is a brief window where ' + 'kube-proxy iptables rules still route DNS queries to the terminating pod. ' + 'The CoreDNS lameduck plugin mitigates this by delaying shutdown. A few ' + 'SERVFAILs during scaling are transient and self-resolving.' + ), + 'anomalies': [{ + 'type': 'coredns_scaling_transient', + 'severity': 'info', + 'isNormalBehavior': True, + 'message': ( + f'{servfail_count} DNS SERVFAIL responses detected. Small numbers are ' + f'NORMAL during CoreDNS scaling events (lameduck propagation delay).' + ), + }], + } + else: + return { + 'servfailCount': servfail_count, + 'anomalies': [{ + 'type': 'high_servfail_rate', + 'severity': 'warning', + 'isNormalBehavior': False, + 'message': ( + f'{servfail_count} DNS SERVFAIL responses detected — this exceeds ' + f'normal CoreDNS scaling transients. Investigate CoreDNS health, ' + f'resource limits, and upstream DNS connectivity.' + ), + }], + } + + +def _analyze_syn_flood(lines: list) -> Dict: + """Detect SYN flood / connection flood patterns. + A high ratio of SYN packets without corresponding SYN-ACK indicates either a SYN flood + attack or an overwhelmed service that can't accept connections fast enough.""" + syn_re = re.compile(r'Flags\s+\[S\]', re.IGNORECASE) + synack_re = re.compile(r'Flags\s+\[S\.\]', re.IGNORECASE) + flow_re = re.compile(r'(\d+\.\d+\.\d+\.\d+)\.(\d+)\s+>\s+(\d+\.\d+\.\d+\.\d+)\.(\d+)') + + syn_count = 0 + synack_count = 0 + syn_sources: Dict[str, int] = {} + syn_targets: Dict[str, int] = {} + + for line in lines: + if synack_re.search(line): + synack_count += 1 + elif syn_re.search(line): + syn_count += 1 + m = flow_re.search(line) + if m: + src_ip = m.group(1) + dst = f"{m.group(3)}:{m.group(4)}" + syn_sources[src_ip] = syn_sources.get(src_ip, 0) + 1 + syn_targets[dst] = syn_targets.get(dst, 0) + 1 + + if syn_count < 20: + return {} + + result: Dict = { + 'synCount': syn_count, + 'synAckCount': synack_count, + 'anomalies': [], + } + + # Half-open ratio: SYN without SYN-ACK + if synack_count > 0: + half_open_ratio = (syn_count - synack_count) / syn_count + else: + half_open_ratio = 1.0 if syn_count > 0 else 0 + + top_sources = sorted(syn_sources.items(), key=lambda x: x[1], reverse=True)[:5] + top_targets = sorted(syn_targets.items(), key=lambda x: x[1], reverse=True)[:5] + result['topSynSources'] = [{'ip': ip, 'count': c} for ip, c in top_sources] + result['topSynTargets'] = [{'target': t, 'count': c} for t, c in top_targets] + + if half_open_ratio > 0.7 and syn_count > 50: + result['anomalies'].append({ + 'type': 'syn_flood', + 'severity': 'critical', + 'message': ( + f'{syn_count} SYN packets but only {synack_count} SYN-ACK responses ' + f'({half_open_ratio*100:.0f}% unanswered). Possible SYN flood attack or ' + f'target service is overwhelmed/unreachable. Top source: ' + f'{top_sources[0][0]} ({top_sources[0][1]} SYNs).' + ), + }) + elif half_open_ratio > 0.4: + result['anomalies'].append({ + 'type': 'connection_pressure', + 'severity': 'warning', + 'message': ( + f'{syn_count} SYN packets with {synack_count} SYN-ACK responses ' + f'({half_open_ratio*100:.0f}% unanswered). Service may be under connection ' + f'pressure or accept queue is full (check net.core.somaxconn).' + ), + }) + elif syn_count > 200: + result['anomalies'].append({ + 'type': 'high_connection_rate', + 'severity': 'info', + 'message': ( + f'{syn_count} new TCP connections in capture window. High but connections ' + f'are being accepted ({synack_count} SYN-ACKs). Monitor for scaling needs.' + ), + }) + + return result + + +def _analyze_tcp_window_zero(lines: list) -> Dict: + """Detect TCP window zero events indicating receiver backpressure. + When a pod's receive buffer is full, it advertises window size 0, telling the sender + to stop. This indicates the application can't consume data fast enough — common with + overwhelmed services, slow consumers, or memory pressure.""" + win_zero_re = re.compile(r'win\s+0\b', re.IGNORECASE) + flow_re = re.compile(r'(\d+\.\d+\.\d+\.\d+)\.(\d+)\s+>\s+(\d+\.\d+\.\d+\.\d+)\.(\d+)') + + zero_window_count = 0 + affected_flows: Dict[str, int] = {} + + for line in lines: + if win_zero_re.search(line): + zero_window_count += 1 + m = flow_re.search(line) + if m: + flow = f"{m.group(1)}:{m.group(2)}->{m.group(3)}:{m.group(4)}" + affected_flows[flow] = affected_flows.get(flow, 0) + 1 + + if zero_window_count == 0: + return {} + + top_flows = sorted(affected_flows.items(), key=lambda x: x[1], reverse=True)[:5] + + result: Dict = { + 'zeroWindowCount': zero_window_count, + 'affectedFlows': len(affected_flows), + 'topAffectedFlows': [{'flow': f, 'count': c} for f, c in top_flows], + 'anomalies': [], + } + + if zero_window_count > 20: + result['anomalies'].append({ + 'type': 'tcp_window_zero_critical', + 'severity': 'critical', + 'message': ( + f'{zero_window_count} TCP zero-window events across {len(affected_flows)} ' + f'flows. Receiver cannot consume data fast enough — application is overwhelmed. ' + f'Check pod memory limits, application processing capacity, and consider ' + f'horizontal scaling. Most affected: {top_flows[0][0]} ({top_flows[0][1]}x).' + ), + }) + elif zero_window_count > 5: + result['anomalies'].append({ + 'type': 'tcp_window_zero_warning', + 'severity': 'warning', + 'message': ( + f'{zero_window_count} TCP zero-window events detected. Receiver is ' + f'experiencing backpressure — may indicate slow application processing ' + f'or insufficient memory for socket buffers.' + ), + }) + + return result + + +def _analyze_retransmissions(lines: list) -> Dict: + """Detect TCP retransmission patterns indicating packet loss or network congestion. + Retransmissions show up as duplicate sequence numbers. High retransmission rates + indicate packet loss (security group drops, NACL drops, ENA throttling, or congestion).""" + retrans_re = re.compile(r'retransmit|retrans', re.IGNORECASE) + dup_ack_re = re.compile(r'dup\s+ack|duplicate\s+ack', re.IGNORECASE) + flow_re = re.compile(r'(\d+\.\d+\.\d+\.\d+)\.(\d+)\s+>\s+(\d+\.\d+\.\d+\.\d+)\.(\d+)') + + retrans_count = 0 + dup_ack_count = 0 + retrans_flows: Dict[str, int] = {} + + # Also detect retransmissions by looking for repeated seq numbers + seq_re = re.compile(r'seq\s+(\d+)[:\s]') + seen_seqs: Dict[str, set] = {} # flow -> set of seq numbers + + for line in lines: + if retrans_re.search(line): + retrans_count += 1 + m = flow_re.search(line) + if m: + flow = f"{m.group(1)}->{m.group(3)}:{m.group(4)}" + retrans_flows[flow] = retrans_flows.get(flow, 0) + 1 + if dup_ack_re.search(line): + dup_ack_count += 1 + + # Track seq numbers per flow for duplicate detection + m_flow = flow_re.search(line) + m_seq = seq_re.search(line) + if m_flow and m_seq: + flow_key = f"{m_flow.group(1)}->{m_flow.group(3)}:{m_flow.group(4)}" + seq_num = m_seq.group(1) + if flow_key not in seen_seqs: + seen_seqs[flow_key] = set() + if seq_num in seen_seqs[flow_key]: + retrans_count += 1 + retrans_flows[flow_key] = retrans_flows.get(flow_key, 0) + 1 + seen_seqs[flow_key].add(seq_num) + + if retrans_count == 0 and dup_ack_count == 0: + return {} + + top_flows = sorted(retrans_flows.items(), key=lambda x: x[1], reverse=True)[:5] + + result: Dict = { + 'retransmissionCount': retrans_count, + 'duplicateAckCount': dup_ack_count, + 'affectedFlows': len(retrans_flows), + 'topRetransmitFlows': [{'flow': f, 'count': c} for f, c in top_flows], + 'anomalies': [], + } + + total_packets = len(lines) + retrans_pct = (retrans_count / total_packets * 100) if total_packets > 0 else 0 + + if retrans_pct > 5: + result['anomalies'].append({ + 'type': 'high_retransmission_rate', + 'severity': 'critical', + 'message': ( + f'{retrans_count} retransmissions ({retrans_pct:.1f}% of packets). ' + f'Severe packet loss — check ENA throttling (linklocal_allowance_exceeded), ' + f'security group/NACL drops, or network congestion. ' + f'{len(retrans_flows)} flows affected.' + ), + }) + elif retrans_pct > 1: + result['anomalies'].append({ + 'type': 'moderate_retransmission_rate', + 'severity': 'warning', + 'message': ( + f'{retrans_count} retransmissions ({retrans_pct:.1f}% of packets). ' + f'Moderate packet loss detected. Check for ENA bandwidth/PPS throttling ' + f'or intermittent network issues.' + ), + }) + elif retrans_count > 0: + result['anomalies'].append({ + 'type': 'low_retransmissions', + 'severity': 'info', + 'message': ( + f'{retrans_count} retransmissions detected ({retrans_pct:.1f}%). ' + f'Low level — within normal range for most workloads.' + ), + }) + + return result + + +def _analyze_connection_refused(lines: list) -> Dict: + """Detect connection refused patterns (RST immediately after SYN). + This indicates the target port is not listening — common when a pod hasn't started, + a service endpoint is stale, or a NetworkPolicy is blocking traffic.""" + flow_re = re.compile(r'(\d+\.\d+\.\d+\.\d+)\.(\d+)\s+>\s+(\d+\.\d+\.\d+\.\d+)\.(\d+)') + syn_re = re.compile(r'Flags\s+\[S\]', re.IGNORECASE) + rst_re = re.compile(r'Flags\s+\[R\.?\]', re.IGNORECASE) + + # Track SYN -> RST pairs (connection refused = RST right after SYN) + recent_syns: Dict[str, str] = {} # "dst:port" -> src line + refused: Dict[str, int] = {} # "dst:port" -> count + + for line in lines: + m = flow_re.search(line) + if not m: + continue + src_ip, src_port, dst_ip, dst_port = m.groups() + + if syn_re.search(line): + key = f"{dst_ip}:{dst_port}" + recent_syns[key] = src_ip + elif rst_re.search(line): + # RST coming FROM the destination back to source + reverse_key = f"{src_ip}:{src_port}" + if reverse_key in recent_syns: + refused[reverse_key] = refused.get(reverse_key, 0) + 1 + + if not refused: + return {} + + top_refused = sorted(refused.items(), key=lambda x: x[1], reverse=True)[:10] + total_refused = sum(refused.values()) + + result: Dict = { + 'totalConnectionRefused': total_refused, + 'uniqueTargets': len(refused), + 'topRefusedTargets': [{'target': t, 'count': c} for t, c in top_refused], + 'anomalies': [], + } + + if total_refused > 50: + result['anomalies'].append({ + 'type': 'mass_connection_refused', + 'severity': 'critical', + 'message': ( + f'{total_refused} connections refused across {len(refused)} targets. ' + f'Services are not listening or pods are not ready. Top target: ' + f'{top_refused[0][0]} ({top_refused[0][1]}x refused). Check pod readiness, ' + f'service endpoints, and NetworkPolicy rules.' + ), + }) + elif total_refused > 10: + result['anomalies'].append({ + 'type': 'connection_refused', + 'severity': 'warning', + 'message': ( + f'{total_refused} connections refused to {len(refused)} targets. ' + f'Some services may not be ready or endpoints are stale.' + ), + }) + + return result + + +def _analyze_traffic_burst(lines: list) -> Dict: + """Detect traffic bursts by analyzing packet timestamps. + Identifies periods of abnormally high packet rates that could indicate + DDoS, thundering herd, or misconfigured retry storms.""" + ts_re = re.compile(r'^(\d{2}:\d{2}:\d{2}\.\d+)\s') + + # Group packets by second + packets_per_second: Dict[str, int] = {} + for line in lines: + m = ts_re.match(line) + if m: + ts = m.group(1).split('.')[0] # truncate to second + packets_per_second[ts] = packets_per_second.get(ts, 0) + 1 + + if len(packets_per_second) < 5: + return {} + + rates = list(packets_per_second.values()) + avg_rate = sum(rates) / len(rates) + max_rate = max(rates) + max_ts = max(packets_per_second, key=packets_per_second.get) + + # Find burst periods (>3x average) + burst_seconds = [(ts, count) for ts, count in packets_per_second.items() + if count > avg_rate * 3 and count > 20] + burst_seconds.sort(key=lambda x: x[1], reverse=True) + + result: Dict = { + 'avgPacketsPerSecond': round(avg_rate, 1), + 'maxPacketsPerSecond': max_rate, + 'peakTime': max_ts, + 'captureDurationSeconds': len(packets_per_second), + 'anomalies': [], + } + + if burst_seconds: + result['burstPeriods'] = [{'time': ts, 'packetsPerSecond': c} + for ts, c in burst_seconds[:10]] + if max_rate > avg_rate * 10 and max_rate > 100: + result['anomalies'].append({ + 'type': 'extreme_traffic_burst', + 'severity': 'critical', + 'message': ( + f'Extreme traffic burst: {max_rate} pps at {max_ts} vs average ' + f'{avg_rate:.0f} pps ({max_rate/avg_rate:.0f}x spike). ' + f'{len(burst_seconds)} burst periods detected. Possible DDoS, ' + f'retry storm, or thundering herd.' + ), + }) + elif burst_seconds: + result['anomalies'].append({ + 'type': 'traffic_burst', + 'severity': 'warning', + 'message': ( + f'Traffic bursts detected: peak {max_rate} pps at {max_ts} vs ' + f'average {avg_rate:.0f} pps. {len(burst_seconds)} periods exceeded ' + f'3x average rate.' + ), + }) + + return result + + +def _analyze_top_talkers(lines: list) -> Dict: + """Identify top bandwidth consumers and communication patterns. + Helps identify which pods/IPs are generating the most traffic and whether + traffic distribution is skewed (one pod hogging bandwidth).""" + flow_re = re.compile( + r'(\d+\.\d+\.\d+\.\d+)\.(\d+)\s+>\s+(\d+\.\d+\.\d+\.\d+)\.(\d+).*length\s+(\d+)' + ) + + src_bytes: Dict[str, int] = {} + dst_bytes: Dict[str, int] = {} + src_packets: Dict[str, int] = {} + dst_packets: Dict[str, int] = {} + flow_bytes: Dict[str, int] = {} + + for line in lines: + m = flow_re.search(line) + if m: + src_ip, src_port, dst_ip, dst_port, length = m.groups() + length = int(length) + src_bytes[src_ip] = src_bytes.get(src_ip, 0) + length + dst_bytes[dst_ip] = dst_bytes.get(dst_ip, 0) + length + src_packets[src_ip] = src_packets.get(src_ip, 0) + 1 + dst_packets[dst_ip] = dst_packets.get(dst_ip, 0) + 1 + flow_key = f"{src_ip}->{dst_ip}:{dst_port}" + flow_bytes[flow_key] = flow_bytes.get(flow_key, 0) + length + + if not src_bytes: + return {} + + top_src = sorted(src_bytes.items(), key=lambda x: x[1], reverse=True)[:10] + top_dst = sorted(dst_bytes.items(), key=lambda x: x[1], reverse=True)[:10] + top_flows = sorted(flow_bytes.items(), key=lambda x: x[1], reverse=True)[:10] + + total_bytes = sum(src_bytes.values()) + + result: Dict = { + 'totalBytes': total_bytes, + 'totalBytesHuman': f'{total_bytes/1024:.1f} KB' if total_bytes < 1048576 else f'{total_bytes/1048576:.1f} MB', + 'uniqueSources': len(src_bytes), + 'uniqueDestinations': len(dst_bytes), + 'topSenders': [{'ip': ip, 'bytes': b, 'packets': src_packets.get(ip, 0)} + for ip, b in top_src], + 'topReceivers': [{'ip': ip, 'bytes': b, 'packets': dst_packets.get(ip, 0)} + for ip, b in top_dst], + 'topFlows': [{'flow': f, 'bytes': b} for f, b in top_flows], + 'anomalies': [], + } + + # Check for traffic skew — one source dominating + if top_src and total_bytes > 0: + top_pct = (top_src[0][1] / total_bytes) * 100 + if top_pct > 80 and len(src_bytes) > 3: + result['anomalies'].append({ + 'type': 'traffic_skew', + 'severity': 'warning', + 'message': ( + f'Traffic heavily skewed: {top_src[0][0]} sends {top_pct:.0f}% of all ' + f'bytes ({top_src[0][1]} bytes). Possible bandwidth hog or ' + f'misconfigured client retry loop.' + ), + }) + + return result + + +def _analyze_mtu_fragmentation(lines: list) -> Dict: + """Detect MTU/fragmentation issues from packet captures. + Fragmented packets indicate MTU mismatch. In EKS, the VPC MTU is typically 9001 + (jumbo frames) but tunnels (VPN, VXLAN) or cross-AZ traffic may have lower MTU. + Excessive fragmentation causes performance degradation and can break PMTUD.""" + frag_re = re.compile(r'frag\s+\d+|offset\s+\d+|flags\s+\[.*MF.*\]', re.IGNORECASE) + df_re = re.compile(r'flags\s+\[.*DF.*\]', re.IGNORECASE) + length_re = re.compile(r'length\s+(\d+)') + + frag_count = 0 + df_count = 0 + large_packets = 0 # packets > 1500 bytes (jumbo) + + for line in lines: + if frag_re.search(line): + frag_count += 1 + if df_re.search(line): + df_count += 1 + m = length_re.search(line) + if m and int(m.group(1)) > 1500: + large_packets += 1 + + if frag_count == 0 and large_packets == 0: + return {} + + result: Dict = { + 'fragmentedPackets': frag_count, + 'dontFragmentPackets': df_count, + 'jumboPackets': large_packets, + 'anomalies': [], + } + + if frag_count > 20: + result['anomalies'].append({ + 'type': 'excessive_fragmentation', + 'severity': 'warning', + 'message': ( + f'{frag_count} fragmented packets detected. MTU mismatch likely — ' + f'check if traffic crosses VPN tunnels, VXLAN overlays, or different ' + f'MTU boundaries. Consider setting pod MTU explicitly or enabling PMTUD. ' + f'EKS VPC default MTU is 9001 (jumbo frames).' + ), + }) + elif frag_count > 0: + result['anomalies'].append({ + 'type': 'minor_fragmentation', + 'severity': 'info', + 'message': ( + f'{frag_count} fragmented packets. Low level — may be normal for ' + f'cross-region or VPN traffic.' + ), + }) + + return result + + +def _analyze_conntrack_pressure(lines: list) -> Dict: + """Estimate conntrack table pressure from unique connection count. + Each TCP/UDP flow consumes a conntrack entry. Default nf_conntrack_max is 131072. + High unique flow counts in a short capture window suggest conntrack exhaustion risk.""" + flow_re = re.compile( + r'(\d+\.\d+\.\d+\.\d+)\.(\d+)\s+>\s+(\d+\.\d+\.\d+\.\d+)\.(\d+)' + ) + + unique_flows = set() + for line in lines: + m = flow_re.search(line) + if m: + # Bidirectional: normalize so A->B and B->A count as one flow + src = f"{m.group(1)}:{m.group(2)}" + dst = f"{m.group(3)}:{m.group(4)}" + flow = tuple(sorted([src, dst])) + unique_flows.add(flow) + + if len(unique_flows) < 100: + return {} + + result: Dict = { + 'uniqueFlows': len(unique_flows), + 'anomalies': [], + } + + # Default conntrack max is 131072; warn at 50% observed in a short window + if len(unique_flows) > 50000: + result['anomalies'].append({ + 'type': 'conntrack_exhaustion_risk', + 'severity': 'critical', + 'message': ( + f'{len(unique_flows)} unique flows observed in capture window. ' + f'Default nf_conntrack_max is 131072 — node may be at risk of ' + f'conntrack table exhaustion. Check: ' + f'cat /proc/sys/net/netfilter/nf_conntrack_count vs nf_conntrack_max. ' + f'Symptoms: "nf_conntrack: table full, dropping packet" in dmesg.' + ), + }) + elif len(unique_flows) > 10000: + result['anomalies'].append({ + 'type': 'high_flow_count', + 'severity': 'warning', + 'message': ( + f'{len(unique_flows)} unique flows in capture window. Monitor conntrack ' + f'usage — high flow counts can exhaust the conntrack table ' + f'(default max 131072).' + ), + }) + else: + result['anomalies'].append({ + 'type': 'flow_count_info', + 'severity': 'info', + 'message': f'{len(unique_flows)} unique flows observed. Within normal range.', + }) + + return result + + +def tcpdump_analyze(arguments: Dict) -> Dict: + """ + Read and analyze a completed tcpdump capture from S3. + Returns decoded packet text, protocol statistics, and top talkers. + + + Inputs: + instanceId: EC2 instance ID (required) + commandId: SSM Command ID from tcpdump_capture (optional — finds latest if omitted) + section: "summary" (first N packets decoded), "stats" (protocol breakdown), "all" (default: "all") + maxPackets: Max decoded packet lines to return (default: 500, max: 3000) + filter: Text filter to apply on decoded lines (e.g., "SYN", "RST", "10.0.0.5") + + Returns: + Decoded packet text, protocol stats, top talkers, and anomaly indicators + """ + instance_id = arguments.get('instanceId') + if not instance_id: + return error_response(400, 'instanceId is required') + + command_id = arguments.get('commandId') + section = arguments.get('section', 'all') + max_packets = min(int(arguments.get('maxPackets', 500)), 3000) + text_filter = arguments.get('filter', '') + + # ── Always find the LATEST capture for this instance ── + # Even if commandId is provided, we verify it is the latest capture. + # This prevents analyzing stale data when a newer capture exists. + metadata = {} + latest_metadata = {} + try: + list_resp = safe_s3_list(f"tcpdump-commands/", max_keys=200) + if list_resp.get('success'): + candidates = [] + for obj in list_resp.get('objects', []): + try: + r = s3_client.get_object(Bucket=LOGS_BUCKET, Key=obj['key']) + m = json.loads(r['Body'].read().decode('utf-8')) + if m.get('instanceId') == instance_id: + candidates.append(m) + except Exception: + continue + if candidates: + candidates.sort(key=lambda x: x.get('startedAt', ''), reverse=True) + latest_metadata = candidates[0] + except Exception: + pass + + if command_id and latest_metadata: + # commandId was provided — check if it matches the latest + if latest_metadata.get('commandId') == command_id: + metadata = latest_metadata + else: + # Requested commandId is NOT the latest — reject with guidance + return error_response(409, + f'commandId {command_id} is not the latest capture for {instance_id}. ' + f'Latest capture is commandId={latest_metadata.get("commandId")} ' + f'started at {latest_metadata.get("startedAt", "unknown")}. ' + f'Omit commandId to auto-use the latest, or run a new tcpdump_capture.') + elif latest_metadata: + metadata = latest_metadata + elif command_id: + # No candidates found at all — try the specific commandId as fallback + try: + meta_resp = s3_client.get_object( + Bucket=LOGS_BUCKET, + Key=f"tcpdump-commands/{command_id}.json", + ) + metadata = json.loads(meta_resp['Body'].read().decode('utf-8')) + except Exception: + pass + + if not metadata: + return error_response(404, f'No tcpdump capture found for {instance_id}. Run tcpdump_capture first.') + + # ── Staleness check ── + capture_age_warning = None + started_at = metadata.get('startedAt', '') + if started_at: + try: + capture_time = datetime.strptime(started_at, '%Y-%m-%dT%H:%M:%SZ') + age_seconds = (datetime.utcnow() - capture_time).total_seconds() + age_minutes = age_seconds / 60 + if age_minutes > 15: + capture_age_warning = ( + f'This capture is {int(age_minutes)} minutes old (started {started_at}). ' + f'Network conditions may have changed. Consider running a fresh tcpdump_capture.' + ) + except (ValueError, TypeError): + pass + + s3_key_txt = metadata.get('s3KeyTxt', '') + s3_key_stats = metadata.get('s3KeyStats', '') + s3_key_pcap = metadata.get('s3Key', '') + + results = { + 'instanceId': instance_id, + 'commandId': metadata.get('commandId', command_id or 'unknown'), + 'captureInfo': { + 'interface': metadata.get('interface', 'unknown'), + 'filter': metadata.get('filter', 'none'), + 'durationSeconds': metadata.get('durationSeconds', 0), + 'startedAt': metadata.get('startedAt', 'unknown'), + 'captureScope': metadata.get('captureScope', 'hostNamespace'), + 'nsenterUsed': metadata.get('nsenterUsed', False), + 'networkNamespace': metadata.get('networkNamespace', 'host'), + 'podName': metadata.get('podName'), + 'podNamespace': metadata.get('podNamespace'), + 'containerPid': metadata.get('containerPid'), + }, + } + if capture_age_warning: + results['stalenessWarning'] = capture_age_warning + + # Read stats + if section in ('stats', 'all'): + stats = {} + if s3_key_stats: + try: + resp = safe_s3_read(s3_key_stats, max_size=65536) + if resp.get('success') and resp.get('content'): + stats = json.loads(resp['content']) + except (json.JSONDecodeError, Exception): + stats = {'error': 'Could not parse stats JSON'} + else: + stats = {'error': 'No stats file found — capture may still be in progress'} + + results['statistics'] = stats + + # Anomaly detection from stats + anomalies = [] + if isinstance(stats, dict) and 'totalPackets' in stats: + total = stats.get('totalPackets', 0) + rst_count = stats.get('tcpFlags', {}).get('rst', 0) + syn_count = stats.get('tcpFlags', {}).get('syn', 0) + retrans = stats.get('possibleRetransmits', 0) + + if total > 0: + rst_pct = (rst_count / total) * 100 + if rst_pct > 5: + anomalies.append({ + 'type': 'high_rst_rate', + 'severity': 'warning' if rst_pct < 15 else 'critical', + 'message': f'{rst_pct:.1f}% of packets are TCP RST ({rst_count}/{total}) — possible connection rejection or firewall drops', + }) + if retrans > 0: + retrans_pct = (retrans / total) * 100 + anomalies.append({ + 'type': 'retransmissions', + 'severity': 'warning' if retrans_pct < 5 else 'critical', + 'message': f'{retrans} possible retransmissions detected ({retrans_pct:.1f}%) — network congestion or packet loss', + }) + if syn_count > 0 and rst_count > syn_count * 0.5: + anomalies.append({ + 'type': 'syn_rst_ratio', + 'severity': 'warning', + 'message': f'High RST-to-SYN ratio ({rst_count} RST vs {syn_count} SYN) — many connections being refused', + }) + icmp_count = stats.get('protocols', {}).get('icmp', 0) + if icmp_count > total * 0.1: + anomalies.append({ + 'type': 'high_icmp', + 'severity': 'info', + 'message': f'{icmp_count} ICMP packets ({(icmp_count/total)*100:.1f}%) — possible ping flood or unreachable destinations', + }) + + results['anomalies'] = anomalies + + # Read decoded text summary + # NOTE: all_lines holds the FULL packet set for analysis; decoded_lines is truncated for response payload + all_lines = [] + if section in ('summary', 'all'): + decoded_lines = [] + if s3_key_txt: + try: + resp = safe_s3_read(s3_key_txt, max_size=2 * 1024 * 1024) # 2MB max + if resp.get('success') and resp.get('content'): + all_lines = resp['content'].split('\n') + + # Apply text filter if provided (only for display, not analysis) + display_lines = all_lines + if text_filter: + pattern = re.compile(re.escape(text_filter), re.IGNORECASE) + display_lines = [l for l in all_lines if pattern.search(l)] + + total_lines = len(display_lines) + decoded_lines = display_lines[:max_packets] + + results['decodedPackets'] = { + 'lines': decoded_lines, + 'totalPackets': total_lines, + 'returnedPackets': len(decoded_lines), + 'truncated': total_lines > max_packets, + 'filter': text_filter or 'none', + 'analyzedPackets': len(all_lines), + } + else: + results['decodedPackets'] = {'error': 'Text summary file is empty or unreadable'} + except Exception as e: + results['decodedPackets'] = {'error': f'Failed to read text summary: {str(e)}'} + else: + results['decodedPackets'] = {'error': 'No text summary file found — capture may still be in progress'} + + # DNS analysis — scan ALL packets (not just truncated display lines) + if section in ('summary', 'all') and all_lines: + dns_analysis = _analyze_dns_packets(all_lines) + if dns_analysis: + results['dnsAnalysis'] = dns_analysis + # Merge DNS anomalies into the main anomalies list + if 'anomalies' in results: + results['anomalies'].extend(dns_analysis.get('anomalies', [])) + else: + results['anomalies'] = dns_analysis.get('anomalies', []) + + # Expected behavior analysis — run on ALL lines for full coverage + # Each analyzer is independent and returns its own findings + if section in ('summary', 'all') and all_lines: + expected_behaviors = {} + expected_anomalies = [] + + tcp_rst = _analyze_tcp_rst_patterns(all_lines) + if tcp_rst: + expected_behaviors['tcpRstPatterns'] = tcp_rst + expected_anomalies.extend(tcp_rst.get('anomalies', [])) + + dnat = _analyze_kube_proxy_dnat(all_lines) + if dnat: + expected_behaviors['kubeProxyDnat'] = dnat + expected_anomalies.extend(dnat.get('anomalies', [])) + + snat = _analyze_vpc_cni_snat(all_lines) + if snat: + expected_behaviors['vpcCniSnat'] = snat + expected_anomalies.extend(snat.get('anomalies', [])) + + keepalive = _analyze_tcp_keepalives(all_lines) + if keepalive: + expected_behaviors['tcpKeepalives'] = keepalive + expected_anomalies.extend(keepalive.get('anomalies', [])) + + icmp = _analyze_icmp_expected(all_lines) + if icmp: + expected_behaviors['icmpPatterns'] = icmp + expected_anomalies.extend(icmp.get('anomalies', [])) + + coredns = _analyze_coredns_transients(all_lines) + if coredns: + expected_behaviors['corednsTransients'] = coredns + expected_anomalies.extend(coredns.get('anomalies', [])) + + # ── Deep network analysis (complex issue detection) ── + syn_flood = _analyze_syn_flood(all_lines) + if syn_flood: + expected_behaviors['synFloodDetection'] = syn_flood + expected_anomalies.extend(syn_flood.get('anomalies', [])) + + win_zero = _analyze_tcp_window_zero(all_lines) + if win_zero: + expected_behaviors['tcpWindowZero'] = win_zero + expected_anomalies.extend(win_zero.get('anomalies', [])) + + retrans = _analyze_retransmissions(all_lines) + if retrans: + expected_behaviors['retransmissions'] = retrans + expected_anomalies.extend(retrans.get('anomalies', [])) + + conn_refused = _analyze_connection_refused(all_lines) + if conn_refused: + expected_behaviors['connectionRefused'] = conn_refused + expected_anomalies.extend(conn_refused.get('anomalies', [])) + + burst = _analyze_traffic_burst(all_lines) + if burst: + expected_behaviors['trafficBurst'] = burst + expected_anomalies.extend(burst.get('anomalies', [])) + + talkers = _analyze_top_talkers(all_lines) + if talkers: + expected_behaviors['topTalkers'] = talkers + expected_anomalies.extend(talkers.get('anomalies', [])) + + mtu_frag = _analyze_mtu_fragmentation(all_lines) + if mtu_frag: + expected_behaviors['mtuFragmentation'] = mtu_frag + expected_anomalies.extend(mtu_frag.get('anomalies', [])) + + conntrack = _analyze_conntrack_pressure(all_lines) + if conntrack: + expected_behaviors['conntrackPressure'] = conntrack + expected_anomalies.extend(conntrack.get('anomalies', [])) + + if expected_behaviors: + results['expectedBehaviors'] = expected_behaviors + if 'anomalies' in results: + results['anomalies'].extend(expected_anomalies) + else: + results['anomalies'] = expected_anomalies + + # Presigned URL for pcap download + if s3_key_pcap: + try: + results['pcapDownloadUrl'] = s3_client.generate_presigned_url( + 'get_object', + Params={'Bucket': LOGS_BUCKET, 'Key': s3_key_pcap}, + ExpiresIn=PCAP_PRESIGNED_URL_EXPIRATION, + ) + results['pcapDownloadUrlExpiresIn'] = f'{PCAP_PRESIGNED_URL_EXPIRATION} seconds' + except Exception: + pass + + results['s3Bucket'] = LOGS_BUCKET + results['s3KeyPcap'] = s3_key_pcap + results['s3KeyTxt'] = s3_key_txt + results['s3KeyStats'] = s3_key_stats + + return success_response(results) diff --git a/mcp/aws-eks-node-diagnostics-mcp/src/ssm-automation-gateway-construct-v2.ts b/mcp/aws-eks-node-diagnostics-mcp/src/ssm-automation-gateway-construct-v2.ts index a43751d..d4e2197 100644 --- a/mcp/aws-eks-node-diagnostics-mcp/src/ssm-automation-gateway-construct-v2.ts +++ b/mcp/aws-eks-node-diagnostics-mcp/src/ssm-automation-gateway-construct-v2.ts @@ -10,7 +10,7 @@ import * as kms from 'aws-cdk-lib/aws-kms'; import * as logs from 'aws-cdk-lib/aws-logs'; import * as cloudwatch from 'aws-cdk-lib/aws-cloudwatch'; import * as ec2 from 'aws-cdk-lib/aws-ec2'; -import * as dynamodb from 'aws-cdk-lib/aws-dynamodb'; +import * as ssm from 'aws-cdk-lib/aws-ssm'; import * as sns from 'aws-cdk-lib/aws-sns'; import * as subscriptions from 'aws-cdk-lib/aws-sns-subscriptions'; import { Construct } from 'constructs'; @@ -92,37 +92,38 @@ export interface SsmAutomationGatewayV2Props { /** * Require human-in-the-loop approval before the mutating collection tools * (`collect`, `batch_collect`) start SSM Automation on nodes (security review - * M1/M2). When true, a call creates a pending approval and notifies approvers - * via SNS; the SSM execution only runs after a human approves out-of-band via - * the approval Function URL. Approval is NOT a tool parameter the agent can - * set — the approve link carries a one-time secret delivered only to humans. + * M1/M2). When true, collection runs through a wrapper SSM Automation document + * whose FIRST step is the native `aws:approve` action: the execution pauses + * inside SSM until one of the designated approvers approves it in the Systems + * Manager console (or via `ssm:SendAutomationSignal`), then the collection + * step runs automatically. Approval is NOT a tool parameter the agent can + * set — the MCP Lambda has no `ssm:SendAutomationSignal` permission, so it + * cannot approve its own requests. * @default true */ readonly requireCollectionApproval?: boolean; /** - * Expose a public (authType NONE) Lambda Function URL so approvers can - * approve/deny with a one-click link. Default false: some accounts run - * guardrails (SCPs or mitigation services) that strip public Lambda - * resource policies, which silently breaks the link with a "Forbidden" - * response. When false, no public endpoint is created — the SNS - * notification instead contains an IAM-authenticated `aws lambda invoke` - * command the approver runs with their own credentials. - * @default false + * IAM principals allowed to approve collection requests (IAM user ARNs, IAM + * role ARNs, or IAM usernames — passed to the `aws:approve` step's Approvers + * field). REQUIRED when `requireCollectionApproval` is true (the default); + * synth fails without it so a deployment can never silently lack approvers. + * Approvers also need `ssm:SendAutomationSignal` to click Approve/Deny. */ - readonly approvalViaPublicUrl?: boolean; + readonly approvalApproverArns?: string[]; /** * Email addresses subscribed to the collection-approval SNS topic. Each - * receives the approve/deny link when a collection is requested. (You can also - * subscribe Slack/PagerDuty/etc. to the topic out of band.) + * receives a notification with the SSM console approval link when a + * collection is requested. (You can also subscribe Slack/PagerDuty/etc. to + * the topic out of band.) * @default [] (no email subscriptions created; subscribe to the topic yourself) */ readonly approvalNotificationEmails?: string[]; /** - * How long (seconds) a pending collection approval remains valid before it - * expires and must be re-requested. + * How long (seconds) the `aws:approve` step waits for a human decision before + * the execution times out and must be re-requested. * @default 900 (15 minutes) */ readonly approvalTtlSeconds?: number; @@ -200,6 +201,36 @@ export interface SsmAutomationGatewayV2Props { */ readonly perCallerRateLimitPerMinute?: number; + /** + * List of restricted tools to enable (e.g., ['tcpdump_capture', 'tcpdump_analyze']). + * Restricted tools perform invasive operations (network packet captures, + * container namespace entry via nsenter) and are completely absent from the + * MCP tool surface unless listed here. + * + * SECURITY: enabling `tcpdump_capture` does NOT bypass human approval — when + * `requireCollectionApproval` is on (the default), every capture runs through + * a wrapper SSM Automation document that pauses at a native `aws:approve` + * step until a designated approver approves it in the Systems Manager console. + * @default [] (tcpdump tools are not available) + */ + readonly enableRestrictedTools?: string[]; + + /** + * Presigned URL expiration in seconds for pcap downloads generated by the + * `tcpdump_*` tools. Network captures may contain credentials in transit and + * other sensitive payloads, so they get a much shorter window than ordinary + * log artifacts. Capped at 300s by the Lambda. + * @default 60 + */ + readonly pcapPresignedUrlExpirationSeconds?: number; + + /** + * Size (bytes) above which a completed pcap capture is flagged as oversized + * in tool responses (advisory warning only). + * @default 209715200 (200 MiB) + */ + readonly maxPcapBytes?: number; + } /** @@ -225,7 +256,6 @@ export class SsmAutomationGatewayV2Construct extends Construct { public readonly gatewayExecutionRole: iam.Role; public readonly encryptionKey?: kms.Key; public readonly sopBucket: s3.Bucket; - public readonly collectionApprovalTable: dynamodb.Table; public readonly collectionApprovalTopic: sns.Topic; constructor(scope: Construct, id: string, props: SsmAutomationGatewayV2Props = {}) { @@ -243,6 +273,10 @@ export class SsmAutomationGatewayV2Construct extends Construct { ? props.allowedRegions : [cdk.Stack.of(this).region]; + // Restricted tools (tcpdump) are opt-in; tcpdump_capture is additionally + // approval-gated at runtime (see the approval section below). + const enabledRestrictedTools = props.enableRestrictedTools ?? []; + // ======================================================================== // KMS ENCRYPTION KEY (if enabled) // ======================================================================== @@ -512,13 +546,16 @@ export class SsmAutomationGatewayV2Construct extends Construct { } } - // Grant SSM Default Host Management role KMS encrypt access for log uploads + // Grant SSM Default Host Management role KMS access for log uploads. + // kms:Decrypt is required in addition to GenerateDataKey/Encrypt: bundles + // over the AWS CLI's 8 MiB threshold use S3 multipart upload, and multipart + // to an SSE-KMS bucket needs kms:Decrypt (UploadPart fails without it). if (this.encryptionKey && props.ssmDefaultHostRoleArn) { this.encryptionKey.addToResourcePolicy(new iam.PolicyStatement({ sid: 'AllowSSMDefaultHostRoleEncrypt', effect: iam.Effect.ALLOW, principals: [new iam.ArnPrincipal(props.ssmDefaultHostRoleArn)], - actions: ['kms:GenerateDataKey', 'kms:Encrypt'], + actions: ['kms:GenerateDataKey', 'kms:Encrypt', 'kms:Decrypt'], resources: ['*'], })); } @@ -714,12 +751,37 @@ export class SsmAutomationGatewayV2Construct extends Construct { resources: startAutomationResources, })); - // NOTE: The Lambda intentionally has NO ssm:SendCommand permission. Log - // collection runs via ssm:StartAutomationExecution (granted above) using the - // AWSSupport-CollectEKSInstanceLogs automation document, whose runCommand - // steps execute under the ssmAutomationRole — not this role. Direct - // SendCommand was only used by the removed tcpdump tools, so dropping it here - // eliminates the Lambda's ability to run arbitrary shell commands on nodes. + // NOTE: The Lambda intentionally has NO ssm:SendCommand permission while + // the approval workflow is on. Log collection AND tcpdump captures run via + // ssm:StartAutomationExecution (granted above) using automation documents + // whose runCommand steps execute under the ssmAutomationRole — not this + // role — and the tcpdump wrapper pauses at a native aws:approve step first. + // This means a poisoned agent cannot run shell commands on nodes without a + // human approving each capture in the SSM console. + // + // The ONLY exception: when the operator explicitly disables approval + // (requireCollectionApproval: false — supervised/test deployments) AND + // enables the restricted tcpdump tools, the Lambda needs direct + // SendCommand for the legacy capture path. Even then it is scoped to the + // allowed documents (AWS-RunShellScript) and EKS-tagged instances only. + if ((props.requireCollectionApproval ?? true) === false + && enabledRestrictedTools.includes('tcpdump_capture')) { + lambdaExecutionRole.addToPolicy(new iam.PolicyStatement({ + sid: 'SSMSendCommandDocsTcpdump', + effect: iam.Effect.ALLOW, + actions: ['ssm:SendCommand'], + resources: ssmDocResources, + })); + lambdaExecutionRole.addToPolicy(new iam.PolicyStatement({ + sid: 'SSMSendCommandInstancesTcpdump', + effect: iam.Effect.ALLOW, + actions: ['ssm:SendCommand'], + resources: ssmInstanceResources, + conditions: { + [clusterConditionOperator]: clusterTagCondition, + }, + })); + } // SSM StopAutomation (region restricted) lambdaExecutionRole.addToPolicy(new iam.PolicyStatement({ @@ -892,27 +954,51 @@ export class SsmAutomationGatewayV2Construct extends Construct { .join(';'); // ── Human-in-the-loop approval for mutating collection tools (M1/M2) ── + // + // SSM-native design: collection runs through wrapper Automation documents + // whose FIRST step is the `aws:approve` action. The execution pauses inside + // SSM until a designated approver clicks Approve/Deny in the Systems + // Manager console (or calls ssm:SendAutomationSignal); only then does the + // document proceed to run AWSSupport-CollectEKSInstanceLogs. Approval state + // lives entirely in SSM — no custom tokens, endpoints, or tables — and + // every decision is CloudTrail-audited. The MCP Lambda deliberately has NO + // ssm:SendAutomationSignal permission, so a poisoned agent cannot approve + // its own requests. const requireCollectionApproval = props.requireCollectionApproval ?? true; const approvalTtlSeconds = props.approvalTtlSeconds ?? 900; + const approverArns = props.approvalApproverArns ?? []; - // Stores pending/approved collection requests. approvalId is a random, - // server-generated id; the secret approve token is stored only as a hash. - // DynamoDB TTL auto-expires stale requests. - const approvalTable = new dynamodb.Table(this, 'CollectionApprovalTable', { - partitionKey: { name: 'approvalId', type: dynamodb.AttributeType.STRING }, - billingMode: dynamodb.BillingMode.PAY_PER_REQUEST, - timeToLiveAttribute: 'ttl', - pointInTimeRecovery: true, - encryption: props.enableEncryption === false - ? dynamodb.TableEncryption.AWS_MANAGED - : dynamodb.TableEncryption.CUSTOMER_MANAGED, - encryptionKey: props.enableEncryption === false ? undefined : this.encryptionKey, - removalPolicy: cdk.RemovalPolicy.DESTROY, - }); - this.collectionApprovalTable = approvalTable; + // Fail-closed at synth: an approval-gated deployment without approvers + // would leave every collection stuck at the approve step forever. + if (requireCollectionApproval && approverArns.length === 0) { + throw new Error( + 'SsmAutomationGatewayV2: `requireCollectionApproval` is enabled (the default) ' + + 'but `approvalApproverArns` is empty. Provide the IAM users/roles allowed to ' + + 'approve collections (APPROVAL_APPROVER_ARNS env var), or explicitly set ' + + '`requireCollectionApproval: false` for a fully supervised/test deployment.', + ); + } + + // The approval wrapper documents are regional SSM documents created only in + // this stack's home region, so approval-gated collect()/batch_collect() + // calls targeting any other allowed region fail at runtime with a 400. + // Surface that at synth time instead of on the first cross-region call. + if (requireCollectionApproval && allowedRegions.length > 1) { + cdk.Annotations.of(this).addWarning( + `requireCollectionApproval is enabled but this deployment allows ` + + `${allowedRegions.length} regions (${allowedRegions.join(', ')}). The approval ` + + `wrapper SSM documents exist only in this stack's home region ` + + `(${cdk.Stack.of(this).region}), so approval-gated collect()/batch_collect() ` + + `calls targeting the other region(s) will fail with a 400. Deploy a stack in ` + + `each region that needs approval-gated collection.`, + ); + } - // Approvers are notified here with the approve/deny link. + // Approvers are notified here with the SSM console approval link. SSM + // requires Automation-approval notification topics to be named with an + // "Automation" prefix. const approvalTopic = new sns.Topic(this, 'CollectionApprovalTopic', { + topicName: `Automation-${cdk.Stack.of(this).stackName}-approvals`, displayName: 'EKS Diagnostics MCP - collection approvals', masterKey: props.enableEncryption === false ? undefined : this.encryptionKey, }); @@ -921,57 +1007,317 @@ export class SsmAutomationGatewayV2Construct extends Construct { } this.collectionApprovalTopic = approvalTopic; - // Approve/deny endpoint. Public Function URL (authType NONE) whose security - // is the one-time high-entropy token in the link (a capability URL) - the - // agent never receives that token, so it cannot approve its own requests. - const approvalHandlerRole = new iam.Role(this, 'ApprovalHandlerRole', { - assumedBy: new iam.ServicePrincipal('lambda.amazonaws.com'), - managedPolicies: [ - iam.ManagedPolicy.fromAwsManagedPolicyName('service-role/AWSLambdaBasicExecutionRole'), - ], - }); - approvalTable.grantReadWriteData(approvalHandlerRole); + // The wrapper automation runs as the SSM automation role: the aws:approve + // step publishes its notification to the topic, and the batch fan-out step + // starts child automations passing the same role. + approvalTopic.grantPublish(this.ssmAutomationRole); + this.ssmAutomationRole.addToPolicy(new iam.PolicyStatement({ + sid: 'PassSelfToChildAutomations', + effect: iam.Effect.ALLOW, + actions: ['iam:PassRole'], + resources: [this.ssmAutomationRole.roleArn], + conditions: { + StringEquals: { + 'iam:PassedToService': 'ssm.amazonaws.com', + }, + }, + })); - const approvalHandlerFunction = new lambda.Function(this, 'ApprovalHandlerFunction', { - functionName: `${cdk.Stack.of(this).stackName}-collection-approval`, - runtime: lambda.Runtime.PYTHON_3_11, - handler: 'index.handler', - role: approvalHandlerRole, - timeout: cdk.Duration.seconds(30), - memorySize: 128, - environment: { - APPROVAL_TABLE_NAME: approvalTable.tableName, + // The MCP Lambda publishes a rich notification (with the console deep link) + // in addition to the bare one the aws:approve step sends. + approvalTopic.grantPublish(lambdaExecutionRole); + + // Wrapper: approve → collect (single instance). + const collectApprovalDocName = `${cdk.Stack.of(this).stackName}-collect-with-approval`; + new ssm.CfnDocument(this, 'CollectApprovalDocument', { + name: collectApprovalDocName, + documentType: 'Automation', + updateMethod: 'NewVersion', + content: { + schemaVersion: '0.3', + description: + 'Human-in-the-loop EKS log collection: pauses at a native aws:approve step ' + + 'until a designated approver approves in the Systems Manager console, then ' + + 'runs AWSSupport-CollectEKSInstanceLogs on the target node.', + assumeRole: '{{ AutomationAssumeRole }}', + parameters: { + EKSInstanceId: { + type: 'String', + description: 'EC2 instance ID of the EKS worker node', + allowedPattern: '^i-[0-9a-f]{8,17}$', + }, + LogDestination: { + type: 'String', + description: 'S3 bucket that receives the log bundle', + }, + AutomationAssumeRole: { + type: 'String', + description: 'Role the automation (and child automation) runs as', + }, + Approvers: { + type: 'StringList', + description: 'IAM principals allowed to approve this collection', + }, + SNSTopicArn: { + type: 'String', + description: 'Topic notified when approval is requested', + }, + }, + mainSteps: [ + { + name: 'waitForHumanApproval', + action: 'aws:approve', + timeoutSeconds: approvalTtlSeconds, + onFailure: 'Abort', + inputs: { + NotificationArn: '{{ SNSTopicArn }}', + Message: + 'An MCP agent requested EKS log collection on instance ' + + '{{ EKSInstanceId }}. Approving runs AWSSupport-CollectEKSInstanceLogs ' + + 'on that node and uploads the bundle to {{ LogDestination }}. ' + + 'Approve or deny this execution in the Systems Manager console ' + + '(Automation → Executions) or via ssm send-automation-signal.', + MinRequiredApprovals: 1, + Approvers: '{{ Approvers }}', + }, + }, + { + name: 'collectLogs', + action: 'aws:executeAutomation', + inputs: { + DocumentName: 'AWSSupport-CollectEKSInstanceLogs', + RuntimeParameters: { + EKSInstanceId: '{{ EKSInstanceId }}', + LogDestination: '{{ LogDestination }}', + AutomationAssumeRole: '{{ AutomationAssumeRole }}', + }, + }, + }, + ], }, - code: lambda.Code.fromInline(this.getApprovalHandlerCode()), }); - // Public URL is opt-in (see approvalViaPublicUrl). Default is the - // IAM-authenticated direct-invoke path, which needs no resource policy - // and therefore survives account guardrails that block public Lambdas. - const approvalViaPublicUrl = props.approvalViaPublicUrl ?? false; - let approvalBaseUrl = ''; - if (approvalViaPublicUrl) { - const approvalFunctionUrl = approvalHandlerFunction.addFunctionUrl({ - authType: lambda.FunctionUrlAuthType.NONE, - }); - approvalBaseUrl = approvalFunctionUrl.url; - new cdk.CfnOutput(this, 'CollectionApprovalUrl', { - description: 'Function URL that approvers use to approve/deny collection requests', - value: approvalFunctionUrl.url, - }); - } - // Main Lambda needs to create/read/consume approvals and notify approvers. - approvalTable.grantReadWriteData(lambdaExecutionRole); - approvalTopic.grantPublish(lambdaExecutionRole); + // Wrapper: approve once → fan out one collection per sampled node. A single + // human approval authorizes the whole batch; the fan-out step emits + // "instanceId|executionId" pairs the Lambda's batch_status tool resolves. + const batchApprovalDocName = `${cdk.Stack.of(this).stackName}-batch-collect-with-approval`; + new ssm.CfnDocument(this, 'BatchCollectApprovalDocument', { + name: batchApprovalDocName, + documentType: 'Automation', + updateMethod: 'NewVersion', + content: { + schemaVersion: '0.3', + description: + 'Human-in-the-loop batch EKS log collection: one aws:approve step authorizes ' + + 'the whole batch, then a fan-out step starts one ' + + 'AWSSupport-CollectEKSInstanceLogs execution per node.', + assumeRole: '{{ AutomationAssumeRole }}', + parameters: { + InstanceIds: { + type: 'StringList', + description: 'EC2 instance IDs of the sampled EKS worker nodes (max 15)', + }, + LogDestination: { + type: 'String', + description: 'S3 bucket that receives the log bundles', + }, + AutomationAssumeRole: { + type: 'String', + description: 'Role the automation (and child automations) runs as', + }, + Approvers: { + type: 'StringList', + description: 'IAM principals allowed to approve this batch', + }, + SNSTopicArn: { + type: 'String', + description: 'Topic notified when approval is requested', + }, + }, + mainSteps: [ + { + name: 'waitForHumanApproval', + action: 'aws:approve', + timeoutSeconds: approvalTtlSeconds, + onFailure: 'Abort', + inputs: { + NotificationArn: '{{ SNSTopicArn }}', + // NOTE: SSM cannot substitute StringList parameters (InstanceIds) + // inside a message string — the execution's Parameters section in + // the console shows the exact node list instead. + Message: + 'An MCP agent requested BATCH EKS log collection. Approving runs ' + + 'AWSSupport-CollectEKSInstanceLogs on every node in this execution\'s ' + + 'InstanceIds parameter (visible on the execution page) and uploads ' + + 'bundles to {{ LogDestination }}. Approve or deny this execution in ' + + 'the Systems Manager console (Automation → Executions) or via ' + + 'ssm send-automation-signal.', + MinRequiredApprovals: 1, + Approvers: '{{ Approvers }}', + }, + }, + { + name: 'fanOutCollections', + action: 'aws:executeScript', + timeoutSeconds: 600, + inputs: { + Runtime: 'python3.11', + Handler: 'handler', + InputPayload: { + InstanceIds: '{{ InstanceIds }}', + LogDestination: '{{ LogDestination }}', + AutomationAssumeRole: '{{ AutomationAssumeRole }}', + }, + Script: [ + 'import boto3', + '', + '', + 'def handler(events, context):', + ' ssm = boto3.client("ssm")', + ' executions, errors = [], []', + ' for iid in events["InstanceIds"][:15]:', + ' try:', + ' resp = ssm.start_automation_execution(', + ' DocumentName="AWSSupport-CollectEKSInstanceLogs",', + ' Parameters={', + ' "EKSInstanceId": [iid],', + ' "LogDestination": [events["LogDestination"]],', + ' "AutomationAssumeRole": [events["AutomationAssumeRole"]],', + ' },', + ' )', + ' executions.append(f"{iid}|{resp[\'AutomationExecutionId\']}")', + ' except Exception as e:', + ' errors.append(f"{iid}|{e}")', + ' return {"executions": executions, "errors": errors}', + ].join('\n'), + }, + outputs: [ + { Name: 'Executions', Selector: '$.Payload.executions', Type: 'StringList' }, + { Name: 'Errors', Selector: '$.Payload.errors', Type: 'StringList' }, + ], + }, + ], + }, + }); - new cdk.CfnOutput(this, 'CollectionApprovalFunctionName', { - description: 'Approval handler Lambda. Approvers invoke it directly (IAM-authenticated) using the command from the SNS notification', - value: approvalHandlerFunction.functionName, + // Wrapper: approve → tcpdump packet capture (single instance, via + // AWS-RunShellScript). The capture script is built and validated by the + // MCP Lambda (BPF-filter allowlist, interface/pod-name/PID pattern checks) + // and passed through the Commands parameter, where the approver can review + // it verbatim on the execution page before approving. The Lambda has no + // ssm:SendCommand of its own in approval mode — this wrapper (running as + // the ssmAutomationRole) is the only way a capture reaches a node, and it + // never proceeds past the aws:approve step without a human decision. + const tcpdumpApprovalDocName = `${cdk.Stack.of(this).stackName}-tcpdump-with-approval`; + new ssm.CfnDocument(this, 'TcpdumpApprovalDocument', { + name: tcpdumpApprovalDocName, + documentType: 'Automation', + updateMethod: 'NewVersion', + content: { + schemaVersion: '0.3', + description: + 'Human-in-the-loop tcpdump packet capture: pauses at a native aws:approve ' + + 'step until a designated approver approves in the Systems Manager console, ' + + 'then sends the capture script to the target EKS node via AWS-RunShellScript.', + assumeRole: '{{ AutomationAssumeRole }}', + parameters: { + InstanceId: { + type: 'String', + description: 'EC2 instance ID of the EKS worker node', + allowedPattern: '^i-[0-9a-f]{8,17}$', + }, + Commands: { + type: 'String', + description: + 'Capture script built and input-validated by the MCP Lambda. ' + + 'Approvers: review this parameter on the execution page before approving.', + }, + ExecutionTimeoutSeconds: { + type: 'String', + description: 'Run Command execution timeout (capture duration + buffer)', + allowedPattern: '^\\d{2,4}$', + default: '240', + }, + DurationSeconds: { + type: 'String', + description: 'Capture duration in seconds (shown to approvers)', + allowedPattern: '^\\d{1,3}$', + }, + Interface: { + type: 'String', + description: 'Network interface captured (shown to approvers)', + allowedPattern: '^[a-zA-Z0-9\\-\\.]+$', + }, + BpfFilter: { + type: 'String', + description: 'BPF filter expression, or "none" (shown to approvers)', + allowedPattern: '^[^\\r\\n]{1,256}$', + }, + CaptureScope: { + type: 'String', + description: 'host, pod//, or container/PID- (shown to approvers)', + allowedPattern: '^[^\\r\\n]{1,320}$', + }, + AutomationAssumeRole: { + type: 'String', + description: 'Role the automation runs as', + }, + Approvers: { + type: 'StringList', + description: 'IAM principals allowed to approve this capture', + }, + SNSTopicArn: { + type: 'String', + description: 'Topic notified when approval is requested', + }, + }, + mainSteps: [ + { + name: 'waitForHumanApproval', + action: 'aws:approve', + timeoutSeconds: approvalTtlSeconds, + onFailure: 'Abort', + inputs: { + NotificationArn: '{{ SNSTopicArn }}', + Message: + 'An MCP agent requested a tcpdump PACKET CAPTURE on instance ' + + '{{ InstanceId }} — scope: {{ CaptureScope }}, interface ' + + '{{ Interface }}, duration {{ DurationSeconds }}s, BPF filter: ' + + '{{ BpfFilter }}. Approving runs the capture script (this ' + + 'execution\'s Commands parameter — review it before approving) as ' + + 'root on that node via AWS-RunShellScript and uploads the pcap to ' + + 'S3. Captured packets may contain sensitive payloads. Approve or ' + + 'deny this execution in the Systems Manager console (Automation → ' + + 'Executions) or via ssm send-automation-signal.', + MinRequiredApprovals: 1, + Approvers: '{{ Approvers }}', + }, + }, + { + name: 'runTcpdump', + action: 'aws:runCommand', + inputs: { + DocumentName: 'AWS-RunShellScript', + InstanceIds: ['{{ InstanceId }}'], + Parameters: { + commands: ['{{ Commands }}'], + executionTimeout: ['{{ ExecutionTimeoutSeconds }}'], + }, + Comment: 'tcpdump capture (human-approved) for {{ InstanceId }}', + }, + }, + ], + }, }); + new cdk.CfnOutput(this, 'CollectionApprovalTopicArn', { description: 'SNS topic that notifies approvers of collection requests', value: approvalTopic.topicArn, }); + new cdk.CfnOutput(this, 'CollectionApprovalDocuments', { + description: 'Approval wrapper SSM documents (approve in Systems Manager console → Automation → Executions)', + value: `${collectApprovalDocName}, ${batchApprovalDocName}, ${tcpdumpApprovalDocName}`, + }); this.ssmAutomationFunction = new lambda.Function(this, 'SSMAutomationFunction', { functionName: `${cdk.Stack.of(this).stackName}-ssm-automation`, @@ -992,13 +1338,23 @@ export class SsmAutomationGatewayV2Construct extends Construct { TOOL_AUTHORIZATION: toolAclSerialized, PER_CALLER_RATE_LIMIT_PER_MINUTE: String(props.perCallerRateLimitPerMinute ?? 60), ALLOWED_CLUSTER_NAMES: (props.allowedClusterNames ?? []).join(','), + // Fail-closed companion (E2): with no allowlist, the Lambda rejects all + // clusters unless the operator acknowledged any-cluster scope. + ALLOW_ANY_CLUSTER_NAME: String(props.allowAnyClusterName ?? false), ALLOW_SELF_MANAGED_NODES: String(props.allowSelfManagedNodes ?? false), REQUIRE_COLLECTION_APPROVAL: String(requireCollectionApproval), - APPROVAL_TABLE_NAME: approvalTable.tableName, APPROVAL_TOPIC_ARN: approvalTopic.topicArn, - APPROVAL_BASE_URL: approvalBaseUrl, - APPROVAL_FUNCTION_NAME: approvalHandlerFunction.functionName, - APPROVAL_TTL_SECONDS: String(approvalTtlSeconds), + COLLECT_APPROVAL_DOCUMENT: collectApprovalDocName, + BATCH_APPROVAL_DOCUMENT: batchApprovalDocName, + TCPDUMP_APPROVAL_DOCUMENT: tcpdumpApprovalDocName, + APPROVAL_APPROVERS: approverArns.join(','), + ENABLED_RESTRICTED_TOOLS: enabledRestrictedTools.join(','), + PCAP_PRESIGNED_URL_EXPIRATION_SECONDS: String(props.pcapPresignedUrlExpirationSeconds ?? 60), + MAX_PCAP_BYTES: String(props.maxPcapBytes ?? 209715200), + // Lets the Lambda tell the agent honestly whether anyone actually + // receives the SNS notification (email subscriptions created at deploy + // time). Out-of-band subscriptions added later aren't reflected here. + APPROVAL_EMAILS_CONFIGURED: String((props.approvalNotificationEmails ?? []).length > 0), STACK_NAME: cdk.Stack.of(this).stackName, }, code: lambda.Code.fromAsset(path.join(__dirname, 'lambda')), @@ -1176,7 +1532,7 @@ export class SsmAutomationGatewayV2Construct extends Construct { Lambda: { LambdaArn: this.ssmAutomationFunction.functionArn, ToolSchema: { - InlinePayload: this.getToolSchemaDefinitions(), + InlinePayload: this.getToolSchemaDefinitions(enabledRestrictedTools), }, }, }, @@ -1314,18 +1670,21 @@ export class SsmAutomationGatewayV2Construct extends Construct { /** * Returns the enhanced tool schema definitions for the MCP Gateway. * collect/batch_collect are mutating (they start SSM Automation on nodes) and - * are gated at runtime by a human approval when requireCollectionApproval is - * on (M1/M2): the first call returns status="pending_approval" + an approvalId, - * and the caller re-invokes with that approvalId once a human approves. + * are gated at runtime by a native SSM aws:approve step when + * requireCollectionApproval is on (M1/M2): the call returns + * status="pending_approval" with an SSM console link, a human approves there, + * and the collection proceeds automatically — the agent just polls status. + * Restricted tools (tcpdump) are only included when explicitly enabled, and + * tcpdump_capture is approval-gated the same way (M3). */ - private getToolSchemaDefinitions(): object[] { + private getToolSchemaDefinitions(enabledRestrictedTools: string[]): object[] { const tools: object[] = [ // ===================================================================== // TIER 1: CORE OPERATIONS // ===================================================================== { Name: 'collect', - Description: 'Start EKS log collection from a worker node. NOTE: collection is a mutating action and may require human approval — if the response has status="pending_approval", a human must approve the request out-of-band, then you re-call collect with the SAME instanceId plus the returned approvalId. Returns executionId for async polling once approved. Recommended workflow: collect → status (poll until complete) → quick_triage (ONE call for full analysis). Do NOT read individual files unless quick_triage/summarize/network_diagnostics are insufficient. Supports cross-region. CITATION: cite the executionId and region returned.', + Description: 'Start EKS log collection from a worker node. NOTE: collection is a mutating action and may require human approval — if the response has status="pending_approval", the SSM Automation execution is PAUSED at a native aws:approve step. Show the user the approvalConsoleUrl, then poll status(executionId) every 30 seconds up to 10 attempts WITHOUT waiting for the user to confirm — collection continues automatically after approval; do NOT re-call collect. If still pending after 10 polls, stop and ask the user to get it approved. Recommended workflow: collect → status (poll until complete) → quick_triage (ONE call for full analysis). Do NOT read individual files unless quick_triage/summarize/network_diagnostics are insufficient. Supports cross-region. CITATION: cite the executionId and region returned.', InputSchema: { Type: 'object', Properties: { @@ -1341,10 +1700,6 @@ export class SsmAutomationGatewayV2Construct extends Construct { Type: 'string', Description: 'AWS region where the instance runs (e.g., us-west-2). Optional: auto-detected from instance if omitted.', }, - approvalId: { - Type: 'string', - Description: 'Approval ID returned by a prior collect call when status was "pending_approval". Pass it (with the same instanceId) after a human approves, to run the collection. Do not fabricate this value.', - }, }, Required: ['instanceId'], }, @@ -1521,7 +1876,7 @@ export class SsmAutomationGatewayV2Construct extends Construct { }, instanceId: { Type: 'string', - Description: 'The EC2 instance ID under investigation. Recommended: when provided, the logKey must belong to this instance, preventing reads of another instance\u2019s logs.', + Description: 'The EC2 instance ID under investigation. Required: the logKey must belong to this instance, preventing reads of another instance\u2019s logs.', }, startByte: { Type: 'integer', @@ -1540,7 +1895,7 @@ export class SsmAutomationGatewayV2Construct extends Construct { Description: 'Number of lines to return when using startLine (default: 1000)', }, }, - Required: ['logKey'], + Required: ['logKey', 'instanceId'], }, OutputSchema: { Type: 'object', @@ -1686,14 +2041,14 @@ export class SsmAutomationGatewayV2Construct extends Construct { }, instanceId: { Type: 'string', - Description: 'The EC2 instance ID under investigation. Recommended: when provided, the logKey must belong to this instance, preventing presigned URLs for another instance\u2019s data.', + Description: 'The EC2 instance ID under investigation. Required: the logKey must belong to this instance, preventing presigned URLs for another instance\u2019s data.', }, expirationMinutes: { Type: 'integer', Description: 'URL expiration in minutes (default: 15, max: 60)', }, }, - Required: ['logKey'], + Required: ['logKey', 'instanceId'], }, OutputSchema: { Type: 'object', @@ -1905,7 +2260,7 @@ export class SsmAutomationGatewayV2Construct extends Construct { }, { Name: 'batch_collect', - Description: 'Smart batch log collection with statistical sampling. Triages all nodes in a cluster, groups unhealthy nodes into buckets by failure signature, and collects from representative samples. Handles 1000+ node clusters efficiently. Defaults to a DRY RUN (preview only); pass dryRun=false to actually collect. Real collection is mutating and may require human approval — if the response has status="pending_approval", a human approves out-of-band and you re-call with dryRun=false plus the returned approvalId. CITATION: Cite batchId, node count, and sampling strategy used.', + Description: 'Smart batch log collection with statistical sampling. Triages all nodes in a cluster, groups unhealthy nodes into buckets by failure signature, and collects from representative samples. Handles 1000+ node clusters efficiently. Defaults to a DRY RUN (preview only); pass dryRun=false to actually collect. Real collection is mutating and may require human approval — if the response has status="pending_approval", the batch is PAUSED at a native SSM aws:approve step. Show the user the approvalConsoleUrl, then poll batch_status(batchId) every 30 seconds up to 10 attempts WITHOUT waiting for the user to confirm — the fan-out happens automatically after approval; do NOT re-call batch_collect. If still pending after 10 polls, stop and ask the user to get it approved. CITATION: Cite batchId, node count, and sampling strategy used.', InputSchema: { Type: 'object', Properties: { @@ -1941,10 +2296,6 @@ export class SsmAutomationGatewayV2Construct extends Construct { Type: 'boolean', Description: 'Preview which nodes would be collected without starting (default: true). Set false to actually collect.', }, - approvalId: { - Type: 'string', - Description: 'Approval ID returned by a prior batch_collect(dryRun=false) call when status was "pending_approval". Pass it (with dryRun=false and the same clusterName) after a human approves. Do not fabricate this value.', - }, }, Required: ['clusterName'], }, @@ -2099,6 +2450,139 @@ export class SsmAutomationGatewayV2Construct extends Construct { }, ]; + // Only register tcpdump tools in the MCP schema when explicitly enabled. + // By default these are completely absent — the agent doesn't know they exist. + if (enabledRestrictedTools.includes('tcpdump_capture')) { + tools.push({ + Name: 'tcpdump_capture', + Description: 'Run tcpdump on an EKS worker node for a specified duration (default 2 minutes), then upload the pcap file to S3. Supports capturing inside a pod/container network namespace — provide podName (auto-resolves PID via crictl/docker) or containerPid (raw PID). For K8s DNS debugging: tcpdump_capture(instanceId, podName="coredns-xxx", podNamespace="kube-system", filter="udp port 53"). Requires confirmCapture=true. NOTE: packet capture is a mutating action and may require human approval — if the response has status="pending_approval", the SSM Automation execution is PAUSED at a native aws:approve step. Show the user the approvalConsoleUrl, then poll tcpdump_capture(executionId, instanceId, confirmCapture=true) every 30 seconds up to 10 attempts WITHOUT waiting for the user to confirm — the capture starts automatically after approval; do NOT submit a new capture request. If still pending after 10 polls, stop and ask the user to get it approved. Once running, poll with commandId (or keep polling executionId) until completed, then use tcpdump_analyze. CITATION: Cite commandId (or executionId), instanceId, and s3Key.', + InputSchema: { + Type: 'object', + Properties: { + instanceId: { + Type: 'string', + Description: 'The EC2 instance ID of the EKS worker node (e.g., i-0123456789abcdef0)', + }, + durationSeconds: { + Type: 'integer', + Description: 'Capture duration in seconds (default: 120, min: 10, max: 300)', + }, + interface: { + Type: 'string', + Description: 'Network interface to capture on (default: "any"). Use "eth0", "eni+", etc.', + }, + filter: { + Type: 'string', + Description: 'BPF filter expression (e.g., "port 443", "host 10.0.0.1 and port 80", "udp port 53")', + }, + podName: { + Type: 'string', + Description: 'Kubernetes pod name to capture from (e.g., "coredns-5d78c9869d-abc12"). Auto-resolves to container PID via crictl/docker on the worker node. Pod must be running on the specified instanceId.', + }, + podNamespace: { + Type: 'string', + Description: 'Kubernetes namespace of the pod (default: "default"). Use "kube-system" for CoreDNS, "amazon-vpc-cni" for VPC CNI pods, etc.', + }, + containerPid: { + Type: 'string', + Description: 'Raw container PID for nsenter (alternative to podName). Use when you already know the PID from "ps ax | grep " on the worker node.', + }, + confirmCapture: { + Type: 'boolean', + Description: 'Must be set to true to confirm the capture. Without this, the tool returns a description of what will happen and asks for confirmation.', + }, + commandId: { + Type: 'string', + Description: 'SSM Command ID from a previous tcpdump_capture call — pass this to poll capture status once it is running', + }, + executionId: { + Type: 'string', + Description: 'SSM Automation execution ID from a pending_approval response — pass this to poll approval status; the capture starts automatically once a human approves', + }, + region: { + Type: 'string', + Description: 'AWS region where the instance runs (optional, auto-detected)', + }, + }, + Required: ['instanceId'], + }, + OutputSchema: { + Type: 'object', + Properties: { + commandId: { Type: 'string', Description: 'SSM Run Command ID for polling (present once the capture is running)' }, + executionId: { Type: 'string', Description: 'SSM Automation execution ID (approval-gated captures)' }, + approvalConsoleUrl: { Type: 'string', Description: 'SSM console link where an approver approves/denies the capture' }, + humanApproval: { Type: 'object', Description: 'state: pending | approved | denied_or_expired, consoleUrl' }, + instanceId: { Type: 'string' }, + status: { Type: 'string', Description: 'in_progress | completed | failed' }, + s3Key: { Type: 'string', Description: 'S3 key of the uploaded pcap file' }, + s3Bucket: { Type: 'string' }, + fileSizeBytes: { Type: 'integer' }, + fileSizeHuman: { Type: 'string' }, + presignedUrl: { Type: 'string', Description: 'Presigned download URL' }, + task: { + Type: 'object', + Description: 'Async task envelope for polling', + Properties: { + taskId: { Type: 'string', Description: 'Same as commandId' }, + state: { Type: 'string', Description: 'running|completed|failed' }, + message: { Type: 'string' }, + progress: { Type: 'integer', Description: '0-100 percent' }, + }, + }, + }, + }, + }); + } + + if (enabledRestrictedTools.includes('tcpdump_analyze')) { + tools.push({ + Name: 'tcpdump_analyze', + Description: 'Read and analyze a completed tcpdump capture from S3. Returns decoded packet text (human-readable), protocol statistics (TCP/UDP/ICMP breakdown), top source/destination IPs, and anomaly detection (high RST rate, retransmissions, SYN floods). Use after tcpdump_capture completes. Supports text filtering to search for specific IPs, ports, or flags in the decoded output. CITATION: Cite commandId, packet count, and any anomalies found.', + InputSchema: { + Type: 'object', + Properties: { + instanceId: { + Type: 'string', + Description: 'The EC2 instance ID (e.g., i-0123456789abcdef0)', + }, + commandId: { + Type: 'string', + Description: 'SSM Command ID from tcpdump_capture. If omitted, returns the latest capture for this instance.', + }, + section: { + Type: 'string', + Description: '"summary" (decoded packets), "stats" (protocol breakdown + anomalies), "all" (default: "all")', + }, + maxPackets: { + Type: 'integer', + Description: 'Max decoded packet lines to return (default: 500, max: 3000)', + }, + filter: { + Type: 'string', + Description: 'Text filter on decoded lines (e.g., "SYN", "RST", "10.0.0.5", "port 443")', + }, + }, + Required: ['instanceId'], + }, + OutputSchema: { + Type: 'object', + Properties: { + instanceId: { Type: 'string' }, + commandId: { Type: 'string' }, + captureInfo: { Type: 'object', Description: 'interface, filter, duration, startedAt' }, + statistics: { Type: 'object', Description: 'totalPackets, protocols (tcp/udp/icmp/arp), ports (dns/http/https), tcpFlags (syn/rst), topSourceIPs, topDestinationIPs' }, + anomalies: { Type: 'array', Description: 'Detected anomalies: high_rst_rate, retransmissions, syn_rst_ratio, high_icmp' }, + decodedPackets: { Type: 'object', Description: 'lines (array of decoded packet strings), totalPackets, returnedPackets, truncated, filter' }, + pcapDownloadUrl: { Type: 'string', Description: 'Presigned URL to download the raw pcap file' }, + s3KeyPcap: { Type: 'string' }, + s3KeyTxt: { Type: 'string' }, + s3KeyStats: { Type: 'string' }, + }, + }, + }); + } + return tools; } @@ -2118,97 +2602,6 @@ export class SsmAutomationGatewayV2Construct extends Construct { * updates the DynamoDB record; the main Lambda performs the actual * collection after the caller re-invokes. */ - private getApprovalHandlerCode(): string { - return ` -import json -import os -import time -import hashlib -import hmac -import boto3 - -TABLE = os.environ['APPROVAL_TABLE_NAME'] -ddb = boto3.client('dynamodb') - - -def _respond(is_http, status_code, title, body): - if is_http: - return { - 'statusCode': status_code, - 'headers': {'Content-Type': 'text/html; charset=utf-8'}, - 'body': f'' - f'

{title}

{body}

', - } - return {'statusCode': status_code, 'result': title, 'message': body} - - -def handler(event, context): - # Function URL events carry requestContext/queryStringParameters; a direct - # IAM-authenticated invoke passes the payload as the event itself. - is_http = isinstance(event, dict) and 'requestContext' in event - if is_http: - params = event.get('queryStringParameters') or {} - else: - params = event if isinstance(event, dict) else {} - - approval_id = str(params.get('approvalId', '') or '') - token = str(params.get('token', '') or '') - decision = str(params.get('decision', '') or '').lower() - - if not approval_id or not token or decision not in ('approve', 'deny'): - return _respond(is_http, 400, 'Invalid request', 'Missing approvalId, token, or a valid decision (approve/deny).') - - try: - resp = ddb.get_item(TableName=TABLE, Key={'approvalId': {'S': approval_id}}) - except Exception as e: - return _respond(is_http, 500, 'Error', f'Could not look up the request: {e}') - - item = resp.get('Item') - if not item: - return _respond(is_http, 404, 'Not found', 'This approval request does not exist or has expired.') - - # Constant-time comparison of the token hash. - token_hash = hashlib.sha256(token.encode('utf-8')).hexdigest() - stored_hash = item.get('tokenHash', {}).get('S', '') - if not stored_hash or not hmac.compare_digest(token_hash, stored_hash): - return _respond(is_http, 403, 'Forbidden', 'Invalid approval token.') - - ttl = int(item.get('ttl', {}).get('N', '0') or 0) - if ttl and int(time.time()) > ttl: - return _respond(is_http, 403, 'Expired', 'This approval request has expired. Ask the agent to request collection again.') - - status = item.get('status', {}).get('S', '') - if status != 'PENDING': - return _respond(is_http, 409, 'Already decided', f'This request is already {status}. No further action taken.') - - new_status = 'APPROVED' if decision == 'approve' else 'DENIED' - try: - ddb.update_item( - TableName=TABLE, - Key={'approvalId': {'S': approval_id}}, - UpdateExpression='SET #s = :new, decidedAt = :t', - ConditionExpression='#s = :pending', - ExpressionAttributeNames={'#s': 'status'}, - ExpressionAttributeValues={ - ':new': {'S': new_status}, - ':pending': {'S': 'PENDING'}, - ':t': {'N': str(int(time.time()))}, - }, - ) - except ddb.exceptions.ConditionalCheckFailedException: - return _respond(is_http, 409, 'Already decided', 'This request was just decided by someone else.') - except Exception as e: - return _respond(is_http, 500, 'Error', f'Could not record the decision: {e}') - - tool = item.get('tool', {}).get('S', 'collect') - target = item.get('target', {}).get('S', '') - if new_status == 'APPROVED': - return _respond(is_http, 200, 'Approved', - f'{tool} on {target} is approved. The agent can now proceed with the collection.') - return _respond(is_http, 200, 'Denied', f'{tool} on {target} was denied. No collection will run.') -`; - } - private getUnzipLambdaCode(): string { return ` import json diff --git a/mcp/aws-eks-node-diagnostics-mcp/src/ssm-automation-gateway-stack-v2.ts b/mcp/aws-eks-node-diagnostics-mcp/src/ssm-automation-gateway-stack-v2.ts index 27cef36..bb0abbe 100644 --- a/mcp/aws-eks-node-diagnostics-mcp/src/ssm-automation-gateway-stack-v2.ts +++ b/mcp/aws-eks-node-diagnostics-mcp/src/ssm-automation-gateway-stack-v2.ts @@ -43,12 +43,16 @@ export class EksNodeLogMcpStack extends cdk.Stack { presignedUrlExpirationSeconds: props.presignedUrlExpirationSeconds, allowSelfManagedNodes: props.allowSelfManagedNodes, requireCollectionApproval: props.requireCollectionApproval, + approvalApproverArns: props.approvalApproverArns, approvalNotificationEmails: props.approvalNotificationEmails, approvalTtlSeconds: props.approvalTtlSeconds, vpcId: props.vpcId, vpcSubnetIds: props.vpcSubnetIds, toolAuthorization: props.toolAuthorization, perCallerRateLimitPerMinute: props.perCallerRateLimitPerMinute, + enableRestrictedTools: props.enableRestrictedTools, + pcapPresignedUrlExpirationSeconds: props.pcapPresignedUrlExpirationSeconds, + maxPcapBytes: props.maxPcapBytes, }); } } diff --git a/mcp/aws-eks-node-diagnostics-mcp/tests/construct-iam.property.test.ts b/mcp/aws-eks-node-diagnostics-mcp/tests/construct-iam.property.test.ts index 059fee8..63553fe 100644 --- a/mcp/aws-eks-node-diagnostics-mcp/tests/construct-iam.property.test.ts +++ b/mcp/aws-eks-node-diagnostics-mcp/tests/construct-iam.property.test.ts @@ -20,7 +20,13 @@ function synthesize(allowedRegions?: string[]) { const stack = new cdk.Stack(app, 'TestStack', { env: { region: 'us-east-1', account: '123456789012' } }); // allowAnyClusterName: tests rely on the wildcard cluster scope, so opt in // to keep them deterministic. Production deploys should set allowedClusterNames. - new SsmAutomationGatewayV2Construct(stack, 'Gateway', { allowedRegions, allowAnyClusterName: true }); + // approvalApproverArns: required when approval is on (the default) — synth + // fails closed without designated approvers. + new SsmAutomationGatewayV2Construct(stack, 'Gateway', { + allowedRegions, + allowAnyClusterName: true, + approvalApproverArns: ['arn:aws:iam::123456789012:role/CollectionApprover'], + }); return Template.fromStack(stack); } diff --git a/mcp/aws-eks-node-diagnostics-mcp/tests/construct-kms-s3.property.test.ts b/mcp/aws-eks-node-diagnostics-mcp/tests/construct-kms-s3.property.test.ts index 5202871..dfb8c34 100644 --- a/mcp/aws-eks-node-diagnostics-mcp/tests/construct-kms-s3.property.test.ts +++ b/mcp/aws-eks-node-diagnostics-mcp/tests/construct-kms-s3.property.test.ts @@ -29,7 +29,13 @@ function synthesize(props: { const stack = new cdk.Stack(app, 'TestStack', { env: { region: 'us-east-1', account: '123456789012' } }); // allowAnyClusterName: tests rely on the wildcard cluster scope, so opt in // to keep them deterministic. Production deploys should set allowedClusterNames. - new SsmAutomationGatewayV2Construct(stack, 'Gateway', { ...props, allowAnyClusterName: true }); + // approvalApproverArns: required when approval is on (the default) — synth + // fails closed without designated approvers. + new SsmAutomationGatewayV2Construct(stack, 'Gateway', { + ...props, + allowAnyClusterName: true, + approvalApproverArns: ['arn:aws:iam::123456789012:role/CollectionApprover'], + }); return Template.fromStack(stack); } diff --git a/mcp/aws-eks-node-diagnostics-mcp/tests/test_collection_approval.py b/mcp/aws-eks-node-diagnostics-mcp/tests/test_collection_approval.py index e7eddc3..e641a11 100644 --- a/mcp/aws-eks-node-diagnostics-mcp/tests/test_collection_approval.py +++ b/mcp/aws-eks-node-diagnostics-mcp/tests/test_collection_approval.py @@ -1,7 +1,8 @@ """ -Unit tests for the human-in-the-loop collection approval gate (M1/M2) and the -E4/E5 hardening helpers. These cover the pure-logic paths that do not require -AWS calls (bypass, disabled, fail-closed, log-key scoping, regex safety). +Unit tests for the SSM-native human-in-the-loop collection approval (M1/M2) and +the E4/E5 hardening helpers. These cover the pure-logic paths that do not +require AWS calls (fail-closed preconditions, wrapper status augmentation, +log-key scoping, regex safety). """ import os import sys @@ -24,28 +25,105 @@ OTHER_INSTANCE = 'i-0fedcba9876543210' -class TestApprovalGate: - def test_bypass_flag_skips_approval(self): - """Server-side _approval_bypass short-circuits the gate (used by batch).""" - result = mod.enforce_collection_approval( - 'collect', INSTANCE, 'us-east-1', {'_approval_bypass': True} - ) - assert result is None - - def test_disabled_returns_none(self, monkeypatch): - """When approval is disabled, the gate lets the call proceed.""" - monkeypatch.setattr(mod, 'REQUIRE_COLLECTION_APPROVAL', False) - result = mod.enforce_collection_approval('collect', INSTANCE, 'us-east-1', {}) - assert result is None - - def test_required_but_unconfigured_fails_closed(self, monkeypatch): - """Approval required but no table configured -> 503, never runs unapproved.""" - monkeypatch.setattr(mod, 'REQUIRE_COLLECTION_APPROVAL', True) - monkeypatch.setattr(mod, 'APPROVAL_TABLE_NAME', '') - result = mod.enforce_collection_approval('collect', INSTANCE, 'us-east-1', {}) +class TestApprovalPreconditions: + def test_unconfigured_fails_closed(self, monkeypatch): + """Approval required but wrapper doc/approvers unset -> 503, never runs unapproved.""" + monkeypatch.setattr(mod, 'COLLECT_APPROVAL_DOCUMENT', '') + monkeypatch.setattr(mod, 'APPROVAL_APPROVERS', []) + result = mod.enforce_approval_preconditions('us-east-1') + assert result is not None + assert result['statusCode'] == 503 + + def test_no_approvers_fails_closed(self, monkeypatch): + """A wrapper document without designated approvers is not usable.""" + monkeypatch.setattr(mod, 'COLLECT_APPROVAL_DOCUMENT', 'stack-collect-with-approval') + monkeypatch.setattr(mod, 'APPROVAL_APPROVERS', []) + result = mod.enforce_approval_preconditions('us-east-1') assert result is not None assert result['statusCode'] == 503 + def test_cross_region_rejected(self, monkeypatch): + """The wrapper doc is regional — approval-gated collection is stack-region only.""" + monkeypatch.setattr(mod, 'COLLECT_APPROVAL_DOCUMENT', 'stack-collect-with-approval') + monkeypatch.setattr(mod, 'APPROVAL_APPROVERS', ['arn:aws:iam::123456789012:role/Approver']) + result = mod.enforce_approval_preconditions('us-west-2') + assert result is not None + assert result['statusCode'] == 400 + + def test_configured_stack_region_proceeds(self, monkeypatch): + monkeypatch.setattr(mod, 'COLLECT_APPROVAL_DOCUMENT', 'stack-collect-with-approval') + monkeypatch.setattr(mod, 'APPROVAL_APPROVERS', ['arn:aws:iam::123456789012:role/Approver']) + assert mod.enforce_approval_preconditions('us-east-1') is None + + +class TestConsoleUrl: + def test_deep_link_shape(self): + url = mod.console_automation_url('us-east-1', 'exec-123') + assert url == ( + 'https://us-east-1.console.aws.amazon.com/systems-manager/' + 'automation/execution/exec-123?region=us-east-1' + ) + + +def _wrapper_execution(approve_status, extra_steps=None): + steps = [{'StepName': 'waitForHumanApproval', 'StepStatus': approve_status}] + steps.extend(extra_steps or []) + return { + 'AutomationExecutionId': 'wrapper-1', + 'AutomationExecutionStatus': 'InProgress', + 'StepExecutions': steps, + } + + +class TestWrapperStatusAugmentation: + def test_waiting_reports_pending_with_console_url(self): + result = {'executionId': 'wrapper-1', 'task': {'message': ''}} + mod.augment_wrapper_status(_wrapper_execution('Waiting'), result, 'us-east-1') + assert result['humanApproval']['state'] == 'pending' + assert 'console.aws.amazon.com' in result['humanApproval']['consoleUrl'] + assert result['task']['message'].startswith('Waiting for human approval') + + def test_denied_or_timed_out_reports_denied(self): + result = {'executionId': 'wrapper-1', 'task': {'state': 'running', 'message': ''}} + mod.augment_wrapper_status(_wrapper_execution('TimedOut'), result, 'us-east-1') + assert result['humanApproval']['state'] == 'denied_or_expired' + assert result['task']['state'] == 'failed' + + def test_approved_exposes_child_execution(self): + collect_step = { + 'StepName': 'collectLogs', + 'StepStatus': 'InProgress', + 'Outputs': {'ExecutionId': ['child-42']}, + } + result = {'executionId': 'wrapper-1'} + mod.augment_wrapper_status( + _wrapper_execution('Success', [collect_step]), result, 'us-east-1' + ) + assert result['humanApproval']['state'] == 'approved' + assert result['childExecutionId'] == 'child-42' + + def test_approved_batch_exposes_children(self): + fanout_step = { + 'StepName': 'fanOutCollections', + 'StepStatus': 'Success', + 'Outputs': {'Executions': [f'{INSTANCE}|child-1', f'{OTHER_INSTANCE}|child-2']}, + } + result = {'executionId': 'wrapper-1'} + mod.augment_wrapper_status( + _wrapper_execution('Success', [fanout_step]), result, 'us-east-1' + ) + assert result['childExecutions'] == [ + {'instanceId': INSTANCE, 'executionId': 'child-1'}, + {'instanceId': OTHER_INSTANCE, 'executionId': 'child-2'}, + ] + + def test_non_wrapper_document_not_flagged(self, monkeypatch): + monkeypatch.setattr(mod, 'COLLECT_APPROVAL_DOCUMENT', 'stack-collect-with-approval') + monkeypatch.setattr(mod, 'BATCH_APPROVAL_DOCUMENT', 'stack-batch-collect-with-approval') + assert mod._is_approval_wrapper('AWSSupport-CollectEKSInstanceLogs') is False + assert mod._is_approval_wrapper('stack-collect-with-approval') is True + assert mod._is_approval_wrapper('') is False + class TestLogKeyScoping: def test_valid_key_no_instance(self): @@ -75,3 +153,94 @@ def test_flags_catastrophic(self, pattern): @pytest.mark.parametrize('pattern', ['ERROR|WARN', 'kubelet.*failed', r'\bOOMKilled\b']) def test_allows_normal(self, pattern): assert mod.is_catastrophic_regex(pattern) is False + + +class TestIdempotentReplayApproval: + """A retried collect() with the same idempotencyToken must carry the same + human-approval context that status() reports — not a bare InProgress.""" + + def _replay(self, monkeypatch, document_name): + monkeypatch.setattr(mod, 'COLLECT_APPROVAL_DOCUMENT', 'stack-collect-with-approval') + monkeypatch.setattr(mod, 'BATCH_APPROVAL_DOCUMENT', 'stack-batch-collect-with-approval') + monkeypatch.setattr(mod, 'resolve_and_validate_region', + lambda arguments, instance_id: ('us-east-1', None)) + monkeypatch.setattr(mod, 'validate_eks_instance', + lambda instance_id, region: None) + monkeypatch.setattr(mod, 'get_regional_client', + lambda service, region: object()) + monkeypatch.setattr( + mod, 'find_execution_by_idempotency_token', + lambda instance_id, token: { + 'executionId': 'wrapper-1', + 'status': 'InProgress', + 'documentName': document_name, + 'region': 'us-east-1', + '_execution': _wrapper_execution('Waiting'), + }, + ) + result = mod.start_log_collection( + {'instanceId': INSTANCE, 'idempotencyToken': 'tok-1'} + ) + assert result['statusCode'] == 200 + return json.loads(result['body']) + + def test_replay_of_pending_wrapper_reports_human_approval(self, monkeypatch): + body = self._replay(monkeypatch, 'stack-collect-with-approval') + assert body['idempotent'] is True + assert body['status'] == 'InProgress' + assert body['humanApproval']['state'] == 'pending' + assert 'console.aws.amazon.com' in body['humanApproval']['consoleUrl'] + assert 'nextStep' in body + + def test_replay_of_plain_collection_is_unchanged(self, monkeypatch): + body = self._replay(monkeypatch, 'AWSSupport-CollectEKSInstanceLogs') + assert body['idempotent'] is True + assert body['status'] == 'InProgress' + assert 'humanApproval' not in body + + +class TestClusterAllowlistFailClosed: + """E2 hardening: an empty allowlist only permits clusters when the operator + explicitly acknowledged any-cluster scope (ALLOW_ANY_CLUSTER_NAME).""" + + def test_empty_allowlist_without_acknowledgment_rejects(self, monkeypatch): + monkeypatch.setattr(mod, 'ALLOWED_CLUSTER_NAMES', set()) + monkeypatch.setattr(mod, 'ALLOW_ANY_CLUSTER_NAME', False) + result = mod.validate_cluster_name('prod-cluster') + assert result is not None and result['statusCode'] == 403 + + def test_empty_allowlist_with_acknowledgment_permits(self, monkeypatch): + monkeypatch.setattr(mod, 'ALLOWED_CLUSTER_NAMES', set()) + monkeypatch.setattr(mod, 'ALLOW_ANY_CLUSTER_NAME', True) + assert mod.validate_cluster_name('prod-cluster') is None + + def test_allowlisted_cluster_permits(self, monkeypatch): + monkeypatch.setattr(mod, 'ALLOWED_CLUSTER_NAMES', {'prod-cluster'}) + monkeypatch.setattr(mod, 'ALLOW_ANY_CLUSTER_NAME', False) + assert mod.validate_cluster_name('prod-cluster') is None + + def test_unlisted_cluster_rejected_even_with_acknowledgment(self, monkeypatch): + # An explicit allowlist always wins over the any-cluster acknowledgment. + monkeypatch.setattr(mod, 'ALLOWED_CLUSTER_NAMES', {'prod-cluster'}) + monkeypatch.setattr(mod, 'ALLOW_ANY_CLUSTER_NAME', True) + result = mod.validate_cluster_name('other-cluster') + assert result is not None and result['statusCode'] == 403 + + +class TestReadArtifactRequireInstanceId: + """E4 hardening: read()/artifact() must always be scoped to the instance + under investigation — omitting instanceId is rejected before any S3 call.""" + + def test_read_without_instance_id_rejected(self): + result = mod.read_log_chunk({'logKey': VALID_KEY}) + assert result['statusCode'] == 400 + assert 'instanceId is required' in result['body'] + + def test_artifact_without_instance_id_rejected(self): + result = mod.get_artifact_reference({'logKey': VALID_KEY}) + assert result['statusCode'] == 400 + assert 'instanceId is required' in result['body'] + + def test_read_cross_instance_key_still_rejected(self): + result = mod.read_log_chunk({'logKey': VALID_KEY, 'instanceId': OTHER_INSTANCE}) + assert result['statusCode'] == 403 diff --git a/mcp/aws-eks-node-diagnostics-mcp/tests/test_tcpdump_approval.py b/mcp/aws-eks-node-diagnostics-mcp/tests/test_tcpdump_approval.py new file mode 100644 index 0000000..aa56ccb --- /dev/null +++ b/mcp/aws-eks-node-diagnostics-mcp/tests/test_tcpdump_approval.py @@ -0,0 +1,262 @@ +""" +Unit tests for the restored tcpdump tools (M3): restricted-tool opt-in, +BPF filter validation, and the human-in-the-loop aws:approve gating of +tcpdump_capture. Pure-logic paths only — no AWS calls. +""" +import os +import sys +import json +import pytest + +os.environ.setdefault('LOGS_BUCKET_NAME', 'test-bucket') +os.environ.setdefault('SSM_AUTOMATION_ROLE_ARN', 'arn:aws:iam::123456789012:role/test') +os.environ.setdefault('AWS_REGION', 'us-east-1') +os.environ.setdefault('ALLOWED_REGIONS', 'us-east-1,us-west-2') +os.environ.setdefault('SOP_BUCKET_NAME', 'test-sop-bucket') + +sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src', 'lambda')) + +mod = __import__('ssm-automation-enhanced') + +INSTANCE = 'i-0123456789abcdef0' +APPROVER = 'arn:aws:iam::123456789012:role/Approver' + + +def _body(result): + return json.loads(result['body']) + + +class TestRestrictedToolGating: + def test_tcpdump_tools_are_restricted(self): + assert 'tcpdump_capture' in mod.RESTRICTED_TOOLS + assert 'tcpdump_analyze' in mod.RESTRICTED_TOOLS + + def test_restricted_tool_denied_unless_enabled(self, monkeypatch): + monkeypatch.setattr(mod, 'ENABLED_RESTRICTED_TOOLS', set()) + result = mod.validate_tool_authorization('tcpdump_capture') + assert result is not None + assert result['statusCode'] == 403 + + def test_restricted_tool_allowed_when_enabled(self, monkeypatch): + monkeypatch.setattr(mod, 'ENABLED_RESTRICTED_TOOLS', {'tcpdump_capture'}) + monkeypatch.setattr(mod, 'TOOL_AUTHORIZATION_ACL', {}) + assert mod.validate_tool_authorization('tcpdump_capture') is None + + +class TestBpfFilterValidation: + @pytest.mark.parametrize('expr', [ + '', 'port 53', 'udp port 53', 'host 10.0.0.1 and port 443', + 'net 10.0.0.0/16', 'src host 10.0.0.5 and dst port 443', + ]) + def test_safe_filters_accepted(self, expr): + assert mod.validate_bpf_filter(expr) is None + + @pytest.mark.parametrize('expr', [ + 'port 53; rm -rf /', 'port `id`', 'port 53 $(reboot)', + 'port 53\nreboot', 'port 53 && curl evil', + # '&' and '!' are hard-rejected even in BPF-legal flag expressions — + # user-supplied filters get the conservative allowlist; the stats + # script builds its own flag filters internally. + 'tcp[tcpflags] & (tcp-syn) != 0', + ]) + def test_injection_attempts_rejected(self, expr): + assert mod.validate_bpf_filter(expr) is not None + + +class TestTcpdumpApprovalPreconditions: + def test_unconfigured_fails_closed(self, monkeypatch): + monkeypatch.setattr(mod, 'TCPDUMP_APPROVAL_DOCUMENT', '') + monkeypatch.setattr(mod, 'APPROVAL_APPROVERS', []) + result = mod.enforce_tcpdump_approval_preconditions('us-east-1') + assert result is not None + assert result['statusCode'] == 503 + + def test_no_approvers_fails_closed(self, monkeypatch): + monkeypatch.setattr(mod, 'TCPDUMP_APPROVAL_DOCUMENT', 'stack-tcpdump-with-approval') + monkeypatch.setattr(mod, 'APPROVAL_APPROVERS', []) + result = mod.enforce_tcpdump_approval_preconditions('us-east-1') + assert result is not None + assert result['statusCode'] == 503 + + def test_cross_region_rejected(self, monkeypatch): + monkeypatch.setattr(mod, 'TCPDUMP_APPROVAL_DOCUMENT', 'stack-tcpdump-with-approval') + monkeypatch.setattr(mod, 'APPROVAL_APPROVERS', [APPROVER]) + result = mod.enforce_tcpdump_approval_preconditions('us-west-2') + assert result is not None + assert result['statusCode'] == 400 + + def test_configured_stack_region_proceeds(self, monkeypatch): + monkeypatch.setattr(mod, 'TCPDUMP_APPROVAL_DOCUMENT', 'stack-tcpdump-with-approval') + monkeypatch.setattr(mod, 'APPROVAL_APPROVERS', [APPROVER]) + assert mod.enforce_tcpdump_approval_preconditions('us-east-1') is None + + +class TestConfirmationGate: + def test_capture_requires_confirmation(self): + result = mod.tcpdump_capture({'instanceId': INSTANCE}) + assert result['statusCode'] == 400 + body = _body(result) + assert body['details']['requiresConfirmation'] is True + + def test_invalid_filter_rejected_before_confirmation(self): + result = mod.tcpdump_capture({ + 'instanceId': INSTANCE, 'filter': 'port 53; rm -rf /', + }) + assert result['statusCode'] == 400 + assert 'BPF filter' in _body(result)['error'] + + +class _FakeSsm: + def __init__(self): + self.calls = [] + + def start_automation_execution(self, **kwargs): + self.calls.append(kwargs) + return {'AutomationExecutionId': 'exec-tcpdump-123'} + + +class _FakeS3: + def __init__(self): + self.objects = {} + + def put_object(self, **kwargs): + self.objects[kwargs['Key']] = kwargs.get('Body', '') + return {} + + +class TestApprovalGatedCapture: + def _setup(self, monkeypatch, fake_ssm, fake_s3): + monkeypatch.setattr(mod, 'REQUIRE_COLLECTION_APPROVAL', True) + monkeypatch.setattr(mod, 'TCPDUMP_APPROVAL_DOCUMENT', 'stack-tcpdump-with-approval') + monkeypatch.setattr(mod, 'APPROVAL_APPROVERS', [APPROVER]) + monkeypatch.setattr(mod, 'APPROVAL_TOPIC_ARN', '') + monkeypatch.setattr(mod, 's3_client', fake_s3) + monkeypatch.setattr(mod, 'resolve_and_validate_region', + lambda args, iid=None: ('us-east-1', None)) + monkeypatch.setattr(mod, 'validate_eks_instance', lambda iid, region: None) + monkeypatch.setattr(mod, 'get_regional_client', lambda svc, region: fake_ssm) + monkeypatch.setattr(mod, 'store_execution_region', lambda eid, region: True) + + def test_capture_pauses_at_approval(self, monkeypatch): + fake_ssm, fake_s3 = _FakeSsm(), _FakeS3() + self._setup(monkeypatch, fake_ssm, fake_s3) + + result = mod.tcpdump_capture({ + 'instanceId': INSTANCE, 'confirmCapture': True, + 'durationSeconds': 30, 'filter': 'udp port 53', + }) + assert result['statusCode'] == 200 + body = _body(result) + assert body['status'] == 'pending_approval' + assert body['executionId'] == 'exec-tcpdump-123' + assert 'approvalConsoleUrl' in body + assert body['humanApproval']['state'] == 'pending' + # nextStep must direct the agent to poll the wrapper execution + assert 'executionId="exec-tcpdump-123"' in body['nextStep'] + + # The wrapper (not send_command) was started, with the script attached + assert len(fake_ssm.calls) == 1 + call = fake_ssm.calls[0] + assert call['DocumentName'] == 'stack-tcpdump-with-approval' + assert call['Parameters']['InstanceId'] == [INSTANCE] + assert 'tcpdump' in call['Parameters']['Commands'][0] + assert call['Parameters']['Approvers'] == [APPROVER] + + # Capture metadata is persisted keyed by execution id + assert 'tcpdump-executions/exec-tcpdump-123.json' in fake_s3.objects + meta = json.loads(fake_s3.objects['tcpdump-executions/exec-tcpdump-123.json']) + assert meta['instanceId'] == INSTANCE + assert meta['filter'] == 'udp port 53' + + def test_unconfigured_approval_fails_closed(self, monkeypatch): + fake_ssm, fake_s3 = _FakeSsm(), _FakeS3() + self._setup(monkeypatch, fake_ssm, fake_s3) + monkeypatch.setattr(mod, 'TCPDUMP_APPROVAL_DOCUMENT', '') + + result = mod.tcpdump_capture({ + 'instanceId': INSTANCE, 'confirmCapture': True, + }) + assert result['statusCode'] == 503 + assert fake_ssm.calls == [] # nothing ran without the approval wrapper + + @pytest.mark.parametrize('extra_args', [ + {}, # host namespace capture + {'podName': 'web-abc123'}, # pod namespace (docker/crictl PID discovery) + {'containerPid': '4242'}, # explicit container PID + ]) + def test_script_has_no_ssm_variable_sequences(self, monkeypatch, extra_args): + """ + Regression: the capture script is passed as a parameter into the + approval wrapper Automation document, and SSM Automation re-resolves + any literal '{{ ... }}' inside substituted parameter values. A Go + template like docker inspect's State.Pid format string caused: + 'Failed to resolve input: .State.Pid ... is not defined in the + Automation Document'. No generated script may contain a literal '{{' + (a bare '}}' is harmless — SSM only resolves opening sequences — and + occurs legitimately where nested JSON objects close). + """ + fake_ssm, fake_s3 = _FakeSsm(), _FakeS3() + self._setup(monkeypatch, fake_ssm, fake_s3) + + result = mod.tcpdump_capture({ + 'instanceId': INSTANCE, 'confirmCapture': True, + 'durationSeconds': 30, **extra_args, + }) + assert result['statusCode'] == 200 + script = fake_ssm.calls[0]['Parameters']['Commands'][0] + assert '{{' not in script, 'script contains SSM variable open sequence' + + +class TestWrapperPolling: + def _execution(self, approve_status, extra_steps=None): + return { + 'AutomationExecutionId': 'exec-tcpdump-123', + 'AutomationExecutionStatus': 'InProgress', + 'DocumentName': 'stack-tcpdump-with-approval', + 'StepExecutions': [ + {'StepName': 'waitForHumanApproval', 'StepStatus': approve_status}, + ] + (extra_steps or []), + } + + def _client_for(self, execution): + class _C: + def get_automation_execution(self, AutomationExecutionId): + return {'AutomationExecution': execution} + return _C() + + def _setup(self, monkeypatch, execution): + monkeypatch.setattr(mod, 'get_execution_region', lambda eid: 'us-east-1') + monkeypatch.setattr(mod, 'get_regional_client', + lambda svc, region: self._client_for(execution)) + monkeypatch.setattr(mod, 'wait_for_approval_decision', + lambda client, eid, ex: ex) + + def test_denied_approval_reports_denied(self, monkeypatch): + self._setup(monkeypatch, self._execution('Failed')) + result = mod._poll_tcpdump_wrapper('exec-tcpdump-123', INSTANCE, {}) + assert result['statusCode'] == 403 + body = _body(result) + assert body['details']['humanApproval']['state'] == 'denied_or_expired' + + def test_pending_approval_keeps_polling(self, monkeypatch): + self._setup(monkeypatch, self._execution('InProgress')) + result = mod._poll_tcpdump_wrapper('exec-tcpdump-123', INSTANCE, {}) + assert result['statusCode'] == 200 + body = _body(result) + assert body['status'] == 'pending_approval' + assert body['humanApproval']['state'] == 'pending' + + def test_approved_without_command_yet_is_in_progress(self, monkeypatch): + self._setup(monkeypatch, self._execution( + 'Success', + extra_steps=[{'StepName': 'runTcpdump', 'StepStatus': 'InProgress'}], + )) + result = mod._poll_tcpdump_wrapper('exec-tcpdump-123', INSTANCE, {}) + assert result['statusCode'] == 200 + body = _body(result) + assert body['status'] == 'in_progress' + assert body['humanApproval']['state'] == 'approved' + + def test_wrapper_document_recognized(self, monkeypatch): + monkeypatch.setattr(mod, 'TCPDUMP_APPROVAL_DOCUMENT', 'stack-tcpdump-with-approval') + assert mod._is_approval_wrapper('stack-tcpdump-with-approval') is True