Add orphaned test-VPC reaper and standardize dry-run across cleaners#2198
Open
sky333999 wants to merge 6 commits into
Open
Add orphaned test-VPC reaper and standardize dry-run across cleaners#2198sky333999 wants to merge 6 commits into
sky333999 wants to merge 6 commits into
Conversation
clean_eks: after the cluster pass, reap dedicated EFA test VPCs (tag:Name efa-test-vpc-*) left behind by failed terraform destroys. Tears the stack down in dependency order — VPC endpoints, NAT gateways, Elastic IPs, leftover ENIs, internet gateways, subnets, route tables, security groups (revoke-then-delete), then the VPC — behind three safety gates: not backing a live EKS cluster (prod + best-effort beta), no active EC2 instances (re-checked before subnet deletion), and an age gate from the newest NAT gateway/VPC endpoint. A cleaner:reaping tag lets a partially-torn-down VPC resume on a later run instead of being stranded once its age signals are gone. Shared dry-run: add tool/clean/clean_flags.go (clean.DryRun, RegisterCommonFlags, Skip) and gate every mutating call across all 12 cleaners. Add a -dry-run flag to each tool and a workflow_dispatch dry_run input (DRY_RUN env: scheduled runs stay live, manual defaults to dry-run). Normalize --tags=clean to build-flag position and move ecs/host region args to flag.Arg(0). Also fix pre-existing bugs surfaced while touching these files: - clean_ecs: inverted-sign expiration deleted clusters with active tasks - clean_security_group: break-in-select spun pagination to timeout - clean_ebs / clean_file_system: age no-op and stale-error under-deletion
sky333999
force-pushed
the
sky333999/efa-vpc-reaper
branch
from
July 12, 2026 23:46
6392598 to
9364aea
Compare
sky333999
marked this pull request as ready for review
July 13, 2026 17:05
…waits, vet-clean Skip)
- vpcsInUseByClusters: treat ResourceNotFoundException (a cluster mid-deletion,
e.g. by terminateClusters just before) as not-in-use and continue, instead of
aborting the entire VPC reap pass; other errors still fail closed.
- waitVpcEndpointsGone / waitNatGatewaysGone: honor ctx cancellation during the
poll interval instead of an unconditional time.Sleep.
- clean.Skip: forward the format string so go vet recognizes it as a printf
wrapper and validates format/arg mismatches at every call site.
- Rename exported ClustersToClean to clustersToClean (package main, no external
consumers).
- Workflow: quote -dry-run="${DRY_RUN}" and correct the DRY_RUN comment.
- Note the single-Describe-page assumption on the per-VPC delete helpers.
The reaper stripped orphaned efa-test-vpc-* VPCs but couldn't delete them: deleteSecurityGroups replayed the IpPermissions from DescribeSecurityGroups straight into Revoke*, but those carry Description fields (and peer-SG UserIdGroupPairs) that Revoke* rejects with InvalidRequest. The revoke thus did nothing, so mutually-referencing EKS SGs survived and both the SGs and the VPC failed with DependencyViolation on delete. Revoke by SecurityGroupRuleId instead: DescribeSecurityGroupRules (filter group-id) -> split into ingress/egress rule IDs -> RevokeSecurityGroup* with SecurityGroupRuleIds. Keep the revoke-all-then-delete two-pass so mutual references are cleared before any delete. Also address two related gaps from the same run: - Raise the VPC-endpoint delete wait from 3m to 5m; a lingering interface- endpoint ENI is itself a VPC/SG dependency and one endpoint timed out. - Add a cleaner:first-seen fallback age signal so VPCs whose NAT gateway and endpoints were already torn down (no age signal) are no longer stranded: tag on first sight, reap once the marker is older than the keep-duration. vpcReapEligibleByAge now reports whether an age signal existed, so "too new" and "no signal" are handled distinctly. The marker write is dry-run gated.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of changes
clean_eks: after the cluster pass, reap dedicated EFA test VPCs (tag:Name efa-test-vpc-*) left behind by failed terraform destroys. Tears the stack down in dependency order — VPC endpoints, NAT gateways, Elastic IPs, leftover ENIs, internet gateways, subnets, route tables, security groups (revoke-then-delete), then the VPC — behind three safety gates: not backing a live EKS cluster (prod + best-effort beta), no active EC2 instances (re-checked before subnet deletion), and an age gate from the newest NAT gateway/VPC endpoint. A cleaner:reaping tag lets a partially-torn-down VPC resume on a later run instead of being stranded once its age signals are gone.
Shared dry-run: add tool/clean/clean_flags.go (clean.DryRun, RegisterCommonFlags, Skip) and gate every mutating call across all 12 cleaners. Add a -dry-run flag to each tool and a workflow_dispatch dry_run input (DRY_RUN env: scheduled runs stay live, manual defaults to dry-run). Normalize --tags=clean to build-flag position and move ecs/host region args to flag.Arg(0).
Also fix pre-existing bugs surfaced while touching these files:
License
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Tests
WIP
Requirements
Before commiting your code, please do the following steps.
make fmtandmake fmt-shmake lintIntegration Tests
To run integration tests against this PR, add the
ready for testinglabel.