Batch GetIamPolicy calls for all google_*_iam_* resources - #18892
Batch GetIamPolicy calls for all google_*_iam_* resources#18892joscha-alisch wants to merge 2 commits into
Conversation
The code here used readIamPolicyWithRetry which is (1) doing retries within retries (2) trying to aquire a lock that is already held by this function. Additionally, the error check was inverted and would actually cause a panic if triggered.
|
Googlers: For automatic test runs see go/terraform-auto-test-runs. @roaks3, a repository maintainer, has been assigned to review your changes. If you have not received review feedback within 2 business days, please leave a comment on this PR asking them to take a look. You can help make sure that review is quick by doing a self-review and by running impacted tests locally. |
|
This PR also relates to #18890. Together they speed up our organizations GCP project management pipelines by a factor of 10 through this efficient batching and also completely eliminate any rate limiting on read requests per minute. |
|
@roaks3 This PR has been waiting for review for 3 weekdays. Please take a look! Use the label |
Currently, only the write path for IAM policies is batched. This makes the write path quick, but the refresh / read path for all
google_*_iam_*resources makes the same redundant API requests for each of those.In case of non-authoritative management of IAM policies via
google_project_iam_memberthis can result in hundreds of unnecessary API calls.This PR introduces request batching for the read path of all
google_*_iam_*resources, similiar to how it's already being done forgoogle_project_service.In my test project with 175 individual
google_project_iam_member, the number of IAM policy API calls goes from 175 down to 1, reducing the time to refresh/plan from 73 seconds to 14.Contributes to
While introducing this, I also noticed an inverted error condition causing a panic and also another deadlock in one of the call sites. Given I had to touch that call site anyway, I also fixed those two issues (guided by tests showing both deadlock and panic). Let me know if you'd rather want this as a separate PR.
Release Note Template for Downstream PRs (will be copied)
See Write release notes for guidance.