Fix GPU image smoke launches with the restricted workload identity - #921
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ART main images build and push, but the post-build SkyPilot smoke launch fails before a pod is provisioned, preventing the downstream Caladan image build from being dispatched. The smoke workflow omits the workload identity already configured by ART's trainer CI and Caladan's image workflow. SkyPilot 0.12 consequently tries to restore its legacy wildcard RBAC, which cks-wb3 admission policy has blocked since September 3.
Configure the existing
skypilot-workloadservice account for the smoke launch. Also print the provision log on failure so the underlying Kubernetes error is visible rather than onlyResourcesUnavailableError.Validation
CI passes at
5a9561440: 783 Megatron lightweight tests and 1,517 unit tests, with 73 skips; lint/type checks are green.Compared recent failed image runs with the last successful September 3 run and the cluster admission-policy creation time. Builds/pushes succeed; smoke provisioning fails.
Confirmed the denial with a Kubernetes server-side dry run; no RBAC was modified. Inspected the workflow-pinned SkyPilot 0.12 bootstrap path to verify the explicit identity skips legacy RBAC setup.
Parsed the workflow, checked the smoke step with
bash -n, and passed the existing GPU image build-context test.Live validation passed against the previously published immutable ART image (
7ef066b5) on free cks-wb3/H200 infrastructure using the hosted SkyPilot service. The pod usedserviceAccountName: skypilot-workloadand printedART_IMAGE_SMOKE_OKafter checking the image revision, SkyPilot/Ray paths, and uv. The validation cluster was explicitly terminated; Kubernetes confirms no owned pods or services remain.This change does not modify the image, cluster admission rules, or service-account permissions.