Add CHS integration to daily tests for A3, A4, and TPU machine types

[simrankaurb]

Description

This PR integrates the CHS into the daily integration testing pipeline for several GPU and TPU topologies across both GKE and Slurm environments.

Key Changes:

• Test Definitions: Added test-validation/test-chs.yml to the post_deploy_tests for A3 Mega, A3 Ultra, A4, A4X, A4X Max, TPU v6e, and TPU v7x daily tests.
• Targeting Logic:
  • Introduced chs_nodepools string variable for GKE test definitions to ensure CHS runs specifically on GPU/TPU machine pools (using the --nodepools flag).
  • Introduced chs_partitions string variable for Slurm test definitions to correctly target Slurm partitions (using the --partitions flag).
• CHS Playbook Refactoring: Updated the CHS run command in test-chs.yml to include the -y flag and cleanly handle the conditional targeting logic (target_flag, target_val, chs_target) depending on the scheduler type.
• Cloud Build Updates: Updated the Cloud Build YAML files for the affected machine types to expose and pass the CHS_REPO secret to the ansible playbooks.

Affected Topologies:

• A3 Mega: gke-a3-megagpu-onspot, ml-a3-megagpu-onspot-slurm-ubuntu
• A3 Ultra: gke-a3-ultragpu-onspot, ml-a3-ultragpu-onspot-slurm
• A4 / A4X: gke-a4-onspot, gke-a4x, gke-a4x-max-bm, ml-a4-highgpu-onspot-slurm, ml-a4x-highgpu-slurm
• TPU: gke-tpu-7x, gke-tpu-v6e, gke-tpu-v6e-flex

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the daily integration testing suite by incorporating the Cluster Health Scanner (CHS) for various GPU and TPU topologies. By standardizing the targeting logic and updating the build configurations to handle necessary secrets, the changes ensure that cluster health validation is consistently performed across both GKE and Slurm environments, improving overall test coverage and reliability.

Highlights

• CHS Integration: Integrated the Cluster Health Scanner (CHS) into the daily integration testing pipeline for A3, A4, and TPU machine types across GKE and Slurm environments.
• Targeting Logic: Introduced dynamic targeting variables (chs_nodepools for GKE and chs_partitions for Slurm) to ensure CHS runs on the correct infrastructure.
• Playbook Updates: Refactored the CHS ansible playbook to support conditional execution flags and automated targeting based on the scheduler type.
• Cloud Build Configuration: Updated multiple Cloud Build YAML files to expose and pass the CHS_REPO secret, enabling secure access for the integration tests.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request integrates the Cluster Health Scanner (CHS) into various daily integration tests for GKE and Slurm by adding the CHS_REPO secret, defining scheduler and target variables in test configurations, and running the CHS test suite post-deployment. Feedback from the reviewer highlights potential AnsibleUndefinedVariable errors in test-chs.yml due to missing default filters for the new variables, and warns about premature variable expansion of CHS_REPO in several Cloud Build YAML files if they are generated within unquoted bash heredocs.