test: allow teardown of training and checkpoint daily test buckets

[sudheer-quad]

This PR enforces force_destroy: true during the creation of the training and checkpoint buckets to streamline resource cleanup across daily integration tests. This ensures the buckets are successfully cleaned up during cluster teardown, regardless of objects stored within.

Submission Checklist

NOTE: Community submissions can take up to 2 weeks to be reviewed.

Please take the following actions before submitting this pull request.

• Fork your PR branch from the Toolkit "develop" branch (not main)
• Test all changes with pre-commit in a local branch #
• Confirm that "make tests" passes all tests
• Add or modify unit tests to cover code changes
• Ensure that unit test coverage remains above 80%
• Update all applicable documentation
• Follow Cluster Toolkit Contribution guidelines #

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the resource cleanup process for GKE integration tests by introducing a configurable force_destroy flag for training and checkpoint buckets. By parameterizing this setting, the toolkit can now reliably delete buckets containing objects during cluster teardown, preventing orphaned resources in daily automated test environments.

Highlights

• Resource Cleanup Improvement: Introduced a new variable force_destroy_buckets across multiple GKE example configurations to manage bucket deletion behavior.
• Dynamic Bucket Destruction: Updated training and checkpoint bucket definitions to use the force_destroy_buckets variable, allowing for flexible cleanup during cluster teardown.
• Daily Test Integration: Enabled force_destroy_buckets: true in daily integration test suites to ensure clean resource removal after test execution.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a new variable force_destroy_buckets (defaulting to false) across several GKE blueprints (gke-a3-ultragpu, gke-a4, gke-a4x-max-bm, and gke-a4x) and maps it to the force_destroy property of the training and checkpoint GCS buckets. Additionally, it enables this variable (force_destroy_buckets: true) in the corresponding daily integration tests to ensure proper cleanup of resources. I have no feedback to provide as the changes are correct and follow the repository's best practices.

[Neelabh94]

Thank you for this fix! Overriding force_destroy: true on the training and checkpoint buckets is a great operational improvement to prevent teardown failures and orphaned resources during daily integration tests.

During an audit of the other daily tests, noticed two other test suites that also deploy blueprints with training_bucket and checkpoint_bucket defaulting to force_destroy: false and could benefit from this same override:

• tools/cloud-build/daily-tests/builds/gke-a4x-max-bm.yaml (uses examples/gke-a4x-max-bm/gke-a4x-max-bm.yaml)
• tools/cloud-build/daily-tests/builds/gke-tpu-v6e-flex.yaml (uses examples/gke-consumption-options/dws-flex-start/gke-tpu-v6e/gke-tpu-v6e.yaml)

Could we include these two files in this PR as well? I've also left a small suggestion on using extended regex in sed for spacing resilience.

[Neelabh94]

Using extended regex (-E) with \s* ensures this substitution remains resilient if the example blueprint is formatted or adjusted with different whitespace:

sed -i -E 's/force_destroy:\s*false/force_destroy: true/g' \$$EXAMPLE_BP