Optional Managed Instance Group (MIG) support for static compute nodes

[arpit974]

Overview

This pull request introduces native, optional support for provisioning static Slurm compute nodes using Google Cloud Managed Instance Groups (MIGs).

Previously, static compute nodes were provisioned as standalone Compute Engine VMs using bulk insertion. With this change, users can choose to have static nodes managed by Google Cloud MIGs, unlocking automatic infrastructure self-healing and centralized group management while keeping Slurm job scheduling and MPI communication fully intact.

• Intelligent AUTO Behavior: By default (provisioning_engine = "AUTO"), the toolkit automatically routes DWS Flex-Start workloads to MIGs and standard compute workloads to Bulk Insert with zero user configuration. Existing clusters operate completely unchanged, while users who want automated hardware self-healing for their dedicated static nodesets can simply set provisioning_engine = "MIG".

---

Why Managed Instance Groups?

1. Automatic Health Repair & Recovery: If an underlying physical server experiences a hardware failure or a VM encounters an unrecoverable fault, Google Cloud's auto-healing automatically replaces the instance. Slurm detects this repair in real-time, avoids scheduling jobs on the recovering node, and brings it back to idle once healthy.
2. Deterministic Naming & Job State: Even inside a managed group, nodes retain their exact, sequential Slurm hostnames (e.g., cluster-nodeset-0, cluster-nodeset-1), maintaining full compatibility with Slurm configs and MPI communication.
3. Large Cluster Scalability: For clusters with large static nodesets exceeding 1,000 instances, the system automatically distributes nodes across multiple indexed groups (e.g., <nodeset>-mig-0, <nodeset>-mig-1) to remain well within Google Cloud's recommended group limits.
4. 100% Backward Compatible: The default setting remains "AUTO", meaning existing blueprints, dynamic auto-scaling nodes, and standard bulk-insert workflows continue to function with zero changes.

---

How It Works

• The AUTO User Scenario: Users do not need to configure anything by default. "AUTO" preserves standard bulk provisioning for general workloads and automatically engages MIGs for DWS Flex-Start. Users explicitly specify "MIG" only when they want GCE-managed auto-repair on their static compute nodes.
• Declarative MIGs with Per-Instance Configs: MIGs are declared in Terraform with a target size of 0. When the cluster boots, the Slurm controller scripts call GCE APIs to add instances to the group using deterministic names and configurations.
• Smart Power & Suspend Management: When Slurm powers down or suspends static nodes, suspend.py removes them from the group. When resumed, resume.py recreates them in the group with the current instance template and binds the Slurm Job ID to instance metadata.
• Auto-Repair Sync: slurmsync.py checks whether GCE is actively repairing a VM. If a node is in a REPAIRING state, Slurm marks it as DOWN with the reason "MIG Auto-Healing instance repair in progress" so no jobs are sent to it until it recovers.
• FQDN & Hostname Resilience: Suspension and status synchronization seamlessly support both short hostnames and Fully Qualified Domain Names (FQDNs) used in private DNS or OS Login configurations.

---

Summary of Changes

• Terraform Modules:
  • schedmd-slurm-gcp-v6-nodeset: Added provisioning_engine input variable with validation. Added safety guardrail ensuring pure static sizing when MIG is selected.
  • schedmd-slurm-gcp-v6-partition: Passed through provisioning_engine configuration to partition definitions.
  • schedmd-slurm-gcp-v6-controller: Added declarative regional MIG resources, automatic multi-group indexing, and safety preconditions to prevent unsupported engine mutations on active nodes.
• Controller Python Scripts:
  • resume.py: Added resume_mig_nodes() to provision per-instance configurations into the target MIG.
  • suspend.py: Added suspend_mig_nodes() with FQDN short-hostname normalization to remove instances via deleteInstances.
  • slurmsync.py: Added GCE Auto-Repair status detection to mark repairing nodes DOWN in Slurm.
  • util.py: Added MIG lookup helpers (is_node_mig, node_mig_name, get_mig_instances, and group size expansion math).
• Automated Unit Tests:
  • Added unit test coverage in test_util.py and test_resume.py covering multi-MIG indexing, FQDN node resolution, API schema variations, and auto-repair status reporting.

---

Testing & Verification

1. Automated Test Suites

• All 183 unit tests passed in Python 3.13 (pytest-check).
• Pre-commit checks passed cleanly across the entire codebase (mypy, terraform_fmt, terraform_validate, codespell, duplicate-diff, module-label-check).

2. Live GCP Deployment & Workload Testing

Verified on a live multi-partition Slurm cluster deployed on Google Cloud covering:

• Pure Static On-Demand MIGs: Executed parallel multi-node MPI jobs (srun, sbatch) with ExitCode 0:0.
• Pure Static Spot MIGs: Verified spot VM creation, state persistence, and job execution with ExitCode 0:0.
• Multi-MIG Scaling: Verified jobs co-scheduling across multiple distinct MIGs.
• Hybrid Partitions: Successfully ran concurrent jobs across a mixed partition containing both static MIG nodes and dynamic bulk-insert nodes.
• Dynamic Scale-to-Zero Partitions: Verified standard dynamic burst nodes remain 100% operational on BULK_INSERT.
• DWS Flex Partitions: Verified Flex-Start dynamic provisioning continues to operate normally.

3. Fault Tolerance & Disaster Recovery

• Simulated Hardware Failure: Deleted an active static node out-of-band directly in Google Cloud. GCE automatically triggered an auto-repair recreation, slurmsync kept Slurm informed, the node safely transitioned back to idle, and subsequent parallel MPI jobs ran with zero errors.
• Controller Reboot: Hard-rebooted the controller VM (sudo reboot). Services restarted cleanly, reconnected to all managed instances, and resumed job scheduling without manual intervention.
• Clean Teardown: Verified gcluster destroy cleans up all instances, MIGs, disks, and templates with 0 orphaned cloud resources.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces support for Managed Instance Groups (MIGs) as an alternative to the existing bulk insert provisioning method for compute nodes. By enabling MIGs, the cluster gains improved instance lifecycle management, including automated repair capabilities. The changes span across Terraform infrastructure definitions to provision the necessary MIG resources and Python scripts to handle the logic for resuming and suspending nodes within these managed groups.

Highlights

• Provisioning Engine Configuration: Introduced a new provisioning_engine variable, allowing users to choose between 'BULK_INSERT' (default) and 'MIG' for compute node provisioning.
• MIG Support Implementation: Added infrastructure support for Managed Instance Groups (MIGs) via google_compute_region_instance_group_manager resources, including automated instance lifecycle management.
• Resume/Suspend Logic Updates: Updated resume.py and suspend.py to handle MIG-specific operations, including atomic instance template updates and per-instance configuration (PIC) for static Slurm hostnames.
• Health and Repair: Enhanced slurmsync.py to recognize 'REPAIRING' states in MIGs, preventing premature node actions during auto-healing processes.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces support for a Managed Instance Group (MIG) provisioning engine alongside the existing BULK_INSERT engine in the SchedMD Slurm GCP v6 controller. The changes span Terraform configurations to conditionally deploy MIGs, and Python scripts to manage MIG-based node resumption, suspension, and synchronization. The review feedback highlights several critical improvements: correcting Terraform coalesce usage with lists, initializing target_size to 0 for stateful MIGs to avoid duplicate instances, replacing inefficient single-request execute_with_futures calls with ensure_execute, and replacing global is_mig_engine checks with per-nodeset mig_name checks to safely support mixed nodeset environments.

[Neelabh94]

Thanks for putting together this PR to support Managed Instance Groups for static compute nodes! This is a great addition for enabling hardware auto-repair and self-healing for long-running Slurm workloads.

Overall the architecture is well thought out. I've left a few inline comments on areas around GCE API compatibility, validation, idempotency, and edge-case handling.