fix(slurm): don't restart slurmctld on a topology-only reconfigure

[nir0s]

Problem

util.scontrol_reconfigure() has two callers in slurmsync.py:

caller	fires when	frequency
reconfigure_slurm()	the config bucket changed (regenerates cloud.conf)	rare
update_topology()	the topology changed — i.e. a node powered up	every autoscale event

#4609 moved an unconditional
systemctl restart slurmctld.service into that shared helper. It is reasonable for the first caller
(which already restarted explicitly on the line above). But it also applied to the second, which had
been doing a plain scontrol reconfigure — so since then an autoscaling cluster restarts its
controller once per scale event. On our cluster that is ~25 slurmctld restarts/day.

The restart is not harmless. A node that registers within 300 s of a slurmctld restart is
subsequently flagged not responding, and jobs running on it are requeued.

Evidence

Reproduced on demand on a live cluster, 4/4, on idle and busy nodes:

• The flag fires at exactly restart + 300 s on a node that registers in the seconds after the
  restart. A node that was already steady before the restart is unaffected.

• A plain scontrol reconfigure (no restart) does not reproduce it.

• It is immune to every relevant configured timeout — each varied to a distinct value and confirmed
  live via scontrol show config across the restart:

  varied	result
  SlurmdTimeout = 150 / 200 / 450	+300
  ResumeTimeout = 600 (global and partition)	+300
  SuspendTimeout = 500	+300
  BatchStartTimeout = 500	+300

  300 s is Slurm's default SlurmdTimeout, which suggests a node registering just after a restart
  is health-checked against the default rather than the configured value. That part may well be a
  Slurm-side issue — but it is only reachable because we restart slurmctld on the hot path.

This is most visible on slow-provisioning machine types. Ours (h4d-standard-192 with Cloud RDMA)
takes 563–576 s to provision and register, so during a batch bring-up node N's power-up restarts the
controller while node N+1 is still registering. Faster types tend to register before the restart
their own power-up triggers, so they never land in the window — which is likely why this has not been
widely reported.

Fix

Make the restart opt-out on the helper, and have the topology path skip it. A topology change takes
effect on scontrol reconfigure (per the topology.conf man page), so the restart is not needed
there. The config-change path in reconfigure_slurm() is deliberately left untouched — node-set
reconciliation may genuinely require the restart, and this change is scoped not to alter it. In effect
this restores the pre-#4609 behaviour for the topology path only.

Testing

• Added a parametrized unit test in tests/test_util.py asserting scontrol reconfigure always runs
  and the restart is opt-out.
• Full script suite passes locally: 173 passed.
• Happy to add an integration-level test if you would like one.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses an issue where frequent slurmctld restarts during topology updates were causing nodes to be incorrectly flagged as unresponsive. By making the service restart optional in the scontrol_reconfigure utility, the system can now apply topology changes without triggering a full controller restart, thereby improving stability during autoscaling operations.

Highlights

• Slurm Controller Stability: Modified the scontrol_reconfigure helper to make the slurmctld service restart optional, preventing unnecessary service restarts during topology updates.
• Autoscaling Optimization: Updated update_topology to call scontrol_reconfigure with restart=False, eliminating a performance bottleneck and preventing node registration failures during autoscaling events.
• Regression Testing: Added a new parameterized unit test in test_util.py to verify that scontrol reconfigure always executes while the slurmctld restart remains configurable.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[gemini-code-assist]

Code Review

This pull request introduces an option to skip restarting the slurmctld daemon during configuration updates, preventing node flapping during topology updates. The review feedback suggests a valuable optimization: when a daemon restart is performed, running scontrol reconfigure is redundant and can cause race conditions. The reviewer recommends only running scontrol reconfigure when restart is False, and provides suggestions to update both the implementation and its unit tests accordingly.

[gemini-code-assist]

When restart is True, slurmctld is restarted. A restarted daemon will automatically read all configuration files upon startup, making a subsequent scontrol reconfigure completely redundant. Furthermore, running scontrol reconfigure immediately after a restart can introduce a race condition where the command fails because the daemon is not yet fully initialized and listening for connections. It is safer and more efficient to only run scontrol reconfigure when restart is False.

Suggested change

	if restart:
	log.info("Running systemctl restart slurmctld.service")
	run("sudo systemctl restart slurmctld.service", timeout=30)
	log.info("Running scontrol reconfigure")
	run(f"{lkp.scontrol} reconfigure")
	if restart:
	log.info("Running systemctl restart slurmctld.service")
	run("sudo systemctl restart slurmctld.service", timeout=30)
	else:
	log.info("Running scontrol reconfigure")
	run(f"{lkp.scontrol} reconfigure")

[gemini-code-assist]

Update the test assertions to match the updated behavior where scontrol reconfigure is only executed when restart is False.

Suggested change

	commands = [c.args[0] for c in mock_run.call_args_list]
	assert "scontrol reconfigure" in commands
	assert ("sudo systemctl restart slurmctld.service" in commands) == expect_restart
	commands = [c.args[0] for c in mock_run.call_args_list]
	if expect_restart:
	assert "sudo systemctl restart slurmctld.service" in commands
	assert "scontrol reconfigure" not in commands
	else:
	assert "sudo systemctl restart slurmctld.service" not in commands
	assert "scontrol reconfigure" in commands

[nir0s]

Additional evidence, from 49 days of production slurmctld.log on an unpatched cluster (2026-07-13 → 2026-08-31; TopologyPlugin=topology/tree, h4d nodeset, autoscaling).

I correlated every not responding event with the most recent preceding Running as primary (slurmctld restart) line:

not-responding events with a preceding restart: 164
top deltas (seconds since last slurmctld restart -> count):
      300 s : 115
      301 s : 48
      600 s : 1
within 295-305 s of a restart: 163 / 164 = 99%

Every not responding event on this cluster in 49 days is anchored to a slurmctld restart, at +300 s. 115 land at exactly 300, 48 at 301, and the lone outlier at 600 = exactly two windows. None occur independently of a restart. 300 s is the default SlurmdTimeout; the cluster's configured value is also 300 here, but the earlier controlled runs varied it to 150/200/450 and the flag stayed at +300 regardless.

Restart rate on the same log, per day: 16, 24, 16, 8, 10, 16, 10, 4, 28, 16, 10, 22 — i.e. ~15/day typical, 28 peak, all from the update_topology() path on an ordinary autoscaling workload.

This is what the PR removes: the topology path no longer restarts, so the window in which a registering node can be falsely flagged no longer opens on every scale event.

(Full disclosure on scope: this is natural-history evidence that the restart causes the flags. Validation that the patched build behaves correctly — specifically that a plain scontrol reconfigure fully applies a topology change on 25.11 + topology.yaml — is in progress on a dev cluster and I'll report it here either way.)

[aslam-quad]

Hi @nir0s

Could you please take a look at the Gemini suggestions?

Thanks

[aslam-quad]

/gcbrun