Support Azure Container Linux: install under /opt/unbounded and Ignition bootstrap - #323
Draft
Philip Lombardi (plombardi89) wants to merge 20 commits into
Draft
Philip Lombardi (plombardi89) wants to merge 20 commits into
Philip Lombardi (plombardi89) wants to merge 20 commits into
Conversation
Philip Lombardi (plombardi89)
had a problem deploying
to
e2e-testing
September 21, 2026 18:09 — with 
GitHub Actions
Failure
GitHub Actions
Failure
Philip Lombardi (plombardi89)
deployed
to
e2e-testing
September 21, 2026 23:43 — with
GitHub Actions
Active
GitHub Actions
Active
The agent library places its own binaries and helper scripts under a fixed /usr/local prefix. Azure Container Linux mounts /usr read-only, so nothing can be installed there and a Flex Node cannot run on it at all. The version that makes the prefix configurable is not released yet, so this pins the commit under review upstream rather than a tag. That is temporary and has to become a release version before this merges. The same build also reports host package capability truthfully instead of assuming a package manager exists, which matters here because the image has none.
Hosts that mount /usr read-only cannot take the agent's own binaries and helper scripts in the default location. A config field lets them name a writable one. The prefix is declared, never inferred. Keying off distribution identity would misclassify a hardened host whose /usr happens to be read-only, and would silently relocate files on any host whose os-release changed. A wrong guess is expensive to recover from, because generated systemd units reference these paths absolutely. Validation is delegated to the agent library rather than duplicated. The prefix is interpolated into generated units and into a shell script, neither of which quotes it, so the accepted syntax is deliberately narrow; keeping that rule in one place stops the two projects from disagreeing about what is accepted.
defaultAgentUpgradePaths hardcoded /usr/local/bin and
/usr/local/lib/aks-flex-node, both under a read-only /usr on Azure Container
Linux, and it is called with no override, so the location could not be
changed.
The prefix is read from this project's own config rather than the agent
library's applied config. That distinction was found the hard way: the
library exposes a lookup for exactly this, but bootstrap here never writes
that file, so on a real host it is absent, the lookup silently returns the
default, and the failure surfaces later as
initialize agent binary layout: no executable agent binary found
which names neither the prefix nor the path it looked in.
An unreadable or absent config yields the empty prefix, which selects the
default. That is what a host installed before this existed actually has on
disk, so nothing changes for it.
Callers that already hold a config pass it directly; the lookup exists for
entry points started by systemd, which cannot inherit the prefix from the
environment that ran bootstrap.
recoveryScriptPath was a second hardcoded location under /usr/local/lib. On
Azure Container Linux this fails bootstrap after the node has already joined:
install-service: write /usr/local/lib/aks-flex-node/aks-flex-node-recovery.sh:
mkdir /usr/local/lib/aks-flex-node: read-only file system
which is a worse failure than refusing up front, because the cluster has a
node in it by then.
The constant stays, because it is also the substitution placeholder inside
the embedded recovery unit. Only the install location becomes prefix
relative; the literal in the unit template has to keep matching the asset.
systemdSystemDir is deliberately not prefix relative. Units have to live where
systemd looks for them, and /etc is writable even when /usr is not.
The prefix work it depends on was consolidated upstream and rebased onto main after the bootstrap ownership change merged, so the previously pinned commit no longer exists on any branch. Still an unreleased commit rather than a tag, and still has to become a release version before this merges.
The recovery script ships with the default-prefix paths as literals, and installation rewrites them for the host's layout. The rewrite looked for the paths resolved from the installed config instead of the literals. On a host with a custom prefix those are the prefixed paths, which the script does not contain, so nothing was replaced and the script kept reading last-good from /usr/local. A failed upgrade could then not be rolled back. The replacement now keys on the default layout, which is what the embedded script contains, and a test pins that the asset and the default layout agree.
UninstallService read the host prefix from /etc/aks-flex-node/config.json when it ran. The AgentReset MachineOperation and the RP delete path both run ResetNode first, which removes that directory, and only then uninstall the service. By then the prefix read as the default, so on a host with a custom prefix the recovery script was left behind. The prefix is now a parameter. The daemon passes the one from its loaded config, and the reset command reads the installed config before anything is removed. The recovery script is removed under that prefix and under the default, so a host that changed prefix is also cleaned up. Removal checks for the file first. On a read-only /usr, unlinking a file that does not exist returns EROFS rather than ENOENT, which would fail the default-prefix sweep on a file that was never there.
Reset called the library's interface and route cleanups directly and skipped its LocalDNS cleanup, so every reset of a node with LocalDNS enabled left the unit, the nft table, the dummy interface and the helper script on the host. It also never removed the nspawn lifecycle helper. Reset now runs the library's CleanupNetwork, which includes the LocalDNS cleanup, and removes the nspawn lifecycle helper. Both helpers live under the host prefix, so reset takes the prefix and removes them under it and under the default. The managed agent binaries are still kept. The e2e RP delete check now also asserts the LocalDNS state, the nspawn helper and the recovery script are gone. The MSI node enables LocalDNS, so this runs against a real LocalDNS setup.
Repave cleans up the network with the library's CleanupNetwork, which removes the LocalDNS helper. Without the prefix it only looks under /usr/local, so on a host with a custom prefix the old helper stayed until StartNode wrote a new one. That was harmless because StartNode runs right after, but it now uses the same prefix as the rest of the node.
…t one install.sh and uninstall.sh always used /usr/local. The agent resolves its binaries from agent.hostPrefix, so on a host with a custom prefix the installer put the binary where the agent does not look. On Azure Container Linux it could not install at all, because /usr is read-only. Both scripts now take the prefix from agent.hostPrefix in the node config when it exists, or from AKS_FLEX_NODE_HOST_PREFIX, which covers the usual case where the config is written after installation. install.sh fails if the two disagree. install.sh also treated Azure Container Linux as Azure Linux 3, since it reports ID=azurelinux with a 3.x VERSION_ID. It is now identified by VARIANT_ID=azurecontainerlinux, or by ID_LIKE=flatcar for an image that drops VARIANT_ID. Installing there with the default prefix fails and says which variable and config field to set. uninstall.sh reads the prefix before running reset, which removes the config. It now also removes the managed blue/green layout, which reset keeps and which uninstall previously left behind, and the recovery unit when it falls back to removing units directly. It can be sourced now, and uninstall_test.sh covers the prefix handling and binary removal.
bootstrap.sh always installed the agent in /usr/local/bin, or wherever --install-dir pointed. The agent resolves its binaries from agent.hostPrefix, so any other combination left it looking in the wrong place, and on Azure Container Linux /usr is read-only. --host-prefix, or AKS_FLEX_NODE_HOST_PREFIX, now sets agent.hostPrefix, and the binary goes in <prefix>/bin. Without the flag the prefix comes from the base config and config overrides, and defaults to /usr/local as before. The prefix is resolved before rendering because fetching bootstrap data runs the installed binary, so bootstrap data cannot move it; a response that tries is refused. --install-dir is deprecated and only accepted when it names the same directory. The test runs every case as root, so a regression that fell back to the default prefix would have replaced a real /usr/local/bin/aks-flex-node. It now runs in a private mount namespace with /usr/local overlaid, fails if any prefixed case writes there, and works from inside its temp dir so a relative path cannot land in the checkout.
scripts/bootstrap_test.sh existed but nothing ran it, so neither make test nor PR checks caught a bootstrap.sh regression. make test-install, which both use, now runs it after the installer and uninstaller tests.
Azure Container Linux is provisioned only by Ignition and has a read-only /usr, so bootstrap.sh cannot be downloaded and run by hand at first boot. aks-flex-node ignition renders an Ignition config that writes the embedded bootstrap.sh, with the base config in place of its placeholder, and any service principal credential, and enables aks-flex-node-bootstrap.service to run the script with the arguments given after --. The agent goes under /opt/aks-flex-node unless --host-prefix or the base config names another prefix. The unit retries with a growing delay until the agent unit exists, and is skipped after that, since bootstrap.sh refuses to run over an existing deployment. It removes the script, which carries the base config, once bootstrap succeeds. The arguments are checked against the script's options, and a test keeps that list in step with the script, so mistakes surface on the workstation instead of at first boot. Reset disables, stops, and removes the unit before stopping the agent, since in the daemon's reset paths stopping the agent ends the reset. uninstall.sh does the same when there is no binary to run reset with.
hack/acl/acl.py boots Azure Container Linux under QEMU from the config that aks-flex-node ignition renders and joins it to a kind cluster, with the AKS Flex controller deployed in the cluster in place of Azure. It checks that the first-boot unit installs the agent under the prefix and removes the script that carries the token, that a workload runs, that a reboot skips the unit, and that reset, through a MachineOperation or the CLI, removes everything it should. Each removed path is first checked to exist. It passes on build 3.0.20260818 with both reset paths, and fails as it should against an agent whose reset does not stop the first-boot unit. The networking, VM launch, and UKI patching are adapted from the unbounded harness; ukiboot.py is copied from it unchanged.
configuration.md did not list agent.hostPrefix. It now says what the prefix moves and what it does not, that hosts with a read-only /usr need it, and to reset before changing it. The getting-started list of what bootstrap.sh does put the agent download after the bootstrap data fetch, but the binary is installed first, and now under the host prefix. The bootstrap design says why the Ignition wrapper keys on the agent unit rather than a completion marker, which reset would leave behind.
Philip Lombardi (plombardi89)
force-pushed
the
phlombar/acl-support
branch
from
September 24, 2026 14:31
ce0a0ad to
79b2808
Compare
Philip Lombardi (plombardi89)
deployed
to
e2e-testing
September 24, 2026 14:31 — with
GitHub Actions
Active
GitHub Actions
Active
Philip Lombardi (plombardi89)
changed the title
The agent library build now includes the fixes from its review: reset takes the host prefix once, the applied config's checksum is checked before its prefix is used, and the nspawn machines order after the nftables flush. hack/acl/ukiboot.py is copied unchanged from the same build. It now requires exactly one UKI, measures the command line in encoded bytes, and cleans up after a failed qemu-nbd start.
Philip Lombardi (plombardi89)
deployed
to
e2e-testing
September 24, 2026 21:28 — with
GitHub Actions
Active
GitHub Actions
Active
newRootCommand attaches the package-level token.Command, so building two roots at once writes its parent from both goroutines. The ignition registration test ran in parallel with the existing root command test, and the race detector in PR checks caught it. It now runs serially.
Philip Lombardi (plombardi89)
deployed
to
e2e-testing
September 24, 2026 21:42 — with
GitHub Actions
Active
GitHub Actions
Active
The host prefix is replaced by the host root from the agent library, /opt/unbounded, on every host. The binary layout, the recovery script and unit, and the library's helpers are all built from the root resolved through symlinks. A host installed by an earlier release keeps its layout under /usr/local. start, the daemon, agent-upgrade, recover-agent-upgrade, and the nspawn hooks first link /opt/unbounded to it, so the paths the older release wrote into links and units still match the ones built here. Reset and uninstall.sh sweep both roots and do not depend on the link. A hidden host-root command prints the root. install.sh and bootstrap.sh ask the downloaded binary for it, from a copy under /var/lib/aks-flex-node rather than a possibly noexec /tmp. A release without the command goes to /usr/local/bin as before, and where that is read-only the scripts say the release is too old. The agent library refuses an AgentUpgrade to such a release on a host installed under /opt/unbounded. Clients that run on the host used the kubelet's exec credential, which names the binary inside the machine, /usr/local/bin/aks-flex-node. They now run the one under the host root. agent.hostPrefix, bootstrap.sh --host-prefix, AKS_FLEX_NODE_HOST_PREFIX, ignition --host-prefix, and the Azure Container Linux check in install.sh are removed. The Azure e2e gains an upgrade from v0.2.0, the agent-upgrade e2e checks that a release before the host root is refused, and the agent library pin moves to Azure/unbounded#815 at d01744cd.
Philip Lombardi (plombardi89)
changed the title
go.mod keeps the agent library pin from Azure/unbounded#815 and takes main's Azure SDK and kubelogin updates.
Philip Lombardi (plombardi89)
had a problem deploying
to
e2e-testing
September 25, 2026 14:49 — with
GitHub Actions
Failure
GitHub Actions
Failure
A rejoin took the agent-upgrade path whenever a managed layout existed, but reset removes the service and the config directory and keeps the layout, so the config copy that follows failed. Under /usr/local the layout's directory was not readable by the SSH user, so the check never matched and every rejoin used install.sh. Under /opt/unbounded it matches. The upgrade path is now taken only while the service is installed.
Philip Lombardi (plombardi89)
deployed
to
e2e-testing
September 25, 2026 15:15 — with
GitHub Actions
Active
GitHub Actions
Active
Author
|
The host prefix is gone, to match Azure/unbounded#815. 6bec9c5 replaces it with the fixed host root:
12bdae6 merges main to clear a |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Azure Container Linux mounts
/usrread-only and is provisioned only by Ignition. This moves AKS Flex Node's host-side files to/opt/unboundedand adds an Ignition bootstrap path so it can run there.Depends on Azure/unbounded#815.
go.modpins a commit from that PR; it has to become a release before this merges.Host root
/opt/unboundedon every host:bin/aks-flex-node, the blue/green layout and recovery script underlib/aks-flex-node, and the agent library's helpers. The config, state, logs and units don't move./opt/unbounded/binisn't on the defaultPATH./usr/local.start, the daemon,agent-upgrade,recover-agent-upgradeand the nspawn hooks first link/opt/unboundedto it, so the units and links that release wrote stay valid and the host can go back to it. Reset anduninstall.shsweep both roots.host-rootcommand prints the root.install.shandbootstrap.shask the downloaded binary for it and install a release without it under/usr/local/binas before. An AgentUpgrade to such a release is refused on a host installed under/opt/unbounded.Ignition
aks-flex-node ignition [flags] -- BOOTSTRAP_ARGSrenders an Ignition config. It writesbootstrap.sh, with the base config embedded, plus any service principal credential, and enablesaks-flex-node-bootstrap.serviceto run it on first boot. The unit retries until the agent is installed and doesn't run after that. It deletes the token-bearing script once bootstrap succeeds, and reset disables and removes it. The arguments are checked againstbootstrap.sh's options before anything is rendered.Tests
bootstrap_test.shnow runs inmake test-installand PR checks, isolated from the host's/usr/localand/var/lib. The Azure e2e upgrades a node installed by v0.2.0 to the build under test, and checks that an AgentUpgrade to a release withouthost-rootis refused.hack/aclis a local harness that boots Azure Container Linux fromaks-flex-node ignitionoutput and joins it to kind.