[WIP] Compatibility layer for EESSI version 2026.x

[bedroge]

Still WIP, but testing some new functionality already...

[bedroge]

bot: build repo:eessi.io-2025.06-compat arch:x86_64/generic

[bedroge]

bot: build repo:eessi.io-2025.06-compat instance:eessi-bot-mc-aws for:arch=x86_64/generic

[eessi-bot-aws]

New job on instance eessi-bot-mc-aws for repository eessi.io-2025.06-compat
Building on: generic
Building for: x86_64/generic
Job dir: /project/def-users/SHARED/jobs/2026.05/pr_235/156325

date	job status	comment
May 11 19:48:57 UTC 2026	submitted	job id 156325 awaits release by job manager
May 11 19:49:15 UTC 2026	released	job awaits launch by Slurm scheduler
May 11 19:55:17 UTC 2026	running	job 156325 is running
May 11 19:57:20 UTC 2026	finished	😢 FAILURE (click triangle for details) Details ✅ job output file slurm-156325.out ❌ some task failed ❌ no tarball found Artefacts Details
May 11 19:57:20 UTC 2026	test result	😢 FAILURE (click triangle for details) Reason Failed for unknown reason Details ✅ job output file slurm-156325.out ✅ no message matching ERROR: ✅ no message matching [\s*FAILED\s*].*Ran .* test case

[bedroge]

bot: build repo:eessi.io-2025.06-compat instance:eessi-bot-mc-aws for:arch=x86_64/generic

[eessi-bot-aws]

New job on instance eessi-bot-mc-aws for repository eessi.io-2025.06-compat
Building on: generic
Building for: x86_64/generic
Job dir: /project/def-users/SHARED/jobs/2026.05/pr_235/156326

date	job status	comment
May 11 21:20:19 UTC 2026	submitted	job id 156326 awaits release by job manager
May 11 21:20:30 UTC 2026	released	job awaits launch by Slurm scheduler
May 11 21:25:33 UTC 2026	running	job 156326 is running
May 12 00:33:38 UTC 2026	finished	😢 FAILURE (click triangle for details) Details ✅ job output file slurm-156326.out ❌ some task failed ✅ found tarball Artefacts eessi-2026.06-compat-linux-x86_64-1778545846.tar.gz size: 1316 MiB (1380624192 bytes) entries: 183549
May 12 00:33:38 UTC 2026	test result	😢 FAILURE (click triangle for details) Reason Failed for unknown reason Details ✅ job output file slurm-156326.out ✅ no message matching ERROR: ✅ no message matching [\s*FAILED\s*].*Ran .* test case

[bedroge]

bot: build repo:eessi.io-2025.06-compat instance:eessi-bot-mc-aws for:arch=x86_64/generic

[eessi-bot-aws]

New job on instance eessi-bot-mc-aws for repository eessi.io-2025.06-compat
Building on: generic
Building for: x86_64/generic
Job dir: /project/def-users/SHARED/jobs/2026.05/pr_235/156327

date	job status	comment
May 12 07:28:32 UTC 2026	submitted	job id 156327 awaits release by job manager
May 12 07:29:10 UTC 2026	released	job awaits launch by Slurm scheduler
May 12 07:35:13 UTC 2026	running	job 156327 is running
May 12 07:56:42 UTC 2026	finished	😢 FAILURE (click triangle for details) Details ✅ job output file slurm-156327.out ❌ some task failed ✅ found tarball Artefacts eessi-2026.06-compat-linux-x86_64-1778572503.tar.gz size: 389 MiB (408295890 bytes) entries: 173293
May 12 07:56:42 UTC 2026	test result	😢 FAILURE (click triangle for details) Reason EESSI test suite was not run, test step itself failed to execute. Details ✅ job output file slurm-156327.out ❌ found message matching ERROR: ✅ no message matching [\s*FAILED\s*].*Ran .* test case

[bedroge]

bot: build repo:eessi.io-2025.06-compat instance:eessi-bot-mc-aws for:arch=x86_64/generic

[eessi-bot-aws]

New job on instance eessi-bot-mc-aws for repository eessi.io-2025.06-compat
Building on: generic
Building for: x86_64/generic
Job dir: /project/def-users/SHARED/jobs/2026.05/pr_235/156328

date	job status	comment
May 12 08:17:53 UTC 2026	submitted	job id 156328 awaits release by job manager
May 12 08:18:48 UTC 2026	released	job awaits launch by Slurm scheduler
May 12 08:19:51 UTC 2026	running	job 156328 is running
May 12 14:40:14 UTC 2026	finished	😢 FAILURE (click triangle for details) Details ✅ job output file slurm-156328.out ❌ some task failed ✅ found tarball Artefacts eessi-2026.06-compat-linux-x86_64-1778596658.tar.gz size: 1327 MiB (1391465593 bytes) entries: 179430
May 12 14:40:14 UTC 2026	test result	😢 FAILURE (click triangle for details) Reason Failed for unknown reason Details ✅ job output file slurm-156328.out ✅ no message matching ERROR: ✅ no message matching [\s*FAILED\s*].*Ran .* test case

[bedroge]

bot: build repo:eessi.io-2025.06-compat instance:eessi-bot-mc-aws for:arch=x86_64/generic

[eessi-bot-aws]

New job on instance eessi-bot-mc-aws for repository eessi.io-2025.06-compat
Building on: generic
Building for: x86_64/generic
Job dir: /project/def-users/SHARED/jobs/2026.05/pr_235/156329

date	job status	comment
May 12 15:00:39 UTC 2026	submitted	job id 156329 awaits release by job manager
May 12 15:01:20 UTC 2026	released	job awaits launch by Slurm scheduler
May 12 15:02:23 UTC 2026	running	job 156329 is running
May 12 19:02:42 UTC 2026	finished	😢 FAILURE (click triangle for details) Details ✅ job output file slurm-156329.out ❌ some task failed ✅ found tarball Artefacts eessi-2026.06-compat-linux-x86_64-1778612402.tar.gz size: 1442 MiB (1512120372 bytes) entries: 196326
May 12 19:02:42 UTC 2026	test result	😢 FAILURE (click triangle for details) Reason EESSI test suite was not run, test step itself failed to execute. Details ✅ job output file slurm-156329.out ❌ found message matching ERROR: ✅ no message matching [\s*FAILED\s*].*Ran .* test case

[bedroge]

bot: build repo:eessi.io-2025.06-compat instance:eessi-bot-mc-aws for:arch=x86_64/generic

[eessi-bot-aws]

New job on instance eessi-bot-mc-aws for repository eessi.io-2025.06-compat
Building on: generic
Building for: x86_64/generic
Job dir: /project/def-users/SHARED/jobs/2026.05/pr_235/159927

date	job status	comment
May 24 10:11:54 UTC 2026	submitted	job id 159927 awaits release by job manager
May 24 10:12:50 UTC 2026	released	job awaits launch by Slurm scheduler
May 24 10:18:53 UTC 2026	running	job 159927 is running
May 24 18:17:21 UTC 2026	finished	😢 FAILURE (click triangle for details) Details ✅ job output file slurm-159927.out ❌ some task failed ✅ found tarball Artefacts eessi-2026.06-compat-linux-x86_64-1779646381.tar.gz size: 2420 MiB (2537696316 bytes) entries: 290359
May 24 18:17:21 UTC 2026	test result	😢 FAILURE (click triangle for details) Reason EESSI test suite was not run, test step itself failed to execute. Details ✅ job output file slurm-159927.out ❌ found message matching ERROR: ✅ no message matching [\s*FAILED\s*].*Ran .* test case

[bedroge]

bot: build repo:eessi.io-2025.06-compat instance:eessi-bot-mc-aws for:arch=x86_64/generic

[eessi-bot-aws]

New job on instance eessi-bot-mc-aws for repository eessi.io-2025.06-compat
Building on: generic
Building for: x86_64/generic
Job dir: /project/def-users/SHARED/jobs/2026.05/pr_235/160214

date	job status	comment
May 25 06:08:13 UTC 2026	submitted	job id 160214 awaits release by job manager
May 25 06:09:16 UTC 2026	released	job awaits launch by Slurm scheduler
May 25 06:15:18 UTC 2026	running	job 160214 is running
May 25 14:11:47 UTC 2026	finished	😢 FAILURE (click triangle for details) Details ✅ job output file slurm-160214.out ❌ some task failed ✅ found tarball Artefacts eessi-2026.06-compat-linux-x86_64-1779718108.tar.gz size: 1965 MiB (2061270859 bytes) entries: 207064
May 25 14:11:47 UTC 2026	test result	😢 FAILURE (click triangle for details) Reason Failed for unknown reason Details ✅ job output file slurm-160214.out ✅ no message matching ERROR: ✅ no message matching [\s*FAILED\s*].*Ran .* test case

[bedroge]

bot: build repo:eessi.io-2025.06-compat instance:eessi-bot-mc-aws for:arch=x86_64/generic

[eessi-bot-aws]

New job on instance eessi-bot-mc-aws for repository eessi.io-2025.06-compat
Building on: generic
Building for: x86_64/generic
Job dir: /project/def-users/SHARED/jobs/2026.05/pr_235/160229

date	job status	comment
May 25 17:56:22 UTC 2026	submitted	job id 160229 awaits release by job manager
May 25 17:57:07 UTC 2026	released	job awaits launch by Slurm scheduler
May 25 18:02:10 UTC 2026	running	job 160229 is running
May 25 21:58:30 UTC 2026	finished	😢 FAILURE (click triangle for details) Details ✅ job output file slurm-160229.out ❌ some task failed ✅ found tarball Artefacts eessi-2026.06-compat-linux-x86_64-1779746079.tar.gz size: 1959 MiB (2054750782 bytes) entries: 206124
May 25 21:58:30 UTC 2026	test result	😢 FAILURE (click triangle for details) Reason Failed for unknown reason Details ✅ job output file slurm-160229.out ✅ no message matching ERROR: ✅ no message matching [\s*FAILED\s*].*Ran .* test case

[bedroge]

bot: build repo:eessi.io-2025.06-compat instance:eessi-bot-mc-aws for:arch=x86_64/generic

[eessi-bot-aws]

New job on instance eessi-bot-mc-aws for repository eessi.io-2025.06-compat
Building on: generic
Building for: x86_64/generic
Job dir: /project/def-users/SHARED/jobs/2026.05/pr_235/160520

date	job status	comment
May 26 08:09:38 UTC 2026	submitted	job id 160520 awaits release by job manager
May 26 08:10:17 UTC 2026	released	job awaits launch by Slurm scheduler
May 26 08:11:20 UTC 2026	running	job 160520 is running
May 26 12:08:19 UTC 2026	finished	😢 FAILURE (click triangle for details) Details ✅ job output file slurm-160520.out ❌ some task failed ✅ found tarball Artefacts eessi-2026.06-compat-linux-x86_64-1779797095.tar.gz size: 1957 MiB (2052887348 bytes) entries: 205922
May 26 12:08:19 UTC 2026	test result	😢 FAILURE (click triangle for details) Reason EESSI test suite was not run, test step itself failed to execute. Details ✅ job output file slurm-160520.out ❌ found message matching ERROR: ✅ no message matching [\s*FAILED\s*].*Ran .* test case

[bedroge]

It got all the way to the end, but the tests failed when trying to install ReFrame due to DNS issues. This is because we had already replaced /etc/resolv.conf by a variant symlink, but that doesn't work in the container, meaning that it's an invalid symlink. Not sure how to fix that, other than skipping the tests in the playbook, and running them in the actual test step of the bot inside a container with a CVMFS mount (and, hence, actually using the variant symlinks). That will require a bit of work though.

[boegel]

It got all the way to the end, but the tests failed when trying to install ReFrame due to DNS issues. This is because we had already replaced /etc/resolv.conf by a variant symlink, but that doesn't work in the container, meaning that it's an invalid symlink. Not sure how to fix that, other than skipping the tests in the playbook, and running them in the actual test step of the bot inside a container with a CVMFS mount (and, hence, actually using the variant symlinks). That will require a bit of work though.

Isn't there a default target for that variant symlink? There definitely should be...
Why doesn't it work in a container?
Can't we bind-mount the /etc/resolv.conf from the host?

[bedroge]

It got all the way to the end, but the tests failed when trying to install ReFrame due to DNS issues. This is because we had already replaced /etc/resolv.conf by a variant symlink, but that doesn't work in the container, meaning that it's an invalid symlink. Not sure how to fix that, other than skipping the tests in the playbook, and running them in the actual test step of the bot inside a container with a CVMFS mount (and, hence, actually using the variant symlinks). That will require a bit of work though.

Isn't there a default target for that variant symlink? There definitely should be... Why doesn't it work in a container? Can't we bind-mount the /etc/resolv.conf from the host?

The issue is that these variant symlinks are a CVMFS feature. You can create them without CVMFS, but then the target is just a weird-looking string / non-existing path like resolv.conf -> '$(EESSI_COMPAT_ETC_RESOLV_CONF):-/etc/resolv.conf'. So you'd really need the CVMFS client to turn that into an actual symlink, and this is done at mount time. I was hoping that I could make them work by adding them to the overlay-upper directory and launching the EESSI container, but obviously that will not work as they're not actually part of the CVMFS repo that way.

I tried looking for some other FUSE-based tool to use variant symlinks in Apptainer, and found for instance https://github.com/cccheng/varlinkfs. But the ones that I found won't work in the way we need it to work. Varlinkfs, for instance, only allows for a single directory, and the naming of the variant symlinks is different.

What I could do in the Ansible playbook is to first make regular symlinks by stripping the $(EESSI_COMPAT_ETC_RESOLV_CONF):- part, and overwrite them with variant symlinks after the tests have run. I don't fully like that, but I can't really think of a better and easy solution.

[bedroge]

bot: build repo:eessi.io-2025.06-compat instance:eessi-bot-mc-aws for:arch=x86_64/generic

[eessi-bot-aws]

New job on instance eessi-bot-mc-aws for repository eessi.io-2025.06-compat
Building on: generic
Building for: x86_64/generic
Job dir: /project/def-users/SHARED/jobs/2026.05/pr_235/161100

date	job status	comment
May 28 13:29:54 UTC 2026	submitted	job id 161100 awaits release by job manager
May 28 13:30:53 UTC 2026	released	job awaits launch by Slurm scheduler
May 28 13:31:57 UTC 2026	running	job 161100 is running
May 28 17:30:53 UTC 2026	finished	😢 FAILURE (click triangle for details) Details ✅ job output file slurm-161100.out ❌ some task failed ✅ found tarball Artefacts eessi-2026.06-compat-linux-x86_64-1779989158.tar.gz size: 2382 MiB (2498289369 bytes) entries: 283543
May 28 17:30:53 UTC 2026	test result	😢 FAILURE (click triangle for details) Reason Failed for unknown reason Details ✅ job output file slurm-161100.out ✅ no message matching ERROR: ✅ no message matching [\s*FAILED\s*].*Ran .* test case

[bedroge]

It got all the way to the end, but the tests failed when trying to install ReFrame due to DNS issues. This is because we had already replaced /etc/resolv.conf by a variant symlink, but that doesn't work in the container, meaning that it's an invalid symlink. Not sure how to fix that, other than skipping the tests in the playbook, and running them in the actual test step of the bot inside a container with a CVMFS mount (and, hence, actually using the variant symlinks). That will require a bit of work though.

Isn't there a default target for that variant symlink? There definitely should be... Why doesn't it work in a container? Can't we bind-mount the /etc/resolv.conf from the host?

The issue is that these variant symlinks are a CVMFS feature. You can create them without CVMFS, but then the target is just a weird-looking string / non-existing path like resolv.conf -> '$(EESSI_COMPAT_ETC_RESOLV_CONF):-/etc/resolv.conf'. So you'd really need the CVMFS client to turn that into an actual symlink, and this is done at mount time. I was hoping that I could make them work by adding them to the overlay-upper directory and launching the EESSI container, but obviously that will not work as they're not actually part of the CVMFS repo that way.

I tried looking for some other FUSE-based tool to use variant symlinks in Apptainer, and found for instance https://github.com/cccheng/varlinkfs. But the ones that I found won't work in the way we need it to work. Varlinkfs, for instance, only allows for a single directory, and the naming of the variant symlinks is different.

What I could do in the Ansible playbook is to first make regular symlinks by stripping the $(EESSI_COMPAT_ETC_RESOLV_CONF):- part, and overwrite them with variant symlinks after the tests have run. I don't fully like that, but I can't really think of a better and easy solution.

For now I've solved it by creating regular symlinks right after a new Prefix is built, and replace them by variant symlinks at the very end. That should probably fix it.

I've also asked an AI friend if it would be possible to implement a FUSE-based tool that would allow us to use the actual variant symlinks. It actually came up with a Python script that does the trick and mounts a directory with variant symlinks at a given mount point (read-only though), where the variant symlinks can be configured by setting environment variables (before mounting, and you can only modify them by remounting). We could take that approach as well, and do the read-only mount for the bot's test step. But I found it a bit cumbersome, so I'm testing the other approach first.

[bedroge]

Last build failed due to an download issue for a elfx86exts dependency (https://crates.io/api/v1/crates/anstyle-parse/0.2.2/download). Let's try again...

bot: build repo:eessi.io-2025.06-compat instance:eessi-bot-mc-aws for:arch=x86_64/generic

[eessi-bot-aws]

New job on instance eessi-bot-mc-aws for repository eessi.io-2025.06-compat
Building on: generic
Building for: x86_64/generic
Job dir: /project/def-users/SHARED/jobs/2026.05/pr_235/161102

date	job status	comment
May 28 17:47:25 UTC 2026	submitted	job id 161102 awaits release by job manager
May 28 17:47:59 UTC 2026	released	job awaits launch by Slurm scheduler
May 28 17:49:02 UTC 2026	running	job 161102 is running