Spack package repository for Muon Collider software stack

Note: the main branch is now used for the 3.x release series.

This repository holds a set of Spack recipes for Muon Collider software (under namespace mucoll) based on Key4hep stack. It is built on top of the key4hep-dev-external environment from the key4hep-stack repository, which is required for installation.

See doc/ReleaseNotes.md for the changelog of tagged releases and the work in progress on the 3.x series.

After installing Spack and downloading the key4hep-spack and mucoll-spack repositories, the whole software stack can be installed using the following commands:

# Setup spack
spack repo add ./key4hep-spack
spack repo add ./mucoll-spack
spack env activate ./mucoll-spack/environments/mucoll-layered
spack concretize --reuse
# Install a layer. The analysis layer (edm4hep/podio + ML + python tools) is the base:
spack install --only-concrete --no-add --fail-fast mucoll-stack+devtools+pytools+ml
# ...or add +sim for the full reconstruction + simulation stack on top:
spack install --only-concrete --no-add --fail-fast mucoll-stack+devtools+pytools+ml+sim

# Load the Muon Collider environment
source $MUCOLL_STACK

Setting up the environment

When signing in to a machine with the installed sofware stack (VM or Docker container), it has to be loaded into the environment:

spack env activate ./mucoll-spack/environments/mucoll-layered
source $MUCOLL_STACK

Package versioning

Preferred convention for version names in Spack is numbers separated by dots, without leading zeros, e.g. 1.2.13.

Muon Collider forks of ILCSoft packages historically used dashed, zero-padded tags with an -MC suffix (e.g. version 0.2.2 maps to tag v00-02-02-MC); the conversion helper MCIlcsoftpackage in packages/mucoll-stack/mucoll_utils.py performs this mapping. Since the Marlin/ILCSoft chain was dropped in the 3.x series (see the release notes), the stack no longer ships any package that relies on this convention — the remaining mucoll packages (e.g. k4reco) declare plain version(...) entries with explicit checksums.

Adding new versions for individual packages

After a new tag for the package is created, e.g. v00-02-00 in the k4Reco repository, it can be added to this Spack repository in two steps:

1. Get the archive checksum for the new tag

spack checksum k4reco 0.2
# Validates archive URL and returns the checksum
    version('0.2', sha256='a5b02425b6970777f9f2982fd2907d38599c00996d24ff0be839a0e315509cd4')

2. Add the returned version definition to the corresponding package file: packages/k4reco/package.py

NOTE: This repository only contains packages maintained by the Muon Collider collaboration. If the version of interest is missing from Spack for some other package, the line with a new version definition should be added to the package file in the corresponding repository.
To see locations of other repositories: spack repo list

Creating a new stack release

To introduce a new release version for the whole software stack, update the version number in packages/mucoll-stack/package.py and then update versions of all the relevant packages in environments/mucoll-common/packages.yaml.
Test this new configuration in a fresh environment:

# Create a development environment
spack env create dev ./mucoll-spack/environments/mucoll-layered/spack.yaml
spack env activate dev

# Add stack with updated version to the environment
spack add mucoll-stack

# Check which packages would be installed
spack spec --reuse -NIt

Packages that are already installed in the mucoll-layered environment are known to Spack and will be reused, providing a clear indication of which part of the dependency tree will be modified by the new release.

Modifying an existing package

It is possible to modify code of one or more packages using the development workflow, which lets Spack build it from your own source folder and use this custom version instead of the one installed in the release.

NOTE: This should primarily be used for simple code changes in a few packages that do not affect the overall build process and dependency tree of other packages in the release. For more global changes it's better to set up a new release.

Assume that we want to make changes in the k4reco package, which other parts of the reconstruction stack depend on.

To leave the original release untouched it is preferable to create a new Spack environment, e.g. called dev_k4reco, using the .lock file of the original environment as a starting point:

# Create a new environment
spack env create dev_k4reco $SPACK_ENV/spack.lock

# Activate the development environment
spack env activate dev_k4reco

The general procedure to replace a package with a custom version is the following:

1. put your new source code in a development folder of your choice, e.g. /opt/dev/k4Reco;
2. find the exact spec of this package in the release and mark it for development in the folder with the new source code;
3. reconcretize the environment to replace the default version of the package with the development one;
4. rebuild the modified package using spack install <package>;
5. rebuild the rest of the release, which will reinstall all the packages that depend on the one you've modified.

You can repeat the last 2 steps each time you modify the source code again.

# Create the development folder
mkdir -p /opt/dev/k4Reco

# Download the original source code (and modify it)
git clone https://github.com/MuonColliderSoft/k4Reco.git --branch main

# Find the exact spec of this package in the current release
spack find k4reco  # k4reco@main

# Mark the package with this spec for development
spack develop -p /opt/dev/k4Reco k4reco@main

# Reconcretize the environment
spack concretize -f --reuse

# Build the modified package
spack install k4reco

# Build the rest of the release
spack install

NOTE: The package spec you mark for development must match exactly the one in the release, even if the actual code comes from a different version of the package. Otherwise you'll have to modify the packages configuration in the relase to properly include the new spec in the dependency tree.

To return to the original version of the release:

# Deactivate the current environment (if on lxplus)
spack env deactivate
# Activate the default environment (if in a Docker container)
spack env activate mucoll-layered

Layered images

The CI builds two images as a layered chain, each built on top of the previous one so packages are installed once and reused down the chain:

• ${REPOSITORY}/mucoll-analysis-${OS}:${VERSION}: the analysis stack, including the machine-learning and python tools (mucoll-stack~devtools+pytools+ml~sim). This is the base layer.
• ${REPOSITORY}/mucoll-sim-${OS}:${VERSION}: built on mucoll-analysis, adds the reconstruction and simulation stack (+sim).

There is no separate reco or ml image: the +sim variant covers both reconstruction and simulation, and the machine-learning tools are now part of the base analysis layer.

Physics validation

On every push to main the CI runs a per-particle sim -> digi -> reco -> plot chain (see validation/ and .github/workflows/physics-validation-template.yaml) against the freshly published mucoll-sim image and publishes the resulting performance plots to GitHub Pages:

https://muoncollidersoft.github.io/mucoll-spack/

The landing page links to one gallery per particle (muon, electron, pion, photon); each gallery is keyed by the analysis settings used to produce it.