Singularity images are duplicated

[CGuichard]

Overview

When using Toil with --batchSystem=slurm and --singularity the singularity ".sif" images are duplicated.

The docker image "debian:stretch-slim" results in the following ".sif" files:

debian:stretch-slim.sif
debian_stretch-slim.sif

The problem is the different separators used between the name an version.

By default cwltool --singularity produces debian:stretch-slim.sif, and Toil seems to duplicate it as debian_stretch-slim.sif. Maybe Toil should align with cwltool and use debian:stretch-slim.sif to avoid this problem?

Environment

• toil (toil[cwl]): 9.5.0
• cwltool: 3.2.20260413085819
• python: 3.12
• singularity: 4.1.5
• singularity cache: ~/.singularity

Reproduce

I will use the following CWL sleep.cwl:

#!/usr/bin/env cwl-runner
cwlVersion: v1.2
class: CommandLineTool
hints:
  DockerRequirement:
    dockerPull: debian:stretch-slim
baseCommand: sleep
arguments: []
inputs:
  duration:
    type: int
    default: 5
    inputBinding:
      position: 1
outputs: []

Execution with cwltool looks like this:

$ cwltool --singularity sleep.cwl --duration 1
INFO /home/cguichard/.venv/bin/cwltool 3.2.20260413085819
INFO Resolved 'sleep.cwl' to 'file:///home/cguichard/sleep.cwl'
INFO Using local copy of Singularity image debian:stretch-slim.sif found in /home/cguichard/.singularity
INFO [job sleep.cwl] /tmp/z70u9qsm$ singularity \
    --quiet \
    run \
    --contain \
    --ipc \
    --cleanenv \
    --no-eval \
    --userns \
    --home \
    /tmp/z70u9qsm:/dIQWRw \
    --mount=type=bind,source=/tmp/ncxmzuem,target=/tmp \
    --pwd \
    /dIQWRw \
    --net \
    --network \
    none \
    /home/cguichard/.singularity/debian:stretch-slim.sif \
    sleep \
    1
INFO [job sleep.cwl] completed success
{}INFO Final process status is success

In my singularity cache I have only one image: debian:stretch-slim.sif.

But when I use Toil I get another image.

$ toil-cwl-runner --singularity --jobStore .toil --batchSystem slurm --slurmArgs "--account ****" sleep.cwl
[2026-06-25T12:30:28+0000] [MainThread] [I] [cwltool] Resolved 'sleep.cwl' to 'file:///home/cguichard/sleep.cwl'
[2026-06-25T12:30:28+0000] [MainThread] [I] [toil.cwl.cwltoil] Prepulling the workflow's containers with Singularity...
Pulling debian:stretch-slim with Singularity...
INFO:cwl_utils.image_puller:Pulling debian:stretch-slim with Singularity...
Image successfully pulled: /home/cguichard/.singularity/debian_stretch-slim.sif
INFO:cwl_utils.image_puller:Image successfully pulled: /home/cguichard/.singularity/debian_stretch-slim.sif
[2026-06-25T12:30:30+0000] [MainThread] [I] [toil.cwl.cwltoil] Creating root job
[2026-06-25T12:30:30+0000] [MainThread] [I] [toil.lib.history] Recording workflow creation of c0753fef-1268-4732-8cab-ccd3ee3cbe9f in file:/home/cguichard/.toil
[2026-06-25T12:30:30+0000] [MainThread] [I] [toil.lib.history] Workflow c0753fef-1268-4732-8cab-ccd3ee3cbe9f is a run of sleep.cwl
[2026-06-25T12:30:31+0000] [MainThread] [I] [toil.cwl.cwltoil] Importing tool-associated files...
[2026-06-25T12:30:31+0000] [MainThread] [I] [toil.cwl.cwltoil] Importing input files...
[2026-06-25T12:30:31+0000] [MainThread] [I] [toil.cwl.cwltoil] Starting workflow
[2026-06-25T12:30:31+0000] [MainThread] [I] [toil] Running Toil version 9.5.0-855577413b6e3ceb33a90b3ec415e7a3fc0badba on host *****.
[2026-06-25T12:30:31+0000] [MainThread] [I] [toil.realtimeLogger] Starting real-time logging.

[2026-06-25T12:30:31+0000] [MainThread] [I] [toil.leader] Issued job 'CWLJob' sleep.cwl kind-CWLJob/instance-***** v1 with job batch system ID: 1 and disk: 3.0 Gi
, memory: 256.0 Mi, cores: 1, accelerators: [], preemptible: False
[2026-06-25T12:30:32+0000] [Thread-2] [I] [toil.batchSystems.abstractGridEngineBatchSystem] Job sleep.cwl with batch system ID 1 queued as job 6184995
[2026-06-25T12:30:33+0000] [MainThread] [I] [toil.leader] 0 jobs are running, 1 jobs are issued and waiting to run
[2026-06-25T12:31:13+0000] [Thread-5 (statsAndLoggingAggregator)] [I] [toil.statsAndLogging] Got message from job at time 06-25-2026 12:31:13: CWL step complete: sleep.cwl
[2026-06-25T12:31:20+0000] [MainThread] [I] [toil.leader] Finished toil run successfully.

Workflow Progress 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 (0 failures) [00:46<00:00, 0.02 jobs/s]
[2026-06-25T12:31:20+0000] [MainThread] [I] [toil.realtimeLogger] Stopping real-time logging server.
[2026-06-25T12:31:20+0000] [MainThread] [I] [toil.realtimeLogger] Joining real-time logging server thread.
[2026-06-25T12:31:20+0000] [MainThread] [I] [toil.cwl.cwltoil] Collecting workflow outputs...
[2026-06-25T12:31:20+0000] [MainThread] [I] [toil.cwl.cwltoil] Stored workflow outputs
[2026-06-25T12:31:20+0000] [MainThread] [I] [toil.cwl.cwltoil] Computing output file checksums...
{}
[2026-06-25T12:31:20+0000] [MainThread] [I] [toil.cwl.cwltoil] CWL run complete!
[2026-06-25T12:31:20+0000] [MainThread] [I] [toil.lib.history] Workflow c0753fef-1268-4732-8cab-ccd3ee3cbe9f stopped. Success: True
[2026-06-25T12:31:20+0000] [MainThread] [I] [toil.common] Successfully deleted the job store: FileJobStore(/home/cguichard/.toil)

And now my singularity cache contains debian_stretch-slim.sif.

To summarize, with cwltool there is only one ".sif" but with Toil we always end up with both, even when clearing the cache empty. I think the first with ":" separator is pulled with cwltool and then Toil ask for the one with "_". Thanks to the cache the second time do not trigger the download, but disk usage is still doubled.

┆Issue is synchronized with this Jira Story
┆Issue Number: TOIL-1848

[adamnovak]

This is because cwltool and the pre-pulling utility Toil calls from cwl-utils don't agree on the cache filenames to use.

I have common-workflow-language/cwltool#2284 and common-workflow-language/cwl-utils#433 open to fix that, but I need to do more work to get the upstream projects to take them.