Fix conda env dependency drift, add a blast stub and smoke test, and detect failed blastp calls

[mrubash1]

Follow-on to #100 (now merged). Seven fixes found while running the pipeline locally on macOS and on x86-64 Linux. None of them are platform-specific; each was reproduced on both.

1. Pin matplotlib and setuptools in envs/analysis.yml

envs/analysis.yml pins its seven direct dependencies but nothing transitive, so a fresh solve installs versions incompatible with its pinned numpy=1.23.5, and two rules fail:

rule leiden_clustering (scanpy -> matplotlib):
  ImportError: Matplotlib requires numpy>=1.25; you have 1.23.5

rule dim_reduction (umap-learn 0.5.3):
  ModuleNotFoundError: No module named 'pkg_resources'

setuptools >= 81 removes pkg_resources, which umap-learn 0.5.3 imports at module scope. The matplotlib failure is an ABI break rather than a dependency conflict: the solver installs a numpy-2-built matplotlib alongside numpy=1.23.5 without complaint, and it fails only when imported — which is why building the environment never caught it, and why an environment that pins numpy has to pin what is compiled against it. The matplotlib pin matches the one already in plotting.yml and cartography_pub.yml.

This is not platform-specific: a fresh solve produced matplotlib 3.11.0 on osx-64 and 3.9.4 on linux-64, and the pipeline failed identically on both.

2. Remove the restore-keys from the conda env cache

restore-keys began with snakemake-conda- while the key begins with conda-, and they are matched as a prefix of the key, so the fallback restore could never fire.

An earlier revision of this branch corrected the prefix. That was wrong, and the correction is reversed here: a restore-keys match still leaves cache-hit false, so the env creation step runs regardless, and that step begins with rm -rf .snakemake/conda. Making the prefix match would therefore download a stale multi-gigabyte cache on every change to envs/*.yml, only to delete it — strictly slower than a prefix that never matched. There is no path in which the restored files are used, so the keys are removed instead.

This cache is also why CI did not catch the drift in (1): the key includes hashFiles('envs/*.yml'), so while the env files are unchanged CI restores previously solved envs and never re-resolves them against the current package index. That is worth addressing separately — a scheduled job that re-solves from scratch would catch this class of drift before it reaches a PR.

3. Add a blast stub and a make smoke-test entry point

The run_blast rule calls blastp -remote, which queues on NCBI's public servers; NCBI's own time-to-completion estimate (RTOE) was observed at ~2.5 hours, which makes end-to-end verification impractical.

run_blast now copies a canned results file when PROTEINCARTOGRAPHY_BLAST_STUB_RESULTS_FILEPATH is set. An env variable is used rather than a CLI flag because snakemake rule environments inherit env variables from the calling process, so neither the Snakefile nor the config needs to change.

This replaces tests.mocks.mock_run_blast, which patched blast_utils.run_blast and so required run_blast.py to import the tests package — the same class of problem #100 fixes, since importing tests.mocks calls find_repo_dirpath().

make smoke-test runs the search-mode pipeline with the stub. That test also now asserts on snakemake's return value, which was previously discarded (so it could pass while the pipeline failed), and checks that outputs are non-empty rather than merely present.

4. Detect failed blastp calls that exit with a status of zero

When the remote server refuses to queue a request, blastp -remote reports the error on stderr, writes an empty results file, and still exits zero:

$ blastp -remote -db nr ... ; echo $?
Error: [blastp] bad_request: Could not queue request: DB operation failed.
0

run_blast.py branched on result.returncode == 0, so it treated this as success. The word-size backoff was never attempted — the situation it exists for — and the empty file surfaced later as a misleading "no hits were returned" from extract_blast_hits.py.

An empty results file is deliberately not treated as a failure, since a query that legitimately has no hits produces one too.

This does not make the remote call faster; it means a refused request is retried and then reported.

5. Pin mamba in envs/cartography_test.yml

mamba was left unpinned and now resolves to mamba 2.x, which removed the mamba env create CLI that snakemake 7.25.3 invokes. --conda-frontend mamba therefore fails with a CreateCondaEnvironmentException and no output from the frontend.

Pinned to 1.4.2, matching cartography_tidy.yml. CI does not hit this because test.yml forces the conda frontend, with a comment noting the mamba frontend "results in errors during env creation" — this is that error, and this is its cause. Anyone using the mamba frontend locally does hit it.

Same class of problem as (1): a dependency left unpinned in 2023 that resolves to an incompatible major version today.

6. Build the wildcard output paths without f-strings

Two rules built an output path from an f-string mixing an interpolated value with an escaped snakemake wildcard:

f"{ANALYSIS_NAME}_aggregated_features_{{plotting_mode}}.html"

The doubled braces render the literal {plotting_mode} that snakemake needs as a wildcard. That is easy to misread, and a formatter can silently change its meaning: make format under Python 3.12+ rewrites the doubled braces to single ones, turning the wildcard into an interpolation of a name that does not exist. (Python 3.12 changed f-string tokenization, PEP 701; the pinned snakefmt and black predate it.)

The resulting Snakefile would still pass lint and would fail at runtime with a NameError in plot_interactive and plot_cluster_distributions.

Concatenating a plain string avoids the escaping, produces a byte-identical path, and cannot be rewritten. Verified that the DAG resolves the same targets (..._aggregated_features_pca_umap.html, ..._P60709_distribution_analysis.svg) and that snakefmt --check now passes under both Python 3.9 (CI's version) and Python 3.12+, where it previously wanted to rewrite the file.

The doubled braces elsewhere in the Snakefile are unaffected: they appear in a plain (non-f) string and in a shell block, neither of which a formatter rewrites.

7. Pin python in the environments that left it unpinned

envs/analysis.yml, envs/plotting.yml and envs/pandas.yml declare no python at all, so each resolves to whatever the newest python conda-forge has builds for. They had drifted apart from each other and from the six environments that do pin it, which are all on 3.9.16: solved as they were, analysis.yml gave python 3.11 and plotting.yml gave 3.10.20.

This is what put analysis.yml on a python new enough to resolve the setuptools >= 81 that removes pkg_resources — the failure in (1). Pinning the interpreter is the more direct fix for that class of drift than pinning each package that reacts to it.

All three now solve at 3.9.16.

Verification

• All three newly-pinned environments solve at python 3.9.16, checked from a snapshot of the env files. The analysis environment imports matplotlib.pyplot, pkg_resources and umap — the three imports that were failing.
• make test: 11 passed (both pipeline integration tests plus the unit tests)
• make smoke-test: 26 of 26 rules complete (~49 s with envs cached)
• Cluster-mode demo end to end: 16 of 16 rules, all final_results outputs produced
• ruff check, ruff format --check, and snakefmt --check: clean, using CI's pinned versions (ruff 0.1.6, snakefmt 0.8.5) under Python 3.9
• Fix pip install . and importing package modules outside a git working tree #100's packaging tests independently confirmed passing on linux/amd64 (Modal)

Not included

A replacement for the remote BLAST interface is proposed separately in #102, which is stacked on this branch. Benchmarking found that the slowness tracks NCBI's global queue load rather than any of the pipeline's blastp arguments — -word_size, -outfmt, -max_target_seqs, and shell=True were all ruled out by probing NCBI's RTOE estimate against a control. This PR deliberately keeps blastp -remote and only makes its failures visible (4).

[mrubash1]

Closing in favour of #108, which is this branch cut down after #103 landed.

Dropped from this PR:

• the analysis.yml matplotlib and setuptools pins — Fix search-mode reliability for BLAST, UniProt, AFDB, and small maps #103 took them verbatim, so they are already on main
• the blast stub and make smoke-test — main now has extended mocks plus test_pipeline_in_search_mode, which covers the same ground, and the stub was the piece that conflicted hardest with the rewritten blast_utils
• the python=3.9.16 pins — CI is green on main without them, and they are the prime suspect for plotting.yml env creation going from ~74s to over 12 minutes without finishing. That is unconfirmed, so it belongs in its own PR where the CI timing is the evidence rather than bundled with unrelated fixes.

Carried over to #108: the conda cache restore-keys removal, the mamba pin, the Snakefile f-string fix, and the blastp exit-zero detection — all four verified as still needed against the current main.