DM-56097: Improve mremove/mexists performance

.github/workflows/build.yaml

@@ -13,7 +13,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ["3.11", "3.12", "3.13", "3.14"]
+        python-version: ["3.11", "3.12", "3.13", "3.14", "3.15", "3.15t"]
 
     steps:
       - uses: actions/checkout@v7
@@ -40,6 +40,12 @@ jobs:
           uv pip install --system cryptography
 
       - name: Install WebDAV packages for testing
+        # wsgidav needs bcrypt, which publishes no free-threaded wheel for
+        # this Python and cannot be built from source because the pyo3 its
+        # Rust extension uses refuses a Python newer than 3.13 and cannot
+        # fall back to the limited API on a free-threaded build. The WebDAV
+        # tests skip when wsgidav is missing.
+        if: matrix.python-version != '3.15t'
         run: |
           uv pip install --system cheroot wsgidav
 

doc/changes/DM-56097.bugfix.rst

@@ -0,0 +1 @@
+``clean_test_environment_for_s3()`` now also clears ``$AWS_REQUEST_CHECKSUM_CALCULATION`` and ``$AWS_RESPONSE_CHECKSUM_VALIDATION``, so that a site which disables checksums does not change the object metadata seen by tests.

doc/changes/DM-56097.perf.rst

@@ -0,0 +1,6 @@
+``ResourcePath.mremove()``, ``ResourcePath.mexists()`` and ``ResourcePath.mtransfer()`` now send a batch of URIs to each worker rather than submitting one task per URI.
+The batch size is fixed per scheme rather than derived from how many URIs a call is given, so a batch never grows large enough that one worker drawing a run of slow URIs stalls the operation with no way to rebalance.
+A scheme sets it from the cost of one of its operations: a local file check is cheap enough that a batch has to reach a thousand URIs before threading it beats a loop in the calling thread, while a scheme whose every operation is a network round trip is worth overlapping immediately.
+Transfers batch separately from existence checks and removals, since a transfer costs far more and scales with a file size that is not known in advance.
+The upper bound on the number of workers is now set per scheme, so a scheme that holds no connection pool can raise it, and ``ResourcePath.mtransfer()`` now uses the bound for the target scheme instead of the global default.
+A batch of URIs that fits in a single chunk is handled in the calling thread rather than being given to a pool.

doc/changes/DM-56097.removal.rst

@@ -0,0 +1,3 @@
+Removed support for the ``$LSST_RESOURCES_EXECUTOR`` environment variable.
+Bulk operations now always use a thread pool.
+Process pools were measured to be no faster for remote URIs and they silently discarded the undo actions registered by ``ResourcePath.mtransfer()`` on a caller's transaction, since those were recorded in the worker process and never reached the parent.