CCTools: Add Deterministic Bucketing Algorithms (det-greedy, det-exhaust)

[tphung3]

Proposed Changes

This PR implements the deterministic version of the bucketing algorithms (greedy and exhaustive) and integrates them into WorkQueue and TaskVine.

This PR also introduces these minor changes:

• The bucketing code in general is refactored to facilitate implementing deterministic bucketing algorithms.
• Quantized Bucketing is added as another allocation option.
• Fix the problem where the list of shell env variables are not set correctly in work_queue_factory.c.
• Change the type of fields in struct vine_resource and struct work_queue_resource from int64_t to double to correctly track cpu usage. For example, before this change, a cpu usage of 1.5 is floored to 1 due to type casting to in64_t. Now the usage is correctly set of (double) 1.5.

Merge Checklist

The following items must be completed before PRs can be merged.
Check these off to verify you have completed all steps.

• make test Run local tests prior to pushing.
• make format Format source code to comply with lint policies. Note that some lint errors can only be resolved manually (e.g., Python)
• make lint Run lint on source code prior to pushing.
• Manual Update: Update the manual to reflect user-visible changes.
• Type Labels: Select a github label for the type: bugfix, enhancement, etc.
• Product Labels: Select a github label for the product: TaskVine, Makeflow, etc.
• PR RTM: Mark your PR as ready to merge.

[Copilot]

Pull request overview

This PR adds deterministic versions of the greedy and exhaustive bucketing algorithms (det-greedy and det-exhaust) to CCTools, integrating them into WorkQueue, TaskVine, and Makeflow. The implementation extracts common code into shared modules and extends the existing bucketing framework with new allocation modes.

Key Changes:

• Implements deterministic greedy and exhaustive bucketing algorithms as alternatives to probabilistic versions
• Refactors existing bucketing code to extract common functionality into shared modules
• Adds new allocation modes across WorkQueue and TaskVine APIs

Reviewed changes

Copilot reviewed 31 out of 31 changed files in this pull request and generated 15 comments.

Show a summary per file

File	Description
dttools/src/bucketing_det_greedy.c/h	New deterministic greedy bucketing algorithm implementation
dttools/src/bucketing_det_exhaust.c/h	New deterministic exhaustive bucketing algorithm implementation
dttools/src/bucketing_greedy_common.c/h	Extracted common code from greedy bucketing for reuse
dttools/src/bucketing_exhaust_common.c/h	Extracted common code from exhaustive bucketing for reuse
dttools/src/bucketing.c	Integrates new deterministic modes with branching logic for predictions
dttools/src/bucketing.h	Adds new bucketing mode enums and test helper function
dttools/src/category.h	Adds allocation mode enums for deterministic algorithms
dttools/src/category.c	Updates mode checking and allocation logic for new modes
dttools/src/bucketing_manager.c	Validates new bucketing modes
work_queue/src/work_queue.h	Adds WorkQueue allocation mode enums
work_queue/src/work_queue.c	Updates transaction logging for new modes
taskvine/src/manager/taskvine.h	Adds TaskVine allocation mode enums
taskvine/src/manager/vine_manager.c	Validates new allocation modes
taskvine/src/manager/vine_txn_log.c	Updates transaction logging for new modes
makeflow/src/dag_visitors.c	Adds JSON conversion for new allocation types
dttools/src/Makefile	Adds new source files to build system
dttools/test/TR_bucketing.sh	Adds test coverage for new algorithms
dttools/src/bucketing_base_test.c	Updates tests to cover new modes
dttools/src/bucketing_manager_test.c	Updates tests to cover new modes
work_queue/test/wq_alloc_test.py	Adds Python test coverage for new modes
taskvine/test/vine_allocations.py	Adds Python test coverage for new modes
work_queue/src/bindings/python3/ndcctools/work_queue_dask.py	Updates documentation with new modes
taskvine/src/bindings/python3/ndcctools/taskvine/dask_executor.py	Updates documentation with new modes
taskvine/src/bindings/python3/ndcctools/taskvine/compat/dask_executor.py	Updates documentation with new modes
taskvine/test/vine_common.sh	Fixes incorrect shebang (was /bin/hh, now /bin/sh)

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[btovar]

ready to merge?

[tphung3]

ready to merge?

The PR is not ready yet :) I'm ironing out some bugs.

[tphung3]

This PR is RTM.