Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions docs/quickcli.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,19 @@ dpgen2 submit input.json
where `input.json` is the input script. A guide of writing the script is found [here](inputscript).
When a workflow is submitted, a ID (WFID) of the workflow will be printed for later reference.

### Run the orchestrator locally and tasks on Slurm

DPGEN2 can run dflow locally while dispatching computational steps to a Slurm cluster over SSH. Start the local orchestrator with:

```bash
export DFLOW_DEBUG=1
dpgen2 submit examples/water/input_dpgen_slurm.json
```

For this arrangement, set `bohrium_config`, `dflow_config`, and `dflow_s3_config` to `null` or omit them. Configure a `dispatcher` executor under each computational `step_configs` entry instead. Important fields include the SSH `host`, `username`, `private_key_file`, and `remote_root`, plus `machine_dict.batch_type: Slurm` and the queue/resources in `resources_dict`.

The maintained `examples/water/input_dpgen_slurm.json` shows separate GPU training/exploration and CPU labeling resources. Commands, modules, container images, remote paths, and queue names remain site-specific. The submission host must be able to reach the Slurm login node and retain local dflow artifacts for the lifetime of the workflow.

## Check the convergence of a workflow
The convergence of stages of the workflow can be checked by the `status` command. It prints the indexes of the finished stages, iterations, and the accurate, candidate and failed ratio of explored configurations of each iteration.
```bash
Expand Down
10 changes: 8 additions & 2 deletions dpgen2/entrypoint/args.py
Original file line number Diff line number Diff line change
Expand Up @@ -646,8 +646,14 @@ def input_args():


def dflow_conf_args():
doc_dflow_config = "The configuration passed to dflow"
doc_dflow_s3_config = "The S3 configuration passed to dflow"
doc_dflow_config = (
"The configuration passed to dflow. It may be null when the workflow "
"orchestrator runs locally in debug mode."
)
doc_dflow_s3_config = (
"The S3 configuration passed to dflow. It may be null for local debug "
"mode when artifacts remain on the submission host."
)

return [
Argument(
Expand Down
7 changes: 1 addition & 6 deletions examples/water/input_dpgen_slurm.json
Original file line number Diff line number Diff line change
@@ -1,11 +1,6 @@
{
"name": "water-dpgen",
"bohrium_config": {
"username": "",
"password": "",
"project_id": 1,
"_comment": "all"
},
"bohrium_config": null,
"default_step_config": {
"template_config": {
"image": "",
Expand Down
1 change: 1 addition & 0 deletions tests/test_check_examples.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
p_examples / "chno" / "input.json",
p_examples / "water" / "input_dpgen_abacus.json",
p_examples / "water" / "input_dpgen_cp2k.json",
p_examples / "water" / "input_dpgen_slurm.json",
p_examples / "diffcsp" / "dpgen.json",
)

Expand Down