From 34d1b4573d15aa83122b6ca9be12d89308eb9ac5 Mon Sep 17 00:00:00 2001 From: MatMaster Date: Thu, 11 Jun 2026 14:22:58 +0800 Subject: [PATCH 01/13] feat(skills): add dpgen-run skill for concurrent learning workflow --- skills/dpgen-run/SKILL.md | 405 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 405 insertions(+) create mode 100644 skills/dpgen-run/SKILL.md diff --git a/skills/dpgen-run/SKILL.md b/skills/dpgen-run/SKILL.md new file mode 100644 index 000000000..b5daa9a12 --- /dev/null +++ b/skills/dpgen-run/SKILL.md @@ -0,0 +1,405 @@ +--- +name: dpgen-run +description: Prepare, explain, validate, and run DP-GEN concurrent learning workflows for training deep potential models via iterative exploration. Use when the user wants to generate or modify `param.json` and `machine.json` for `dpgen run`, configure training/exploration/labeling iterations, select descriptor types, set trust levels, define model_devi_jobs, or inspect run outputs. +compatibility: Requires a runnable environment with Python and an activated DP-GEN runtime where `dpgen` is available in PATH for the outer run command. Real execution also requires DeePMD-kit, LAMMPS (with DeePMD plugin for model_devi), and any backend-specific software required by the selected `fp_style`. For scheduler execution, each stage environment must be explicitly activated in `resources.source_list`. +license: LGPL-3.0-or-later +metadata: + author: MatMaster + version: 0.1.0 + repository: https://github.com/deepmodeling/dpgen +--- + +# DP-GEN Run (Concurrent Learning) + +Use this skill when the user wants to prepare, explain, validate, or execute the `dpgen run` concurrent learning workflow. + +This skill is for the main DP-GEN iterative loop: train an ensemble of deep potential models, explore configuration space via LAMMPS MD, select uncertain structures, label them with first-principles calculations, and feed new data back into training. + +## Core Rule (Critical) + +DP-GEN run always uses **two parameter classes** and therefore **two JSON files**: + +- **Workflow parameters** -> `param.json` +- **Execution / machine parameters** -> `machine.json` + +Run exactly: + +```bash +dpgen run param.json machine.json +``` + +Environment boundary rule: + +- Outer layer: run `dpgen run param.json machine.json` in an activated environment where `dpgen --version` works. +- Inner layer: for scheduler stages, explicitly activate runtime in `resources.source_list` on the server side. + +## Critical Pitfalls (Must Embed) + +These are verified failure modes discovered through testing. Treat as hard rules: + +1. **`se_atten_v2` spelling** — The descriptor type is `se_atten_v2` (double-t in "atten"). Writing `se_attn_v2` will silently fail or error. Always verify the exact string. + +2. **`remote_root` is mandatory** — dpdispatcher requires `remote_root` even for local Shell execution. Set it to a writable path like `/tmp/dpgen_train`. Omitting it causes runtime errors. + +3. **`batch_type: "Shell"` is case-sensitive** — Must be capitalized `"Shell"`, not `"shell"` or `"SHELL"`. Same for `"Slurm"`, `"PBS"`, etc. + +4. **CP2K `user_fp_params` KIND format** — When multiple element kinds are needed, use `"_": ["H", "O"]` with parallel arrays for `POTENTIAL` and `BASIS_SET`. This maps to repeated KIND sections. + +5. **`type_map.raw` ordering** — The `type_map.raw` in your `init_data_sys` directories must exactly match the `type_map` array in `param.json`. Mismatches cause silent data corruption. + +6. **DeePMD-kit 3.x format** — `default_training_param` uses nested structure: `model.descriptor`, `model.fitting_net`, `learning_rate`, `loss`, `training`. The `training` block includes `training_data.systems` (left empty — dpgen fills it). + +7. **`model_devi_engine` defaults to LAMMPS** — No need to set `model_devi_engine` explicitly when using LAMMPS. Only set it for alternative engines. + +8. **`fp_style: "cp2k"` is native** — dpgen v0.12+ supports CP2K natively via `user_fp_params` dict. No external plugins required. + +9. **`sys_configs` format** — List of lists: outer list = systems, inner list = file paths (POSCAR/xyz) for that system. Example: `[["path/to/POSCAR_1"], ["path/to/POSCAR_2"]]`. + +10. **`init_data_sys` format** — List of paths to directories in deepmd/npy format. Each directory must contain `type_map.raw`, `type.raw`, and `set.000/` with numpy arrays. + +## Agent responsibilities + +When using this skill, the agent should: + +1. confirm that the task is a `dpgen run` concurrent learning workflow +1. check whether existing configs or templates are already available +1. collect only the missing training, exploration, FP, and machine inputs +1. select appropriate descriptor type based on system complexity +1. generate or patch `param.json` +1. generate or patch `machine.json` +1. explain important parameters in plain language when asked +1. validate the workflow before execution +1. provide the exact command for running +1. after execution, summarize outputs and next inspection targets + +## Working policy + +### 1. Ask only for missing inputs + +Do not ask the user for everything if part of the configuration is already available. + +If the user already provides: + +- a partial `param.json` +- a partial `machine.json` +- a known training template +- a known cluster template + +then patch those files instead of rebuilding everything from scratch. + +### 2. Preserve the user's scientific choices + +Do not silently change: + +- descriptor family or type +- fitting net structure +- fp backend +- trust thresholds +- `type_map` ordering +- `model_devi_jobs` schedule +- ensemble or temperature settings + +If a value looks scientifically questionable, explain the concern instead of silently replacing it. + +### 3. Descriptor selection guidance + +dpgen (v0.13.x) only supports the **TensorFlow backend** of DeePMD-kit. It does NOT support PyTorch-only models (such as DPA-2, DPA-3). Do not recommend or generate configs for PyTorch-only descriptors. + +When the user has not specified a descriptor type, recommend based on: + +| Descriptor | Use case | Notes | +|---|---|---| +| `se_e2_a` | Simple systems, fast training, well-tested | Classic two-body embedding, requires explicit `sel` list | +| `se_atten` | Multi-component systems, moderate complexity | Attention-based, supports `sel: "auto"` | +| `se_atten_v2` | Modern default, best accuracy/cost balance | **Spell exactly `se_atten_v2`** (double-t). Supports `sel: "auto"`, `attn_layer: 0` | + +For new projects, prefer `se_atten_v2` unless the user has specific needs. If the user requires DPA-2/DPA-3 or PyTorch models, redirect them to **dpgen2** which supports the PyTorch backend. + +### 4. Keep local and scheduler execution explicit + +If the user wants local execution, produce local-friendly commands. + +If the user wants scheduler execution, produce scheduler-friendly commands and keep queue, partition, and resource requests explicit. + +Do not invent scheduler module names or executable paths. + +### 5. Do not invent environment activation commands + +If the user already has a working activation command such as: + +- `conda activate ...` +- `module load ...` +- `source ...` + +reuse it exactly. + +If execution is requested and the activation method is unknown, ask the user for the precise activation command. + +Do not guess conda environment names, module names, or site-specific paths. + +### 5.1 Outer launcher policy + +Use an activated DP-GEN environment and verify with: + +```bash +dpgen --version +``` + +Do not start run from a shell where `dpgen` is unavailable. + +### 5.2 Outer vs inner runtime boundaries (critical) + +Treat run execution as two separate environment layers: + +1. Outer layer: the shell that launches `dpgen run param.json machine.json` (must have `dpgen` in PATH) +1. Inner layer: stage tasks dispatched by DP-GEN (`train` / `model_devi` / `fp`) on server/runtime side + +Even if the outer layer is correct, inner stage tasks still need explicit runtime setup in `machine.json`. +Do not assume the outer shell environment will be inherited by dispatched stage jobs. +For scheduler-style execution, `resources.source_list` must explicitly activate the required runtime environment. + +### 6. Prefer reproducible output layout + +When generating a run workflow, keep files organized and predictable. + +Recommended structure: + +```text +project/ +├── param.json +├── machine.json +├── init_data/ +│ └── system_000/ +│ ├── type_map.raw +│ ├── type.raw +│ └── set.000/ +├── assets/ +│ └── structures/ +│ └── POSCAR_* +├── cp2k_basis_pp_file/ (if using CP2K) +│ ├── BASIS_MOLOPT +│ └── GTH_POTENTIALS +└── iter.*/ (created by dpgen) +``` + +## Minimum required inputs + +Collect the following information before generating files. + +### System information + +- `type_map` — ordered element symbols +- `mass_map` — atomic masses matching `type_map` order +- `init_data_prefix` — prefix for init_data_sys paths +- `init_data_sys` — list of paths to initial training data (deepmd/npy format) +- `sys_configs_prefix` — prefix for sys_configs paths +- `sys_configs` — list of lists of structure file paths + +### Training setup + +- `numb_models` — number of ensemble models (default: 4) +- `default_training_param` — full DeePMD training input: + - `model.descriptor` — descriptor type and settings + - `model.fitting_net` — fitting network settings + - `learning_rate` — learning rate schedule + - `loss` — loss function weights + - `training` — training steps and output settings + +### Exploration setup (model_devi) + +- `model_devi_dt` — MD timestep in ps (e.g. 0.0005 = 0.5 fs) +- `model_devi_skip` — number of initial frames to skip +- `model_devi_f_trust_lo` — lower force deviation threshold (eV/Å) +- `model_devi_f_trust_hi` — upper force deviation threshold (eV/Å) +- `model_devi_clean_traj` — whether to clean trajectory files after selection +- `model_devi_jobs` — list of exploration job specifications: + - `sys_idx` — which systems to explore + - `temps` — temperatures (K) + - `press` — pressures (bar) + - `trj_freq` — trajectory save frequency + - `nsteps` — number of MD steps + - `ensemble` — `"nvt"` or `"npt"` + +### FP setup + +- `fp_style` — backend: `"vasp"`, `"cp2k"`, `"abacus"`, `"gaussian"`, `"pwscf"`, or `"none"` +- `fp_task_max` — maximum number of FP tasks per iteration +- `fp_task_min` — minimum number to trigger FP +- Backend-specific settings: + - VASP: `fp_pp_path`, `fp_pp_files`, `fp_incar` or `fp_params` + - CP2K: `user_fp_params` (nested dict representing cp2k input) + - ABACUS: `user_fp_params`, `fp_pp_path`, `fp_pp_files`, `fp_orb_files` + - Gaussian: `fp_params` (keywords, nproc, multiplicity) + - PWSCF: `user_fp_params` + +### Execution setup + +For each stage `train`, `model_devi`, and `fp`, collect or preserve: + +- `command` +- `machine.batch_type` +- `machine.context_type` +- `machine.local_root` +- `machine.remote_root` **(mandatory — even for local Shell)** +- `resources.number_node` +- `resources.cpu_per_node` +- `resources.gpu_per_node` +- `resources.group_size` +- `resources.source_list` (required for scheduler jobs; use it to activate environment explicitly) +- any explicit queue / partition / custom scheduler flags if the user already uses them + +Choose a runtime profile first, then fill the matching template: + +- server-local Slurm: `assets/machine.template.server-local-slurm.json` +- pure local shell testing: `assets/machine.template.local-shell.json` + +## How to build `param.json` + +Construct `param.json` around these logical blocks: + +1. element and mass definitions (`type_map`, `mass_map`) +1. data source configuration (`init_data_prefix`, `init_data_sys`, `sys_configs_prefix`, `sys_configs`) +1. model ensemble count (`numb_models`) +1. default DeePMD training parameters (`default_training_param`) +1. exploration settings (`model_devi_dt`, `model_devi_skip`, `model_devi_f_trust_lo/hi`, `model_devi_clean_traj`) +1. exploration job schedule (`model_devi_jobs`) +1. FP backend settings (`fp_style`, `fp_task_max`, `fp_task_min`, backend-specific params) + +Key fields always required: + +- `type_map` +- `mass_map` +- `init_data_sys` +- `sys_configs` +- `numb_models` +- `default_training_param` +- `model_devi_dt` +- `model_devi_f_trust_lo` +- `model_devi_f_trust_hi` +- `model_devi_jobs` +- `fp_style` +- `fp_task_max` +- `fp_task_min` + +Trust level guidance: + +- `model_devi_f_trust_lo`: structures below this are "accurate" — not selected +- `model_devi_f_trust_hi`: structures above this are "failed" — discarded +- Between lo and hi: "candidate" — selected for FP labeling +- Typical starting values: lo=0.05–0.10, hi=0.15–0.30 (eV/Å) +- Tighter thresholds = more FP cost, better accuracy +- System-specific tuning is recommended after initial iterations + +Official reference examples: + +- `assets/param.example.water-cp2k.json` +- `assets/param.example.water-vasp.json` + +## How to build `machine.json` + +Construct `machine.json` with separate stage blocks for: + +- `train` +- `model_devi` +- `fp` + +For each stage, keep the following explicit: + +- `command` — the executable (`dp` for train, `lmp` for model_devi, FP command for fp) +- machine or context configuration +- resources +- queue or partition if needed +- cpu and gpu counts +- custom scheduler flags +- environment activation commands + +Do not merge all stages into one vague machine block. + +Stage-specific commands: + +- `train`: `"dp"` (DeePMD-kit training) +- `model_devi`: `"lmp"` (LAMMPS with DeePMD plugin) +- `fp`: backend-specific (`"vasp_std"`, `"cp2k.popt"`, `"abacus"`, `"pw.x"`, `"g16"`) + +## Validation before run + +Before execution, validate the workflow in this order: + +1. confirm outer-layer `dpgen` is available: + +```bash +dpgen --version +``` + +2. validate JSON syntax: + +```bash +python -m json.tool param.json +python -m json.tool machine.json +``` + +3. verify `init_data_sys` directories exist and contain proper deepmd/npy format: + +```bash +# Each directory must have type_map.raw, type.raw, set.000/ +ls init_data/*/type_map.raw +``` + +4. verify `type_map.raw` content matches `param.json` `type_map` ordering +5. verify `sys_configs` structure files exist +6. verify stage commands match the selected software stack +7. for CP2K: verify basis set and potential files are accessible +8. only then run: + +```bash +dpgen run param.json machine.json +``` + +## Output contract + +Always provide: + +1. final absolute paths to `param.json` and `machine.json` +1. the exact run command (`dpgen run param.json machine.json`) +1. a short pre-run checklist +1. any unresolved required fields +1. if execution was performed, the main output locations and next files to inspect + +## Guardrails + +- Never merge workflow and machine parameters into one file. +- Never run `dpgen run` before both JSON files are present. +- Never hardcode personal cluster, account, queue, or path settings as universal defaults. +- Never silently change the user's scientific choices. +- Keep `type_map` ordering consistent with `init_data_sys` type_map.raw files. +- If required inputs are missing, stop and ask instead of guessing. +- Always spell `se_atten_v2` correctly (not `se_attn_v2`). +- Always include `remote_root` in machine config — it is mandatory even for local execution. +- Always capitalize `batch_type` values: `"Shell"`, `"Slurm"`, `"PBS"`. +- For CP2K KIND sections with multiple elements, use `"_": ["elem1", "elem2"]` array format. +- Do not assume outer-shell activation is inherited by stage jobs; for scheduler execution, require explicit `source_list` per stage. +- If the user already has working templates, patch them rather than overwriting them blindly. +- Do not set `model_devi_engine` unless using a non-LAMMPS engine (it defaults to LAMMPS). +- Leave `training.training_data.systems` as an empty list `[]` in `default_training_param` — dpgen fills this automatically. + +## References and bundled files + +Use these bundled files: + +- `assets/param.template.json` +- `assets/param.example.water-cp2k.json` +- `assets/param.example.water-vasp.json` +- `assets/machine.template.json` +- `assets/machine.template.server-local-slurm.json` +- `assets/machine.template.local-shell.json` +- `references/param-fields.md` +- `references/machine-fields.md` +- `references/workflow-notes.md` +- `references/descriptor-types.md` + +External references: + +- DP-GEN run overview: https://docs.deepmodeling.com/projects/dpgen/en/latest/run/index.html +- run parameter definitions: https://docs.deepmodeling.com/projects/dpgen/en/latest/run/param.html +- run machine definitions: https://docs.deepmodeling.com/projects/dpgen/en/latest/run/mdata.html +- DeePMD-kit model documentation: https://docs.deepmodeling.com/projects/deepmd/en/latest/model/index.html From 841acdaf8744faa96ef7d28c99deb9455a7ecc77 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Thu, 11 Jun 2026 06:51:07 +0000 Subject: [PATCH 02/13] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- skills/dpgen-run/SKILL.md | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/skills/dpgen-run/SKILL.md b/skills/dpgen-run/SKILL.md index b5daa9a12..ba4a66a17 100644 --- a/skills/dpgen-run/SKILL.md +++ b/skills/dpgen-run/SKILL.md @@ -39,23 +39,23 @@ These are verified failure modes discovered through testing. Treat as hard rules 1. **`se_atten_v2` spelling** — The descriptor type is `se_atten_v2` (double-t in "atten"). Writing `se_attn_v2` will silently fail or error. Always verify the exact string. -2. **`remote_root` is mandatory** — dpdispatcher requires `remote_root` even for local Shell execution. Set it to a writable path like `/tmp/dpgen_train`. Omitting it causes runtime errors. +1. **`remote_root` is mandatory** — dpdispatcher requires `remote_root` even for local Shell execution. Set it to a writable path like `/tmp/dpgen_train`. Omitting it causes runtime errors. -3. **`batch_type: "Shell"` is case-sensitive** — Must be capitalized `"Shell"`, not `"shell"` or `"SHELL"`. Same for `"Slurm"`, `"PBS"`, etc. +1. **`batch_type: "Shell"` is case-sensitive** — Must be capitalized `"Shell"`, not `"shell"` or `"SHELL"`. Same for `"Slurm"`, `"PBS"`, etc. -4. **CP2K `user_fp_params` KIND format** — When multiple element kinds are needed, use `"_": ["H", "O"]` with parallel arrays for `POTENTIAL` and `BASIS_SET`. This maps to repeated KIND sections. +1. **CP2K `user_fp_params` KIND format** — When multiple element kinds are needed, use `"_": ["H", "O"]` with parallel arrays for `POTENTIAL` and `BASIS_SET`. This maps to repeated KIND sections. -5. **`type_map.raw` ordering** — The `type_map.raw` in your `init_data_sys` directories must exactly match the `type_map` array in `param.json`. Mismatches cause silent data corruption. +1. **`type_map.raw` ordering** — The `type_map.raw` in your `init_data_sys` directories must exactly match the `type_map` array in `param.json`. Mismatches cause silent data corruption. -6. **DeePMD-kit 3.x format** — `default_training_param` uses nested structure: `model.descriptor`, `model.fitting_net`, `learning_rate`, `loss`, `training`. The `training` block includes `training_data.systems` (left empty — dpgen fills it). +1. **DeePMD-kit 3.x format** — `default_training_param` uses nested structure: `model.descriptor`, `model.fitting_net`, `learning_rate`, `loss`, `training`. The `training` block includes `training_data.systems` (left empty — dpgen fills it). -7. **`model_devi_engine` defaults to LAMMPS** — No need to set `model_devi_engine` explicitly when using LAMMPS. Only set it for alternative engines. +1. **`model_devi_engine` defaults to LAMMPS** — No need to set `model_devi_engine` explicitly when using LAMMPS. Only set it for alternative engines. -8. **`fp_style: "cp2k"` is native** — dpgen v0.12+ supports CP2K natively via `user_fp_params` dict. No external plugins required. +1. **`fp_style: "cp2k"` is native** — dpgen v0.12+ supports CP2K natively via `user_fp_params` dict. No external plugins required. -9. **`sys_configs` format** — List of lists: outer list = systems, inner list = file paths (POSCAR/xyz) for that system. Example: `[["path/to/POSCAR_1"], ["path/to/POSCAR_2"]]`. +1. **`sys_configs` format** — List of lists: outer list = systems, inner list = file paths (POSCAR/xyz) for that system. Example: `[["path/to/POSCAR_1"], ["path/to/POSCAR_2"]]`. -10. **`init_data_sys` format** — List of paths to directories in deepmd/npy format. Each directory must contain `type_map.raw`, `type.raw`, and `set.000/` with numpy arrays. +1. **`init_data_sys` format** — List of paths to directories in deepmd/npy format. Each directory must contain `type_map.raw`, `type.raw`, and `set.000/` with numpy arrays. ## Agent responsibilities @@ -107,11 +107,11 @@ dpgen (v0.13.x) only supports the **TensorFlow backend** of DeePMD-kit. It does When the user has not specified a descriptor type, recommend based on: -| Descriptor | Use case | Notes | -|---|---|---| -| `se_e2_a` | Simple systems, fast training, well-tested | Classic two-body embedding, requires explicit `sel` list | -| `se_atten` | Multi-component systems, moderate complexity | Attention-based, supports `sel: "auto"` | -| `se_atten_v2` | Modern default, best accuracy/cost balance | **Spell exactly `se_atten_v2`** (double-t). Supports `sel: "auto"`, `attn_layer: 0` | +| Descriptor | Use case | Notes | +| ------------- | -------------------------------------------- | ----------------------------------------------------------------------------------- | +| `se_e2_a` | Simple systems, fast training, well-tested | Classic two-body embedding, requires explicit `sel` list | +| `se_atten` | Multi-component systems, moderate complexity | Attention-based, supports `sel: "auto"` | +| `se_atten_v2` | Modern default, best accuracy/cost balance | **Spell exactly `se_atten_v2`** (double-t). Supports `sel: "auto"`, `attn_layer: 0` | For new projects, prefer `se_atten_v2` unless the user has specific needs. If the user requires DPA-2/DPA-3 or PyTorch models, redirect them to **dpgen2** which supports the PyTorch backend. @@ -346,10 +346,10 @@ ls init_data/*/type_map.raw ``` 4. verify `type_map.raw` content matches `param.json` `type_map` ordering -5. verify `sys_configs` structure files exist -6. verify stage commands match the selected software stack -7. for CP2K: verify basis set and potential files are accessible -8. only then run: +1. verify `sys_configs` structure files exist +1. verify stage commands match the selected software stack +1. for CP2K: verify basis set and potential files are accessible +1. only then run: ```bash dpgen run param.json machine.json From cd01b79ee05f015f6479a6226aded8aafc35206a Mon Sep 17 00:00:00 2001 From: SchrodingersCattt Date: Sun, 14 Jun 2026 07:04:12 +0000 Subject: [PATCH 03/13] docs: address dpgen-run skill review --- skills/dpgen-run/SKILL.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/skills/dpgen-run/SKILL.md b/skills/dpgen-run/SKILL.md index ba4a66a17..a964424c5 100644 --- a/skills/dpgen-run/SKILL.md +++ b/skills/dpgen-run/SKILL.md @@ -103,7 +103,7 @@ If a value looks scientifically questionable, explain the concern instead of sil ### 3. Descriptor selection guidance -dpgen (v0.13.x) only supports the **TensorFlow backend** of DeePMD-kit. It does NOT support PyTorch-only models (such as DPA-2, DPA-3). Do not recommend or generate configs for PyTorch-only descriptors. +dpgen includes `train_backend` options for both `"tensorflow"` and `"pytorch"`, so backend support may include PyTorch depending on the installed DeePMD-kit stack. If backend or descriptor compatibility is unclear, verify it against the repo's supported `train_backend` values (`"tensorflow"`, `"pytorch"`) and the user's installed software before generating configs. When the user has not specified a descriptor type, recommend based on: @@ -113,7 +113,7 @@ When the user has not specified a descriptor type, recommend based on: | `se_atten` | Multi-component systems, moderate complexity | Attention-based, supports `sel: "auto"` | | `se_atten_v2` | Modern default, best accuracy/cost balance | **Spell exactly `se_atten_v2`** (double-t). Supports `sel: "auto"`, `attn_layer: 0` | -For new projects, prefer `se_atten_v2` unless the user has specific needs. If the user requires DPA-2/DPA-3 or PyTorch models, redirect them to **dpgen2** which supports the PyTorch backend. +For new projects, prefer `se_atten_v2` unless the user has specific needs. If the user requires PyTorch-only models or DPA-2/DPA-3, prefer **dpgen2** unless the current dpgen and DeePMD-kit installation are verified to support the requested model. ### 4. Keep local and scheduler execution explicit @@ -346,10 +346,10 @@ ls init_data/*/type_map.raw ``` 4. verify `type_map.raw` content matches `param.json` `type_map` ordering -1. verify `sys_configs` structure files exist -1. verify stage commands match the selected software stack -1. for CP2K: verify basis set and potential files are accessible -1. only then run: +5. verify `sys_configs` structure files exist +6. verify stage commands match the selected software stack +7. for CP2K: verify basis set and potential files are accessible +8. only then run: ```bash dpgen run param.json machine.json @@ -380,7 +380,7 @@ Always provide: - Do not assume outer-shell activation is inherited by stage jobs; for scheduler execution, require explicit `source_list` per stage. - If the user already has working templates, patch them rather than overwriting them blindly. - Do not set `model_devi_engine` unless using a non-LAMMPS engine (it defaults to LAMMPS). -- Leave `training.training_data.systems` as an empty list `[]` in `default_training_param` — dpgen fills this automatically. +- In `default_training_param`, leave `training.training_data.systems` unset or empty — `dpgen/generator/run.py` fills it from `init_data_sys` automatically. ## References and bundled files From 39adc8fc5a980dd9e8b7424fff14bc0cd8a684e1 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Sun, 14 Jun 2026 07:05:02 +0000 Subject: [PATCH 04/13] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- skills/dpgen-run/SKILL.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/skills/dpgen-run/SKILL.md b/skills/dpgen-run/SKILL.md index a964424c5..2aa70ee81 100644 --- a/skills/dpgen-run/SKILL.md +++ b/skills/dpgen-run/SKILL.md @@ -346,10 +346,10 @@ ls init_data/*/type_map.raw ``` 4. verify `type_map.raw` content matches `param.json` `type_map` ordering -5. verify `sys_configs` structure files exist -6. verify stage commands match the selected software stack -7. for CP2K: verify basis set and potential files are accessible -8. only then run: +1. verify `sys_configs` structure files exist +1. verify stage commands match the selected software stack +1. for CP2K: verify basis set and potential files are accessible +1. only then run: ```bash dpgen run param.json machine.json From 8397b836323d941b91702f4f6aa40f75df12a4ce Mon Sep 17 00:00:00 2001 From: SchrodingersCattt Date: Sun, 14 Jun 2026 07:09:10 +0000 Subject: [PATCH 05/13] docs: clarify dpgen backend compatibility --- skills/dpgen-run/SKILL.md | 48 +++++++++++++++++++-------------------- 1 file changed, 24 insertions(+), 24 deletions(-) diff --git a/skills/dpgen-run/SKILL.md b/skills/dpgen-run/SKILL.md index 2aa70ee81..779b56515 100644 --- a/skills/dpgen-run/SKILL.md +++ b/skills/dpgen-run/SKILL.md @@ -47,7 +47,7 @@ These are verified failure modes discovered through testing. Treat as hard rules 1. **`type_map.raw` ordering** — The `type_map.raw` in your `init_data_sys` directories must exactly match the `type_map` array in `param.json`. Mismatches cause silent data corruption. -1. **DeePMD-kit 3.x format** — `default_training_param` uses nested structure: `model.descriptor`, `model.fitting_net`, `learning_rate`, `loss`, `training`. The `training` block includes `training_data.systems` (left empty — dpgen fills it). +1. **DeePMD-kit input format** — `default_training_param` uses nested structure: `model.descriptor`, `model.fitting_net`, `learning_rate`, `loss`, `training`. The `training` block may leave `training_data.systems` unset or empty — dpgen fills it. Check the configured `deepmd_version` and installed DeePMD-kit before using 3.x-specific features. 1. **`model_devi_engine` defaults to LAMMPS** — No need to set `model_devi_engine` explicitly when using LAMMPS. Only set it for alternative engines. @@ -103,7 +103,7 @@ If a value looks scientifically questionable, explain the concern instead of sil ### 3. Descriptor selection guidance -dpgen includes `train_backend` options for both `"tensorflow"` and `"pytorch"`, so backend support may include PyTorch depending on the installed DeePMD-kit stack. If backend or descriptor compatibility is unclear, verify it against the repo's supported `train_backend` values (`"tensorflow"`, `"pytorch"`) and the user's installed software before generating configs. +dpgen exposes `train_backend` options for `"tensorflow"` and `"pytorch"`, but that option alone does not guarantee every PyTorch descriptor or every DeePMD-kit 3.x feature works in the user's installed stack. If backend or descriptor compatibility is unclear, verify it against the repo's `train_backend` options, the configured `deepmd_version`, and the installed DeePMD-kit version before generating configs. When the user has not specified a descriptor type, recommend based on: @@ -325,35 +325,35 @@ Stage-specific commands: Before execution, validate the workflow in this order: -1. confirm outer-layer `dpgen` is available: +- Step 1: confirm outer-layer `dpgen` is available: -```bash -dpgen --version -``` + ```bash + dpgen --version + ``` -2. validate JSON syntax: +- Step 2: validate JSON syntax: -```bash -python -m json.tool param.json -python -m json.tool machine.json -``` + ```bash + python -m json.tool param.json + python -m json.tool machine.json + ``` -3. verify `init_data_sys` directories exist and contain proper deepmd/npy format: +- Step 3: verify `init_data_sys` directories exist and contain proper deepmd/npy format: -```bash -# Each directory must have type_map.raw, type.raw, set.000/ -ls init_data/*/type_map.raw -``` + ```bash + # Each directory must have type_map.raw, type.raw, set.000/ + ls init_data/*/type_map.raw + ``` -4. verify `type_map.raw` content matches `param.json` `type_map` ordering -1. verify `sys_configs` structure files exist -1. verify stage commands match the selected software stack -1. for CP2K: verify basis set and potential files are accessible -1. only then run: +- Step 4: verify `type_map.raw` content matches `param.json` `type_map` ordering. +- Step 5: verify `sys_configs` structure files exist. +- Step 6: verify stage commands match the selected software stack. +- Step 7: for CP2K, verify basis set and potential files are accessible. +- Step 8: only then run: -```bash -dpgen run param.json machine.json -``` + ```bash + dpgen run param.json machine.json + ``` ## Output contract From f1ee9580323d186470ec8f720274d8bba78b0a2d Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Sun, 14 Jun 2026 07:11:55 +0000 Subject: [PATCH 06/13] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- skills/dpgen-run/SKILL.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/skills/dpgen-run/SKILL.md b/skills/dpgen-run/SKILL.md index 779b56515..6089f68d3 100644 --- a/skills/dpgen-run/SKILL.md +++ b/skills/dpgen-run/SKILL.md @@ -346,9 +346,13 @@ Before execution, validate the workflow in this order: ``` - Step 4: verify `type_map.raw` content matches `param.json` `type_map` ordering. + - Step 5: verify `sys_configs` structure files exist. + - Step 6: verify stage commands match the selected software stack. + - Step 7: for CP2K, verify basis set and potential files are accessible. + - Step 8: only then run: ```bash From ae2c3457b458a81534c9da4d419760a8c1776b19 Mon Sep 17 00:00:00 2001 From: SchrodingersCattt Date: Wed, 12 Aug 2026 04:43:56 +0000 Subject: [PATCH 07/13] docs(skills): complete dpgen-run guardrails --- skills/dpgen-run/SKILL.md | 56 ++++++++++++++++++++++------------- tests/test_dpgen_run_skill.py | 29 ++++++++++++++++++ 2 files changed, 64 insertions(+), 21 deletions(-) create mode 100644 tests/test_dpgen_run_skill.py diff --git a/skills/dpgen-run/SKILL.md b/skills/dpgen-run/SKILL.md index 6089f68d3..41759fd60 100644 --- a/skills/dpgen-run/SKILL.md +++ b/skills/dpgen-run/SKILL.md @@ -39,9 +39,14 @@ These are verified failure modes discovered through testing. Treat as hard rules 1. **`se_atten_v2` spelling** — The descriptor type is `se_atten_v2` (double-t in "atten"). Writing `se_attn_v2` will silently fail or error. Always verify the exact string. -1. **`remote_root` is mandatory** — dpdispatcher requires `remote_root` even for local Shell execution. Set it to a writable path like `/tmp/dpgen_train`. Omitting it causes runtime errors. +1. **`remote_root` is context-dependent** — require it for remote or scheduler + contexts whose installed dpdispatcher schema needs a remote working root. + Lazy/local contexts can omit it. Preserve an existing working omission and + validate against the installed dpdispatcher version. -1. **`batch_type: "Shell"` is case-sensitive** — Must be capitalized `"Shell"`, not `"shell"` or `"SHELL"`. Same for `"Slurm"`, `"PBS"`, etc. +1. **`batch_type` must be registered** — canonical names and lowercase aliases + can both be valid. Preserve an existing working value and validate new values + against the installed dpdispatcher version. 1. **CP2K `user_fp_params` KIND format** — When multiple element kinds are needed, use `"_": ["H", "O"]` with parallel arrays for `POTENTIAL` and `BASIS_SET`. This maps to repeated KIND sections. @@ -74,6 +79,13 @@ When using this skill, the agent should: ## Working policy +### 0. Treat execution as separately authorized + +Prepare, explain, and validate by default. Execute `dpgen run` only when the +user has explicitly requested execution and, after validation, confirmed the +exact command. Supplying configuration files, passing validation, or showing the +command does not by itself authorize an HPC or first-principles workload. + ### 1. Ask only for missing inputs Do not ask the user for everything if part of the configuration is already available. @@ -240,7 +252,8 @@ For each stage `train`, `model_devi`, and `fp`, collect or preserve: - `machine.batch_type` - `machine.context_type` - `machine.local_root` -- `machine.remote_root` **(mandatory — even for local Shell)** +- `machine.remote_root` when required by the selected context; preserve a + working omission for lazy/local execution - `resources.number_node` - `resources.cpu_per_node` - `resources.gpu_per_node` @@ -250,8 +263,10 @@ For each stage `train`, `model_devi`, and `fp`, collect or preserve: Choose a runtime profile first, then fill the matching template: -- server-local Slurm: `assets/machine.template.server-local-slurm.json` -- pure local shell testing: `assets/machine.template.local-shell.json` +- server-local scheduler: + [existing scheduler example](../../examples/machine/DeePMD-kit-1.x/machine-lsf-slurm-cp2k.json) +- pure local shell testing: + [existing local example](../../examples/run/ch4/machine.json) ## How to build `param.json` @@ -292,8 +307,10 @@ Trust level guidance: Official reference examples: -- `assets/param.example.water-cp2k.json` -- `assets/param.example.water-vasp.json` +- CP2K: + [methane example](../../examples/run/dp2.x-lammps-cp2k/param_CH4_deepmd-kit-2.0.1.json) +- VASP: + [CH4 example](../../examples/run/dp2.x-lammps-vasp/CH4/param_CH4_deepmd-kit-2.x.json) ## How to build `machine.json` @@ -378,28 +395,25 @@ Always provide: - Keep `type_map` ordering consistent with `init_data_sys` type_map.raw files. - If required inputs are missing, stop and ask instead of guessing. - Always spell `se_atten_v2` correctly (not `se_attn_v2`). -- Always include `remote_root` in machine config — it is mandatory even for local execution. -- Always capitalize `batch_type` values: `"Shell"`, `"Slurm"`, `"PBS"`. +- Include `remote_root` when the selected context requires it; do not add it + solely to rewrite a working lazy/local configuration. +- Use a `batch_type` accepted by the installed dpdispatcher version and + preserve an existing working canonical name or alias. - For CP2K KIND sections with multiple elements, use `"_": ["elem1", "elem2"]` array format. - Do not assume outer-shell activation is inherited by stage jobs; for scheduler execution, require explicit `source_list` per stage. - If the user already has working templates, patch them rather than overwriting them blindly. - Do not set `model_devi_engine` unless using a non-LAMMPS engine (it defaults to LAMMPS). - In `default_training_param`, leave `training.training_data.systems` unset or empty — `dpgen/generator/run.py` fills it from `init_data_sys` automatically. -## References and bundled files +## Repository examples and references -Use these bundled files: +Use these checked-in repository examples as starting points: -- `assets/param.template.json` -- `assets/param.example.water-cp2k.json` -- `assets/param.example.water-vasp.json` -- `assets/machine.template.json` -- `assets/machine.template.server-local-slurm.json` -- `assets/machine.template.local-shell.json` -- `references/param-fields.md` -- `references/machine-fields.md` -- `references/workflow-notes.md` -- `references/descriptor-types.md` +- [local CH4 parameter example](../../examples/run/ch4/param.json) +- [local CH4 machine example](../../examples/run/ch4/machine.json) +- [CP2K parameter example](../../examples/run/dp2.x-lammps-cp2k/param_CH4_deepmd-kit-2.0.1.json) +- [VASP parameter example](../../examples/run/dp2.x-lammps-vasp/CH4/param_CH4_deepmd-kit-2.x.json) +- [scheduler machine example](../../examples/machine/DeePMD-kit-1.x/machine-lsf-slurm-cp2k.json) External references: diff --git a/tests/test_dpgen_run_skill.py b/tests/test_dpgen_run_skill.py new file mode 100644 index 000000000..cc783c953 --- /dev/null +++ b/tests/test_dpgen_run_skill.py @@ -0,0 +1,29 @@ +import re +import unittest +from pathlib import Path + + +class TestDPGenRunSkill(unittest.TestCase): + def test_all_repository_relative_links_exist(self): + repository_root = Path(__file__).resolve().parents[1] + skill_path = repository_root / "skills" / "dpgen-run" / "SKILL.md" + link_pattern = re.compile(r"\[[^]]+\]\(([^)]+)\)") + + missing = [] + for target in link_pattern.findall(skill_path.read_text()): + if "://" in target or target.startswith("#"): + continue + target_path = target.split("#", 1)[0] + if not (skill_path.parent / target_path).resolve().exists(): + missing.append(target) + + self.assertEqual( + missing, + [], + "Missing repository-relative links in dpgen-run skill: " + + ", ".join(missing), + ) + + +if __name__ == "__main__": + unittest.main() From 730249a672e2ad89350bca94b1b0fcbad58eb039 Mon Sep 17 00:00:00 2001 From: SchrodingersCattt Date: Wed, 12 Aug 2026 05:24:40 +0000 Subject: [PATCH 08/13] docs(skills): validate current machine examples --- skills/dpgen-run/SKILL.md | 4 ++-- tests/test_dpgen_run_skill.py | 38 +++++++++++++++++++++++++++++++++-- 2 files changed, 38 insertions(+), 4 deletions(-) diff --git a/skills/dpgen-run/SKILL.md b/skills/dpgen-run/SKILL.md index 41759fd60..3375aa5c3 100644 --- a/skills/dpgen-run/SKILL.md +++ b/skills/dpgen-run/SKILL.md @@ -266,7 +266,7 @@ Choose a runtime profile first, then fill the matching template: - server-local scheduler: [existing scheduler example](../../examples/machine/DeePMD-kit-1.x/machine-lsf-slurm-cp2k.json) - pure local shell testing: - [existing local example](../../examples/run/ch4/machine.json) + [existing local example](../../examples/machine/DeePMD-kit-1.x/machine-local.json) ## How to build `param.json` @@ -410,7 +410,7 @@ Always provide: Use these checked-in repository examples as starting points: - [local CH4 parameter example](../../examples/run/ch4/param.json) -- [local CH4 machine example](../../examples/run/ch4/machine.json) +- [local machine example](../../examples/machine/DeePMD-kit-1.x/machine-local.json) - [CP2K parameter example](../../examples/run/dp2.x-lammps-cp2k/param_CH4_deepmd-kit-2.0.1.json) - [VASP parameter example](../../examples/run/dp2.x-lammps-vasp/CH4/param_CH4_deepmd-kit-2.x.json) - [scheduler machine example](../../examples/machine/DeePMD-kit-1.x/machine-lsf-slurm-cp2k.json) diff --git a/tests/test_dpgen_run_skill.py b/tests/test_dpgen_run_skill.py index cc783c953..31fdaf6db 100644 --- a/tests/test_dpgen_run_skill.py +++ b/tests/test_dpgen_run_skill.py @@ -1,7 +1,10 @@ +import json import re import unittest from pathlib import Path +from dpgen.remote.decide_machine import convert_mdata + class TestDPGenRunSkill(unittest.TestCase): def test_all_repository_relative_links_exist(self): @@ -13,9 +16,14 @@ def test_all_repository_relative_links_exist(self): for target in link_pattern.findall(skill_path.read_text()): if "://" in target or target.startswith("#"): continue - target_path = target.split("#", 1)[0] - if not (skill_path.parent / target_path).resolve().exists(): + resolved_target = (skill_path.parent / target.split("#", 1)[0]).resolve() + try: + resolved_target.relative_to(repository_root) + except ValueError: missing.append(target) + else: + if not resolved_target.exists(): + missing.append(target) self.assertEqual( missing, @@ -24,6 +32,32 @@ def test_all_repository_relative_links_exist(self): + ", ".join(missing), ) + def test_linked_machine_examples_use_current_schema(self): + repository_root = Path(__file__).resolve().parents[1] + examples = [ + repository_root + / "examples" + / "machine" + / "DeePMD-kit-1.x" + / "machine-local.json", + repository_root + / "examples" + / "machine" + / "DeePMD-kit-1.x" + / "machine-lsf-slurm-cp2k.json", + ] + + for example in examples: + with self.subTest(example=example): + machine_data = json.loads(example.read_text()) + for stage in ("train", "model_devi", "fp"): + self.assertIn(stage, machine_data) + converted = convert_mdata(machine_data) + for stage in ("train", "model_devi", "fp"): + self.assertIn(f"{stage}_machine", converted) + self.assertIn(f"{stage}_resources", converted) + self.assertIn(f"{stage}_command", converted) + if __name__ == "__main__": unittest.main() From 91ecc4275bd8c3a9f98ca6d714ea38dc24316c76 Mon Sep 17 00:00:00 2001 From: SchrodingersCattt Date: Wed, 12 Aug 2026 05:55:35 +0000 Subject: [PATCH 09/13] test: validate documented machine examples --- tests/test_dpgen_run_skill.py | 72 ++++++++++++++++++++++++++++------- 1 file changed, 58 insertions(+), 14 deletions(-) diff --git a/tests/test_dpgen_run_skill.py b/tests/test_dpgen_run_skill.py index 31fdaf6db..c5f9abbef 100644 --- a/tests/test_dpgen_run_skill.py +++ b/tests/test_dpgen_run_skill.py @@ -5,15 +5,16 @@ from dpgen.remote.decide_machine import convert_mdata +LINK_PATTERN = re.compile(r"\[[^]]+\]\(([^)]+)\)") + class TestDPGenRunSkill(unittest.TestCase): def test_all_repository_relative_links_exist(self): repository_root = Path(__file__).resolve().parents[1] skill_path = repository_root / "skills" / "dpgen-run" / "SKILL.md" - link_pattern = re.compile(r"\[[^]]+\]\(([^)]+)\)") missing = [] - for target in link_pattern.findall(skill_path.read_text()): + for target in LINK_PATTERN.findall(skill_path.read_text()): if "://" in target or target.startswith("#"): continue resolved_target = (skill_path.parent / target.split("#", 1)[0]).resolve() @@ -34,29 +35,72 @@ def test_all_repository_relative_links_exist(self): def test_linked_machine_examples_use_current_schema(self): repository_root = Path(__file__).resolve().parents[1] - examples = [ - repository_root - / "examples" - / "machine" - / "DeePMD-kit-1.x" - / "machine-local.json", - repository_root - / "examples" - / "machine" - / "DeePMD-kit-1.x" - / "machine-lsf-slurm-cp2k.json", - ] + skill_path = repository_root / "skills" / "dpgen-run" / "SKILL.md" + machine_examples_root = repository_root / "examples" / "machine" + examples = set() + for target in LINK_PATTERN.findall(skill_path.read_text()): + target_path = target.split("#", 1)[0] + if not target_path.endswith(".json"): + continue + resolved_target = (skill_path.parent / target_path).resolve() + try: + resolved_target.relative_to(machine_examples_root) + except ValueError: + continue + examples.add(resolved_target) + self.assertTrue(examples, "No examples/machine/*.json links found") for example in examples: with self.subTest(example=example): machine_data = json.loads(example.read_text()) + stage_values = {} for stage in ("train", "model_devi", "fp"): self.assertIn(stage, machine_data) + stage_data = machine_data[stage] + if isinstance(stage_data, list): + self.assertTrue(stage_data) + stage_data = stage_data[0] + self.assertIsInstance(stage_data, dict) + stage_values[stage] = stage_data + + machine = stage_data.get("machine") + resources = stage_data.get("resources") + command = stage_data.get("command") + self.assertIsInstance(machine, dict) + self.assertTrue(machine) + self.assertIsInstance(resources, dict) + self.assertTrue(resources) + self.assertIsInstance(command, str) + self.assertTrue(command.strip()) + for field in ("batch_type", "context_type", "local_root"): + self.assertIn(field, machine) + self.assertTrue(machine[field]) + for field in ( + "number_node", + "cpu_per_node", + "gpu_per_node", + "group_size", + ): + self.assertIn(field, resources) converted = convert_mdata(machine_data) for stage in ("train", "model_devi", "fp"): self.assertIn(f"{stage}_machine", converted) self.assertIn(f"{stage}_resources", converted) self.assertIn(f"{stage}_command", converted) + converted_machine = converted[f"{stage}_machine"] + converted_resources = converted[f"{stage}_resources"] + converted_command = converted[f"{stage}_command"] + self.assertIsInstance(converted_machine, dict) + self.assertTrue(converted_machine) + self.assertIsInstance(converted_resources, dict) + self.assertTrue(converted_resources) + self.assertIsInstance(converted_command, str) + self.assertTrue(converted_command.strip()) + self.assertEqual(converted_machine, stage_values[stage]["machine"]) + self.assertEqual( + converted_resources, stage_values[stage]["resources"] + ) + self.assertEqual(converted_command, stage_values[stage]["command"]) if __name__ == "__main__": From 90e1be0804a0577916b90ec6d54a05b8f838d57f Mon Sep 17 00:00:00 2001 From: njzjz-bot Date: Sun, 30 Aug 2026 00:21:51 +0800 Subject: [PATCH 10/13] docs: correct DP-GEN skill validation Coding-Agent: Codex Codex-Version: codex-cli 0.151.0 Model: gpt-5.6-sol Reasoning-Effort: xhigh --- skills/dpgen-run/SKILL.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/skills/dpgen-run/SKILL.md b/skills/dpgen-run/SKILL.md index 3375aa5c3..f232a6e35 100644 --- a/skills/dpgen-run/SKILL.md +++ b/skills/dpgen-run/SKILL.md @@ -30,7 +30,7 @@ dpgen run param.json machine.json Environment boundary rule: -- Outer layer: run `dpgen run param.json machine.json` in an activated environment where `dpgen --version` works. +- Outer layer: run `dpgen run param.json machine.json` in an activated environment where `dpgen -h` works. - Inner layer: for scheduler stages, explicitly activate runtime in `resources.source_list` on the server side. ## Critical Pitfalls (Must Embed) @@ -154,7 +154,7 @@ Do not guess conda environment names, module names, or site-specific paths. Use an activated DP-GEN environment and verify with: ```bash -dpgen --version +dpgen -h ``` Do not start run from a shell where `dpgen` is unavailable. @@ -238,7 +238,7 @@ Collect the following information before generating files. - `fp_task_max` — maximum number of FP tasks per iteration - `fp_task_min` — minimum number to trigger FP - Backend-specific settings: - - VASP: `fp_pp_path`, `fp_pp_files`, `fp_incar` or `fp_params` + - VASP: `fp_pp_path`, `fp_pp_files`, and `fp_incar` - CP2K: `user_fp_params` (nested dict representing cp2k input) - ABACUS: `user_fp_params`, `fp_pp_path`, `fp_pp_files`, `fp_orb_files` - Gaussian: `fp_params` (keywords, nproc, multiplicity) @@ -345,7 +345,7 @@ Before execution, validate the workflow in this order: - Step 1: confirm outer-layer `dpgen` is available: ```bash - dpgen --version + dpgen -h ``` - Step 2: validate JSON syntax: From 5ae7c0e2696525b069065b7510b6316d8e0b42fe Mon Sep 17 00:00:00 2001 From: SchrodingersCattt Date: Sun, 30 Aug 2026 01:36:14 +0000 Subject: [PATCH 11/13] fix(skills): validate dpgen-run examples --- skills/dpgen-run/SKILL.md | 33 +++++++++++++++++++-------------- tests/test_dpgen_run_skill.py | 30 ++++++++++++++++++++++++++++++ 2 files changed, 49 insertions(+), 14 deletions(-) diff --git a/skills/dpgen-run/SKILL.md b/skills/dpgen-run/SKILL.md index f232a6e35..d30cec8e6 100644 --- a/skills/dpgen-run/SKILL.md +++ b/skills/dpgen-run/SKILL.md @@ -1,7 +1,6 @@ --- name: dpgen-run description: Prepare, explain, validate, and run DP-GEN concurrent learning workflows for training deep potential models via iterative exploration. Use when the user wants to generate or modify `param.json` and `machine.json` for `dpgen run`, configure training/exploration/labeling iterations, select descriptor types, set trust levels, define model_devi_jobs, or inspect run outputs. -compatibility: Requires a runnable environment with Python and an activated DP-GEN runtime where `dpgen` is available in PATH for the outer run command. Real execution also requires DeePMD-kit, LAMMPS (with DeePMD plugin for model_devi), and any backend-specific software required by the selected `fp_style`. For scheduler execution, each stage environment must be explicitly activated in `resources.source_list`. license: LGPL-3.0-or-later metadata: author: MatMaster @@ -15,6 +14,13 @@ Use this skill when the user wants to prepare, explain, validate, or execute the This skill is for the main DP-GEN iterative loop: train an ensemble of deep potential models, explore configuration space via LAMMPS MD, select uncertain structures, label them with first-principles calculations, and feed new data back into training. +## Requirements + +Preparation and validation require Python and DP-GEN. Actual workflow execution +also requires DeePMD-kit, a compatible exploration engine such as LAMMPS, and +the software selected by `fp_style`. Scheduler stages must activate their +runtime explicitly in `resources.source_list`. + ## Core Rule (Critical) DP-GEN run always uses **two parameter classes** and therefore **two JSON files**: @@ -52,7 +58,7 @@ These are verified failure modes discovered through testing. Treat as hard rules 1. **`type_map.raw` ordering** — The `type_map.raw` in your `init_data_sys` directories must exactly match the `type_map` array in `param.json`. Mismatches cause silent data corruption. -1. **DeePMD-kit input format** — `default_training_param` uses nested structure: `model.descriptor`, `model.fitting_net`, `learning_rate`, `loss`, `training`. The `training` block may leave `training_data.systems` unset or empty — dpgen fills it. Check the configured `deepmd_version` and installed DeePMD-kit before using 3.x-specific features. +1. **DeePMD-kit input format** — `default_training_param` uses `model`, `learning_rate`, `loss`, and `training` blocks. DeePMD-kit 2.x–3.x place systems under `training.training_data`, while 1.x uses `training.systems`; dpgen fills the version-appropriate systems field. Check the configured `deepmd_version` and installed DeePMD-kit before generating the rest of the training input or using 3.x-specific features. 1. **`model_devi_engine` defaults to LAMMPS** — No need to set `model_devi_engine` explicitly when using LAMMPS. Only set it for alternative engines. @@ -201,15 +207,15 @@ Collect the following information before generating files. ### System information - `type_map` — ordered element symbols -- `mass_map` — atomic masses matching `type_map` order -- `init_data_prefix` — prefix for init_data_sys paths +- `mass_map` — optional atomic masses matching `type_map` order; defaults to `"auto"` +- `init_data_prefix` — optional prefix for `init_data_sys` paths - `init_data_sys` — list of paths to initial training data (deepmd/npy format) -- `sys_configs_prefix` — prefix for sys_configs paths +- `sys_configs_prefix` — optional prefix for `sys_configs` paths - `sys_configs` — list of lists of structure file paths ### Training setup -- `numb_models` — number of ensemble models (default: 4) +- `numb_models` — number of ensemble models (4 is recommended) - `default_training_param` — full DeePMD training input: - `model.descriptor` — descriptor type and settings - `model.fitting_net` — fitting network settings @@ -234,7 +240,7 @@ Collect the following information before generating files. ### FP setup -- `fp_style` — backend: `"vasp"`, `"cp2k"`, `"abacus"`, `"gaussian"`, `"pwscf"`, or `"none"` +- `fp_style` — a labeling backend registered in the [run argument schema](../../dpgen/generator/arginfo.py), such as `"vasp"`, `"cp2k"`, `"abacus"`, `"gaussian"`, or `"pwscf"`; `"none"` is not a valid current choice - `fp_task_max` — maximum number of FP tasks per iteration - `fp_task_min` — minimum number to trigger FP - Backend-specific settings: @@ -283,12 +289,12 @@ Construct `param.json` around these logical blocks: Key fields always required: - `type_map` -- `mass_map` - `init_data_sys` - `sys_configs` - `numb_models` - `default_training_param` - `model_devi_dt` +- `model_devi_skip` - `model_devi_f_trust_lo` - `model_devi_f_trust_hi` - `model_devi_jobs` @@ -357,10 +363,9 @@ Before execution, validate the workflow in this order: - Step 3: verify `init_data_sys` directories exist and contain proper deepmd/npy format: - ```bash - # Each directory must have type_map.raw, type.raw, set.000/ - ls init_data/*/type_map.raw - ``` + Resolve each `init_data_sys` entry against `init_data_prefix` when a prefix + is set; do not assume the literal directory `init_data/`. Each NumPy system + must contain `type_map.raw`, `type.raw`, and `set.000/`. - Step 4: verify `type_map.raw` content matches `param.json` `type_map` ordering. @@ -395,6 +400,7 @@ Always provide: - Keep `type_map` ordering consistent with `init_data_sys` type_map.raw files. - If required inputs are missing, stop and ask instead of guessing. - Always spell `se_atten_v2` correctly (not `se_attn_v2`). +- Do not use `fp_style: "none"`; stop before `dpgen run` when only preparation or validation was authorized. - Include `remote_root` when the selected context requires it; do not add it solely to rewrite a working lazy/local configuration. - Use a `batch_type` accepted by the installed dpdispatcher version and @@ -403,13 +409,12 @@ Always provide: - Do not assume outer-shell activation is inherited by stage jobs; for scheduler execution, require explicit `source_list` per stage. - If the user already has working templates, patch them rather than overwriting them blindly. - Do not set `model_devi_engine` unless using a non-LAMMPS engine (it defaults to LAMMPS). -- In `default_training_param`, leave `training.training_data.systems` unset or empty — `dpgen/generator/run.py` fills it from `init_data_sys` automatically. +- In `default_training_param`, leave `training.training_data.systems` unset or empty for DeePMD-kit 2.x–3.x, or `training.systems` unset or empty for 1.x — `dpgen/generator/run.py` fills it from `init_data_sys` automatically. ## Repository examples and references Use these checked-in repository examples as starting points: -- [local CH4 parameter example](../../examples/run/ch4/param.json) - [local machine example](../../examples/machine/DeePMD-kit-1.x/machine-local.json) - [CP2K parameter example](../../examples/run/dp2.x-lammps-cp2k/param_CH4_deepmd-kit-2.0.1.json) - [VASP parameter example](../../examples/run/dp2.x-lammps-vasp/CH4/param_CH4_deepmd-kit-2.x.json) diff --git a/tests/test_dpgen_run_skill.py b/tests/test_dpgen_run_skill.py index c5f9abbef..baef98b85 100644 --- a/tests/test_dpgen_run_skill.py +++ b/tests/test_dpgen_run_skill.py @@ -3,7 +3,9 @@ import unittest from pathlib import Path +from dpgen.generator.arginfo import run_jdata_arginfo from dpgen.remote.decide_machine import convert_mdata +from dpgen.util import normalize LINK_PATTERN = re.compile(r"\[[^]]+\]\(([^)]+)\)") @@ -33,6 +35,34 @@ def test_all_repository_relative_links_exist(self): + ", ".join(missing), ) + def test_linked_parameter_examples_use_current_schema(self): + repository_root = Path(__file__).resolve().parents[1] + skill_path = repository_root / "skills" / "dpgen-run" / "SKILL.md" + parameter_examples_root = repository_root / "examples" / "run" + examples = set() + for target in LINK_PATTERN.findall(skill_path.read_text()): + target_path = target.split("#", 1)[0] + if not target_path.endswith(".json"): + continue + resolved_target = (skill_path.parent / target_path).resolve() + try: + resolved_target.relative_to(parameter_examples_root) + except ValueError: + continue + examples.add(resolved_target) + self.assertTrue(examples, "No examples/run/*.json links found") + + for example in examples: + with self.subTest(example=example): + parameter_data = json.loads(example.read_text()) + normalized = normalize( + run_jdata_arginfo(), + parameter_data, + strict_check=False, + ) + self.assertIsInstance(normalized, dict) + self.assertTrue(normalized) + def test_linked_machine_examples_use_current_schema(self): repository_root = Path(__file__).resolve().parents[1] skill_path = repository_root / "skills" / "dpgen-run" / "SKILL.md" From b7c2a8fe5d64acdece7f7c305368faa00ed60db3 Mon Sep 17 00:00:00 2001 From: SchrodingersCattt Date: Sun, 30 Aug 2026 01:52:35 +0000 Subject: [PATCH 12/13] refactor(skills): load dpgen-run references progressively --- skills/dpgen-run/SKILL.md | 437 ++---------------- skills/dpgen-run/references/machine-json.md | 56 +++ skills/dpgen-run/references/param-json.md | 107 +++++ .../references/validation-and-run.md | 85 ++++ skills/dpgen-run/references/workflow.md | 97 ++++ tests/test_dpgen_run_skill.py | 106 +++-- 6 files changed, 436 insertions(+), 452 deletions(-) create mode 100644 skills/dpgen-run/references/machine-json.md create mode 100644 skills/dpgen-run/references/param-json.md create mode 100644 skills/dpgen-run/references/validation-and-run.md create mode 100644 skills/dpgen-run/references/workflow.md diff --git a/skills/dpgen-run/SKILL.md b/skills/dpgen-run/SKILL.md index d30cec8e6..039108a35 100644 --- a/skills/dpgen-run/SKILL.md +++ b/skills/dpgen-run/SKILL.md @@ -1,428 +1,45 @@ --- name: dpgen-run -description: Prepare, explain, validate, and run DP-GEN concurrent learning workflows for training deep potential models via iterative exploration. Use when the user wants to generate or modify `param.json` and `machine.json` for `dpgen run`, configure training/exploration/labeling iterations, select descriptor types, set trust levels, define model_devi_jobs, or inspect run outputs. +description: Prepare, explain, validate, and run DP-GEN concurrent-learning workflows. Use for dpgen run param.json or machine.json, training and exploration settings, labeling backends, trust levels, model_devi_jobs, execution, or output inspection. license: LGPL-3.0-or-later metadata: author: MatMaster - version: 0.1.0 + version: 0.2.0 repository: https://github.com/deepmodeling/dpgen --- -# DP-GEN Run (Concurrent Learning) +# DP-GEN Run -Use this skill when the user wants to prepare, explain, validate, or execute the `dpgen run` concurrent learning workflow. +Use this skill for the iterative DP-GEN loop: train an ensemble, explore with MD, select uncertain structures, label them, and add the labels to training data. -This skill is for the main DP-GEN iterative loop: train an ensemble of deep potential models, explore configuration space via LAMMPS MD, select uncertain structures, label them with first-principles calculations, and feed new data back into training. +## Core contract -## Requirements +- Keep workflow settings in `param.json` and execution settings in `machine.json`. +- The exact launcher is `dpgen run param.json machine.json`. +- Treat the launcher shell and dispatched stage environments as separate layers. +- Prepare and validate by default. Execute only when the user explicitly requests it and confirms the exact command after validation. -Preparation and validation require Python and DP-GEN. Actual workflow execution -also requires DeePMD-kit, a compatible exploration engine such as LAMMPS, and -the software selected by `fp_style`. Scheduler stages must activate their -runtime explicitly in `resources.source_list`. +## Workflow -## Core Rule (Critical) +1. Confirm this is a `dpgen run` task and inspect existing files first. +1. Collect only missing inputs; patch working configurations instead of rebuilding them. +1. Load only the references needed for the current task. +1. Preserve scientific and site-specific choices unless the user approves changes. +1. Validate files, data, commands, and environments before proposing execution. +1. Report exact paths, the command, unresolved fields, and post-run inspection targets. -DP-GEN run always uses **two parameter classes** and therefore **two JSON files**: +## Progressive references -- **Workflow parameters** -> `param.json` -- **Execution / machine parameters** -> `machine.json` +Read only the relevant files: -Run exactly: +- [Workflow policy](references/workflow.md): scope, requirements, missing-input policy, layout, and reporting. +- [param.json guide](references/param-json.md): systems, training, exploration, FP fields, descriptors, thresholds, and examples. +- [machine.json guide](references/machine-json.md): stage commands, contexts, resources, scheduler environments, and examples. +- [Validation and execution](references/validation-and-run.md): schema checks, data checks, authorization, launch, and output inspection. -```bash -dpgen run param.json machine.json -``` +## Non-negotiable guardrails -Environment boundary rule: - -- Outer layer: run `dpgen run param.json machine.json` in an activated environment where `dpgen -h` works. -- Inner layer: for scheduler stages, explicitly activate runtime in `resources.source_list` on the server side. - -## Critical Pitfalls (Must Embed) - -These are verified failure modes discovered through testing. Treat as hard rules: - -1. **`se_atten_v2` spelling** — The descriptor type is `se_atten_v2` (double-t in "atten"). Writing `se_attn_v2` will silently fail or error. Always verify the exact string. - -1. **`remote_root` is context-dependent** — require it for remote or scheduler - contexts whose installed dpdispatcher schema needs a remote working root. - Lazy/local contexts can omit it. Preserve an existing working omission and - validate against the installed dpdispatcher version. - -1. **`batch_type` must be registered** — canonical names and lowercase aliases - can both be valid. Preserve an existing working value and validate new values - against the installed dpdispatcher version. - -1. **CP2K `user_fp_params` KIND format** — When multiple element kinds are needed, use `"_": ["H", "O"]` with parallel arrays for `POTENTIAL` and `BASIS_SET`. This maps to repeated KIND sections. - -1. **`type_map.raw` ordering** — The `type_map.raw` in your `init_data_sys` directories must exactly match the `type_map` array in `param.json`. Mismatches cause silent data corruption. - -1. **DeePMD-kit input format** — `default_training_param` uses `model`, `learning_rate`, `loss`, and `training` blocks. DeePMD-kit 2.x–3.x place systems under `training.training_data`, while 1.x uses `training.systems`; dpgen fills the version-appropriate systems field. Check the configured `deepmd_version` and installed DeePMD-kit before generating the rest of the training input or using 3.x-specific features. - -1. **`model_devi_engine` defaults to LAMMPS** — No need to set `model_devi_engine` explicitly when using LAMMPS. Only set it for alternative engines. - -1. **`fp_style: "cp2k"` is native** — dpgen v0.12+ supports CP2K natively via `user_fp_params` dict. No external plugins required. - -1. **`sys_configs` format** — List of lists: outer list = systems, inner list = file paths (POSCAR/xyz) for that system. Example: `[["path/to/POSCAR_1"], ["path/to/POSCAR_2"]]`. - -1. **`init_data_sys` format** — List of paths to directories in deepmd/npy format. Each directory must contain `type_map.raw`, `type.raw`, and `set.000/` with numpy arrays. - -## Agent responsibilities - -When using this skill, the agent should: - -1. confirm that the task is a `dpgen run` concurrent learning workflow -1. check whether existing configs or templates are already available -1. collect only the missing training, exploration, FP, and machine inputs -1. select appropriate descriptor type based on system complexity -1. generate or patch `param.json` -1. generate or patch `machine.json` -1. explain important parameters in plain language when asked -1. validate the workflow before execution -1. provide the exact command for running -1. after execution, summarize outputs and next inspection targets - -## Working policy - -### 0. Treat execution as separately authorized - -Prepare, explain, and validate by default. Execute `dpgen run` only when the -user has explicitly requested execution and, after validation, confirmed the -exact command. Supplying configuration files, passing validation, or showing the -command does not by itself authorize an HPC or first-principles workload. - -### 1. Ask only for missing inputs - -Do not ask the user for everything if part of the configuration is already available. - -If the user already provides: - -- a partial `param.json` -- a partial `machine.json` -- a known training template -- a known cluster template - -then patch those files instead of rebuilding everything from scratch. - -### 2. Preserve the user's scientific choices - -Do not silently change: - -- descriptor family or type -- fitting net structure -- fp backend -- trust thresholds -- `type_map` ordering -- `model_devi_jobs` schedule -- ensemble or temperature settings - -If a value looks scientifically questionable, explain the concern instead of silently replacing it. - -### 3. Descriptor selection guidance - -dpgen exposes `train_backend` options for `"tensorflow"` and `"pytorch"`, but that option alone does not guarantee every PyTorch descriptor or every DeePMD-kit 3.x feature works in the user's installed stack. If backend or descriptor compatibility is unclear, verify it against the repo's `train_backend` options, the configured `deepmd_version`, and the installed DeePMD-kit version before generating configs. - -When the user has not specified a descriptor type, recommend based on: - -| Descriptor | Use case | Notes | -| ------------- | -------------------------------------------- | ----------------------------------------------------------------------------------- | -| `se_e2_a` | Simple systems, fast training, well-tested | Classic two-body embedding, requires explicit `sel` list | -| `se_atten` | Multi-component systems, moderate complexity | Attention-based, supports `sel: "auto"` | -| `se_atten_v2` | Modern default, best accuracy/cost balance | **Spell exactly `se_atten_v2`** (double-t). Supports `sel: "auto"`, `attn_layer: 0` | - -For new projects, prefer `se_atten_v2` unless the user has specific needs. If the user requires PyTorch-only models or DPA-2/DPA-3, prefer **dpgen2** unless the current dpgen and DeePMD-kit installation are verified to support the requested model. - -### 4. Keep local and scheduler execution explicit - -If the user wants local execution, produce local-friendly commands. - -If the user wants scheduler execution, produce scheduler-friendly commands and keep queue, partition, and resource requests explicit. - -Do not invent scheduler module names or executable paths. - -### 5. Do not invent environment activation commands - -If the user already has a working activation command such as: - -- `conda activate ...` -- `module load ...` -- `source ...` - -reuse it exactly. - -If execution is requested and the activation method is unknown, ask the user for the precise activation command. - -Do not guess conda environment names, module names, or site-specific paths. - -### 5.1 Outer launcher policy - -Use an activated DP-GEN environment and verify with: - -```bash -dpgen -h -``` - -Do not start run from a shell where `dpgen` is unavailable. - -### 5.2 Outer vs inner runtime boundaries (critical) - -Treat run execution as two separate environment layers: - -1. Outer layer: the shell that launches `dpgen run param.json machine.json` (must have `dpgen` in PATH) -1. Inner layer: stage tasks dispatched by DP-GEN (`train` / `model_devi` / `fp`) on server/runtime side - -Even if the outer layer is correct, inner stage tasks still need explicit runtime setup in `machine.json`. -Do not assume the outer shell environment will be inherited by dispatched stage jobs. -For scheduler-style execution, `resources.source_list` must explicitly activate the required runtime environment. - -### 6. Prefer reproducible output layout - -When generating a run workflow, keep files organized and predictable. - -Recommended structure: - -```text -project/ -├── param.json -├── machine.json -├── init_data/ -│ └── system_000/ -│ ├── type_map.raw -│ ├── type.raw -│ └── set.000/ -├── assets/ -│ └── structures/ -│ └── POSCAR_* -├── cp2k_basis_pp_file/ (if using CP2K) -│ ├── BASIS_MOLOPT -│ └── GTH_POTENTIALS -└── iter.*/ (created by dpgen) -``` - -## Minimum required inputs - -Collect the following information before generating files. - -### System information - -- `type_map` — ordered element symbols -- `mass_map` — optional atomic masses matching `type_map` order; defaults to `"auto"` -- `init_data_prefix` — optional prefix for `init_data_sys` paths -- `init_data_sys` — list of paths to initial training data (deepmd/npy format) -- `sys_configs_prefix` — optional prefix for `sys_configs` paths -- `sys_configs` — list of lists of structure file paths - -### Training setup - -- `numb_models` — number of ensemble models (4 is recommended) -- `default_training_param` — full DeePMD training input: - - `model.descriptor` — descriptor type and settings - - `model.fitting_net` — fitting network settings - - `learning_rate` — learning rate schedule - - `loss` — loss function weights - - `training` — training steps and output settings - -### Exploration setup (model_devi) - -- `model_devi_dt` — MD timestep in ps (e.g. 0.0005 = 0.5 fs) -- `model_devi_skip` — number of initial frames to skip -- `model_devi_f_trust_lo` — lower force deviation threshold (eV/Å) -- `model_devi_f_trust_hi` — upper force deviation threshold (eV/Å) -- `model_devi_clean_traj` — whether to clean trajectory files after selection -- `model_devi_jobs` — list of exploration job specifications: - - `sys_idx` — which systems to explore - - `temps` — temperatures (K) - - `press` — pressures (bar) - - `trj_freq` — trajectory save frequency - - `nsteps` — number of MD steps - - `ensemble` — `"nvt"` or `"npt"` - -### FP setup - -- `fp_style` — a labeling backend registered in the [run argument schema](../../dpgen/generator/arginfo.py), such as `"vasp"`, `"cp2k"`, `"abacus"`, `"gaussian"`, or `"pwscf"`; `"none"` is not a valid current choice -- `fp_task_max` — maximum number of FP tasks per iteration -- `fp_task_min` — minimum number to trigger FP -- Backend-specific settings: - - VASP: `fp_pp_path`, `fp_pp_files`, and `fp_incar` - - CP2K: `user_fp_params` (nested dict representing cp2k input) - - ABACUS: `user_fp_params`, `fp_pp_path`, `fp_pp_files`, `fp_orb_files` - - Gaussian: `fp_params` (keywords, nproc, multiplicity) - - PWSCF: `user_fp_params` - -### Execution setup - -For each stage `train`, `model_devi`, and `fp`, collect or preserve: - -- `command` -- `machine.batch_type` -- `machine.context_type` -- `machine.local_root` -- `machine.remote_root` when required by the selected context; preserve a - working omission for lazy/local execution -- `resources.number_node` -- `resources.cpu_per_node` -- `resources.gpu_per_node` -- `resources.group_size` -- `resources.source_list` (required for scheduler jobs; use it to activate environment explicitly) -- any explicit queue / partition / custom scheduler flags if the user already uses them - -Choose a runtime profile first, then fill the matching template: - -- server-local scheduler: - [existing scheduler example](../../examples/machine/DeePMD-kit-1.x/machine-lsf-slurm-cp2k.json) -- pure local shell testing: - [existing local example](../../examples/machine/DeePMD-kit-1.x/machine-local.json) - -## How to build `param.json` - -Construct `param.json` around these logical blocks: - -1. element and mass definitions (`type_map`, `mass_map`) -1. data source configuration (`init_data_prefix`, `init_data_sys`, `sys_configs_prefix`, `sys_configs`) -1. model ensemble count (`numb_models`) -1. default DeePMD training parameters (`default_training_param`) -1. exploration settings (`model_devi_dt`, `model_devi_skip`, `model_devi_f_trust_lo/hi`, `model_devi_clean_traj`) -1. exploration job schedule (`model_devi_jobs`) -1. FP backend settings (`fp_style`, `fp_task_max`, `fp_task_min`, backend-specific params) - -Key fields always required: - -- `type_map` -- `init_data_sys` -- `sys_configs` -- `numb_models` -- `default_training_param` -- `model_devi_dt` -- `model_devi_skip` -- `model_devi_f_trust_lo` -- `model_devi_f_trust_hi` -- `model_devi_jobs` -- `fp_style` -- `fp_task_max` -- `fp_task_min` - -Trust level guidance: - -- `model_devi_f_trust_lo`: structures below this are "accurate" — not selected -- `model_devi_f_trust_hi`: structures above this are "failed" — discarded -- Between lo and hi: "candidate" — selected for FP labeling -- Typical starting values: lo=0.05–0.10, hi=0.15–0.30 (eV/Å) -- Tighter thresholds = more FP cost, better accuracy -- System-specific tuning is recommended after initial iterations - -Official reference examples: - -- CP2K: - [methane example](../../examples/run/dp2.x-lammps-cp2k/param_CH4_deepmd-kit-2.0.1.json) -- VASP: - [CH4 example](../../examples/run/dp2.x-lammps-vasp/CH4/param_CH4_deepmd-kit-2.x.json) - -## How to build `machine.json` - -Construct `machine.json` with separate stage blocks for: - -- `train` -- `model_devi` -- `fp` - -For each stage, keep the following explicit: - -- `command` — the executable (`dp` for train, `lmp` for model_devi, FP command for fp) -- machine or context configuration -- resources -- queue or partition if needed -- cpu and gpu counts -- custom scheduler flags -- environment activation commands - -Do not merge all stages into one vague machine block. - -Stage-specific commands: - -- `train`: `"dp"` (DeePMD-kit training) -- `model_devi`: `"lmp"` (LAMMPS with DeePMD plugin) -- `fp`: backend-specific (`"vasp_std"`, `"cp2k.popt"`, `"abacus"`, `"pw.x"`, `"g16"`) - -## Validation before run - -Before execution, validate the workflow in this order: - -- Step 1: confirm outer-layer `dpgen` is available: - - ```bash - dpgen -h - ``` - -- Step 2: validate JSON syntax: - - ```bash - python -m json.tool param.json - python -m json.tool machine.json - ``` - -- Step 3: verify `init_data_sys` directories exist and contain proper deepmd/npy format: - - Resolve each `init_data_sys` entry against `init_data_prefix` when a prefix - is set; do not assume the literal directory `init_data/`. Each NumPy system - must contain `type_map.raw`, `type.raw`, and `set.000/`. - -- Step 4: verify `type_map.raw` content matches `param.json` `type_map` ordering. - -- Step 5: verify `sys_configs` structure files exist. - -- Step 6: verify stage commands match the selected software stack. - -- Step 7: for CP2K, verify basis set and potential files are accessible. - -- Step 8: only then run: - - ```bash - dpgen run param.json machine.json - ``` - -## Output contract - -Always provide: - -1. final absolute paths to `param.json` and `machine.json` -1. the exact run command (`dpgen run param.json machine.json`) -1. a short pre-run checklist -1. any unresolved required fields -1. if execution was performed, the main output locations and next files to inspect - -## Guardrails - -- Never merge workflow and machine parameters into one file. -- Never run `dpgen run` before both JSON files are present. -- Never hardcode personal cluster, account, queue, or path settings as universal defaults. -- Never silently change the user's scientific choices. -- Keep `type_map` ordering consistent with `init_data_sys` type_map.raw files. -- If required inputs are missing, stop and ask instead of guessing. -- Always spell `se_atten_v2` correctly (not `se_attn_v2`). -- Do not use `fp_style: "none"`; stop before `dpgen run` when only preparation or validation was authorized. -- Include `remote_root` when the selected context requires it; do not add it - solely to rewrite a working lazy/local configuration. -- Use a `batch_type` accepted by the installed dpdispatcher version and - preserve an existing working canonical name or alias. -- For CP2K KIND sections with multiple elements, use `"_": ["elem1", "elem2"]` array format. -- Do not assume outer-shell activation is inherited by stage jobs; for scheduler execution, require explicit `source_list` per stage. -- If the user already has working templates, patch them rather than overwriting them blindly. -- Do not set `model_devi_engine` unless using a non-LAMMPS engine (it defaults to LAMMPS). -- In `default_training_param`, leave `training.training_data.systems` unset or empty for DeePMD-kit 2.x–3.x, or `training.systems` unset or empty for 1.x — `dpgen/generator/run.py` fills it from `init_data_sys` automatically. - -## Repository examples and references - -Use these checked-in repository examples as starting points: - -- [local machine example](../../examples/machine/DeePMD-kit-1.x/machine-local.json) -- [CP2K parameter example](../../examples/run/dp2.x-lammps-cp2k/param_CH4_deepmd-kit-2.0.1.json) -- [VASP parameter example](../../examples/run/dp2.x-lammps-vasp/CH4/param_CH4_deepmd-kit-2.x.json) -- [scheduler machine example](../../examples/machine/DeePMD-kit-1.x/machine-lsf-slurm-cp2k.json) - -External references: - -- DP-GEN run overview: https://docs.deepmodeling.com/projects/dpgen/en/latest/run/index.html -- run parameter definitions: https://docs.deepmodeling.com/projects/dpgen/en/latest/run/param.html -- run machine definitions: https://docs.deepmodeling.com/projects/dpgen/en/latest/run/mdata.html -- DeePMD-kit model documentation: https://docs.deepmodeling.com/projects/deepmd/en/latest/model/index.html +- Never merge the two JSON files or invent cluster paths, queues, modules, or activation commands. +- Preserve descriptor, fitting network, FP backend, thresholds, schedules, ensemble settings, and `type_map` ordering. +- Spell `se_atten_v2` exactly; do not use `fp_style: "none"` for current `dpgen run`. +- Stop and ask when required scientific or execution inputs cannot be discovered safely. diff --git a/skills/dpgen-run/references/machine-json.md b/skills/dpgen-run/references/machine-json.md new file mode 100644 index 000000000..1ecb12a65 --- /dev/null +++ b/skills/dpgen-run/references/machine-json.md @@ -0,0 +1,56 @@ +# DP-GEN Run machine.json Guide + +Load this reference when creating, patching, explaining, or reviewing `machine.json`. + +## Runtime boundary + +Treat execution as two environments: + +1. the outer shell that launches `dpgen run param.json machine.json` +1. the dispatched `train`, `model_devi`, and `fp` tasks + +The outer shell must resolve `dpgen`. Scheduler jobs must activate their own software in `resources.source_list`; outer activation is not inherited reliably. + +## Stage blocks + +Keep separate `train`, `model_devi`, and `fp` blocks. For each stage, collect or preserve: + +- `command` +- `machine.batch_type` +- `machine.context_type` +- `machine.local_root` +- `machine.remote_root` when the selected context requires it +- `resources.number_node` +- `resources.cpu_per_node` +- `resources.gpu_per_node` +- `resources.group_size` +- `resources.source_list` for scheduler runtime activation +- queue, partition, account, and custom scheduler flags when applicable + +Typical commands are `dp` for training, `lmp` for LAMMPS exploration, and a backend executable such as `vasp_std`, `cp2k.popt`, `abacus`, `pw.x`, or `g16` for labeling. Preserve known working commands. + +## Context and batch compatibility + +`remote_root` is context-dependent. Require it for remote or scheduler contexts whose installed dpdispatcher schema needs a remote working root. Preserve a working omission for lazy or local execution and validate against the installed dpdispatcher version. + +`batch_type` must be registered by the installed dpdispatcher. Canonical names and lowercase aliases can both be valid. Preserve a working value; validate newly introduced values rather than normalizing them by assumption. + +## Starting profiles + +Choose a profile that matches where the outer command runs: + +- server-local scheduler: [scheduler example](../../../examples/machine/DeePMD-kit-1.x/machine-lsf-slurm-cp2k.json) +- pure local shell: [local example](../../../examples/machine/DeePMD-kit-1.x/machine-local.json) + +Patch the closest existing working configuration. Do not transplant site-specific paths or scheduler settings blindly. + +## Guardrails + +- Never combine all stages into one vague block. +- Never invent executables, modules, activation commands, queues, accounts, or paths. +- Keep CPU, GPU, node, and grouping requests explicit. +- Require explicit `source_list` activation for scheduler stages. +- Preserve working local omissions and installed-version aliases. +- Ensure commands match the scientific stack selected in `param.json`. + +External machine reference: https://docs.deepmodeling.com/projects/dpgen/en/latest/run/mdata.html diff --git a/skills/dpgen-run/references/param-json.md b/skills/dpgen-run/references/param-json.md new file mode 100644 index 000000000..0efc5bdec --- /dev/null +++ b/skills/dpgen-run/references/param-json.md @@ -0,0 +1,107 @@ +# DP-GEN Run param.json Guide + +Load this reference when creating, patching, explaining, or reviewing `param.json`. + +The current authoritative schema is [dpgen/generator/arginfo.py](../../../dpgen/generator/arginfo.py). + +## System and data fields + +Collect or preserve: + +- `type_map`: ordered element symbols +- `mass_map`: optional; defaults to `"auto"` +- `init_data_prefix`: optional prefix for `init_data_sys` +- `init_data_sys`: DeepMD NumPy-system directories +- `sys_configs_prefix`: optional prefix for `sys_configs` +- `sys_configs`: list of systems, each represented by a list of structure paths + +`init_data_sys` entries must resolve to directories containing `type_map.raw`, `type.raw`, and `set.000/`. The ordering in every `type_map.raw` must match `type_map`. + +`sys_configs` is a list of lists. The outer list separates systems; each inner list contains structure files for that system. + +## Training fields + +Required core inputs include: + +- `numb_models`: ensemble size; four is a common recommendation, not a schema default +- `default_training_param.model` +- `default_training_param.learning_rate` +- `default_training_param.loss` +- `default_training_param.training` + +DeePMD-kit 1.x uses `training.systems`. DeePMD-kit 2.x and 3.x use `training.training_data.systems`. Leave the version-appropriate systems value unset or empty because DP-GEN fills it from `init_data_sys`. Verify `deepmd_version` and installed DeePMD-kit before using version-specific features. + +### Descriptor selection + +Do not change a user-selected descriptor silently. + +| Descriptor | Typical use | Notes | +| ------------- | --------------------------------------------- | ------------------------------------- | +| `se_e2_a` | simple systems and conservative compatibility | explicit `sel` list | +| `se_atten` | multicomponent systems | supports `sel: "auto"` | +| `se_atten_v2` | modern attention workflow | spell exactly; supports `sel: "auto"` | + +DP-GEN exposes `train_backend` values `"tensorflow"` and `"pytorch"`, but this alone does not guarantee every descriptor or DeePMD-kit feature works in the installed stack. Verify backend, `deepmd_version`, and DeePMD-kit compatibility. Prefer dpgen2 when a requested DPA-2/DPA-3 or PyTorch-only workflow is not verified with this DP-GEN installation. + +## Exploration fields + +Collect or preserve: + +- `model_devi_dt`: MD timestep in ps +- `model_devi_skip`: initial frames to skip +- `model_devi_f_trust_lo` +- `model_devi_f_trust_hi` +- `model_devi_clean_traj` +- `model_devi_jobs` + +Each `model_devi_jobs` item normally defines `sys_idx`, `temps`, `press`, `trj_freq`, `nsteps`, and `ensemble`. + +Do not set `model_devi_engine` for ordinary LAMMPS exploration; LAMMPS is the default. Set it only for a verified alternative engine. + +### Trust-window interpretation + +- below `model_devi_f_trust_lo`: accurate, normally not selected +- between low and high: candidate for labeling +- above `model_devi_f_trust_hi`: failed or unreliable, normally discarded + +Values such as 0.05-0.10 eV/Angstrom for the low threshold and 0.15-0.30 eV/Angstrom for the high threshold are starting ranges, not universal defaults. Tighter thresholds increase labeling cost and require system-specific justification. + +## First-principles fields + +`fp_style` must be a backend accepted by the current schema, for example `"vasp"`, `"cp2k"`, `"abacus"`, `"gaussian"`, or `"pwscf"`. The value `"none"` is not valid for current `dpgen run`. + +Always collect: + +- `fp_style` +- `fp_task_max` +- `fp_task_min` + +Backend-specific inputs include: + +- VASP: `fp_pp_path`, `fp_pp_files`, `fp_incar` +- CP2K: `user_fp_params` +- ABACUS: `user_fp_params`, `fp_pp_path`, `fp_pp_files`, `fp_orb_files` +- Gaussian: `fp_params` +- PWSCF: `user_fp_params` + +CP2K is supported natively. For multiple KIND sections in `user_fp_params`, use `"_": ["H", "O"]` with parallel `POTENTIAL` and `BASIS_SET` arrays. + +## Logical construction order + +1. elements and masses +1. initial data and exploration structures +1. model ensemble and DeePMD training input +1. model-deviation controls and job schedule +1. first-principles backend and task limits + +Do not assume a field is optional from an old example. Validate against the current schema. + +## Checked repository examples + +- [CP2K methane parameters](../../../examples/run/dp2.x-lammps-cp2k/param_CH4_deepmd-kit-2.0.1.json) +- [VASP CH4 parameters](../../../examples/run/dp2.x-lammps-vasp/CH4/param_CH4_deepmd-kit-2.x.json) + +External references: + +- https://docs.deepmodeling.com/projects/dpgen/en/latest/run/param.html +- https://docs.deepmodeling.com/projects/deepmd/en/latest/model/index.html diff --git a/skills/dpgen-run/references/validation-and-run.md b/skills/dpgen-run/references/validation-and-run.md new file mode 100644 index 000000000..8ae59c4ed --- /dev/null +++ b/skills/dpgen-run/references/validation-and-run.md @@ -0,0 +1,85 @@ +# DP-GEN Run Validation and Execution + +Load this reference before claiming a configuration is valid or before executing `dpgen run`. + +## 1. Verify the outer environment + +```bash +dpgen -h +``` + +Stop if the launcher is unavailable. Actual execution also requires DeePMD-kit, the exploration engine, the selected first-principles software, and scheduler access where applicable. + +## 2. Check JSON syntax + +```bash +python -m json.tool param.json +python -m json.tool machine.json +``` + +## 3. Validate against installed schemas + +Use the installed DP-GEN and dpdispatcher code, not documentation alone. + +```python +import json + +from dpgen.generator.arginfo import run_jdata_arginfo +from dpgen.remote.decide_machine import convert_mdata +from dpgen.util import normalize + +with open("param.json") as stream: + normalize(run_jdata_arginfo(), json.load(stream), strict_check=False) + +with open("machine.json") as stream: + convert_mdata(json.load(stream)) +``` + +Schema normalization is necessary but not sufficient; it does not prove paths, executables, scheduler permissions, or scientific choices are correct. + +## 4. Validate data and structures + +Resolve every `init_data_sys` entry against `init_data_prefix` when set. Verify each NumPy system contains: + +- `type_map.raw` +- `type.raw` +- `set.000/` with the expected arrays + +Compare every `type_map.raw` line-for-line with `param.json.type_map`. Verify every structure referenced through `sys_configs_prefix` and `sys_configs` exists. + +## 5. Validate scientific stages + +Confirm: + +- DeePMD-kit version, backend, descriptor, and training-input layout agree +- each exploration job references valid systems and MD settings +- force trust thresholds are ordered and scientifically intentional +- FP inputs and support files match `fp_style` +- CP2K basis and potential files are accessible when CP2K is selected +- `fp_task_min` and `fp_task_max` reflect intended cost limits + +## 6. Validate execution stages + +For `train`, `model_devi`, and `fp`, verify: + +- commands exist in the dispatched environment +- context and batch types are accepted by installed dpdispatcher +- required local and remote roots exist +- scheduler resources and flags are valid +- `resources.source_list` activates the correct inner runtime + +A successful outer `dpgen -h` does not validate dispatched environments. + +## 7. Confirm before launch + +Show the exact files, validation summary, unresolved risks, and command: + +```bash +dpgen run param.json machine.json +``` + +Execute only after the user explicitly requests the run and confirms this exact validated command. If the user requested only preparation or validation, stop here. + +## 8. Inspect outputs + +After launch, report the current `iter.*` directory and the status of training, model-deviation, and FP tasks. Summarize failures from stage logs and report selected and labeled structure counts when available. diff --git a/skills/dpgen-run/references/workflow.md b/skills/dpgen-run/references/workflow.md new file mode 100644 index 000000000..0d9f69e81 --- /dev/null +++ b/skills/dpgen-run/references/workflow.md @@ -0,0 +1,97 @@ +# DP-GEN Run Workflow Policy + +Load this reference for end-to-end preparation, task scoping, project layout, or result reporting. + +## Scope and requirements + +`dpgen run` implements the concurrent-learning loop: + +1. train an ensemble of Deep Potential models +1. explore configuration space, normally with LAMMPS MD +1. select configurations in the model-deviation candidate window +1. label selected structures with the configured first-principles backend +1. add labeled data to the next training iteration + +Preparation and validation require Python and DP-GEN. Execution additionally requires DeePMD-kit, a compatible exploration engine, the software selected by `fp_style`, and any scheduler runtime. + +The workflow always uses: + +- `param.json` for scientific and workflow parameters +- `machine.json` for commands, contexts, and resources +- `dpgen run param.json machine.json` as the launcher + +## Working policy + +### Inspect before asking + +Look for existing configuration files, training inputs, dataset metadata, and machine templates. Ask only for values that cannot be discovered. Patch existing working files rather than rebuilding them without need. + +### Preserve scientific choices + +Do not silently change: + +- descriptor family or settings +- fitting network +- training backend +- first-principles backend +- trust thresholds +- `type_map` ordering +- `model_devi_jobs` schedule +- ensemble size, temperatures, pressures, or MD ensembles + +Explain a concern and request direction before changing a scientific choice. + +### Preserve site-specific choices + +Reuse known activation commands, executable names, queues, partitions, accounts, paths, and scheduler flags exactly. Never guess conda environments, modules, personal paths, or cluster policy. + +### Keep execution separately authorized + +Preparing files, validating them, or displaying a command does not authorize an HPC, MD, training, or first-principles workload. Run only when the user explicitly requests execution and confirms the exact validated command. + +## Recommended layout + +```text +project/ +|-- param.json +|-- machine.json +|-- init_data/ +| `-- system_000/ +| |-- type_map.raw +| |-- type.raw +| `-- set.000/ +|-- assets/ +| `-- structures/ +|-- cp2k_basis_pp_file/ # only when CP2K needs it +`-- iter.*/ # created by DP-GEN +``` + +Keep repeated experiments in separate, clearly named directories derived from one reviewed base configuration. + +## Result contract + +Before execution, report: + +1. absolute paths to `param.json` and `machine.json` +1. exact launcher command +1. validation results +1. unresolved required inputs +1. expected cost-bearing stages + +After execution, report: + +1. run and current iteration status +1. failed or pending stages +1. main output and log locations +1. candidate and labeled counts when available +1. the next files that need inspection + +## General guardrails + +- Never run before both JSON files exist and pass validation. +- Keep `type_map` consistent from data through training and labeling. +- Do not overwrite working user templates blindly. +- Do not assume outer-shell activation reaches dispatched jobs. +- Stop rather than guess a missing scientific or site-specific value. + +Official overview: https://docs.deepmodeling.com/projects/dpgen/en/latest/run/index.html diff --git a/tests/test_dpgen_run_skill.py b/tests/test_dpgen_run_skill.py index baef98b85..ab9787110 100644 --- a/tests/test_dpgen_run_skill.py +++ b/tests/test_dpgen_run_skill.py @@ -8,25 +8,61 @@ from dpgen.util import normalize LINK_PATTERN = re.compile(r"\[[^]]+\]\(([^)]+)\)") +SKILL_RELATIVE_PATH = Path("skills") / "dpgen-run" / "SKILL.md" -class TestDPGenRunSkill(unittest.TestCase): - def test_all_repository_relative_links_exist(self): - repository_root = Path(__file__).resolve().parents[1] - skill_path = repository_root / "skills" / "dpgen-run" / "SKILL.md" +def skill_markdown_files(repository_root): + skill_root = repository_root / SKILL_RELATIVE_PATH.parent + return sorted(skill_root.rglob("*.md")) - missing = [] - for target in LINK_PATTERN.findall(skill_path.read_text()): - if "://" in target or target.startswith("#"): + +def linked_json_examples(repository_root, examples_root): + examples = set() + for markdown_path in skill_markdown_files(repository_root): + for target in LINK_PATTERN.findall(markdown_path.read_text()): + target_path = target.split("#", 1)[0] + if not target_path.endswith(".json"): continue - resolved_target = (skill_path.parent / target.split("#", 1)[0]).resolve() + resolved_target = (markdown_path.parent / target_path).resolve() try: - resolved_target.relative_to(repository_root) + resolved_target.relative_to(examples_root) except ValueError: - missing.append(target) - else: - if not resolved_target.exists(): - missing.append(target) + continue + examples.add(resolved_target) + return examples + + +class TestDPGenRunSkill(unittest.TestCase): + def test_main_skill_is_small_progressive_router(self): + repository_root = Path(__file__).resolve().parents[1] + skill_path = repository_root / SKILL_RELATIVE_PATH + skill_text = skill_path.read_text() + + self.assertLessEqual(len(skill_text.splitlines()), 50) + reference_targets = { + target + for target in LINK_PATTERN.findall(skill_text) + if target.startswith("references/") + } + self.assertEqual(len(reference_targets), 4) + + def test_all_repository_relative_links_exist(self): + repository_root = Path(__file__).resolve().parents[1] + missing = [] + for markdown_path in skill_markdown_files(repository_root): + for target in LINK_PATTERN.findall(markdown_path.read_text()): + if "://" in target or target.startswith("#"): + continue + resolved_target = ( + markdown_path.parent / target.split("#", 1)[0] + ).resolve() + try: + resolved_target.relative_to(repository_root) + except ValueError: + missing.append(f"{markdown_path.name}: {target}") + else: + if not resolved_target.exists(): + missing.append(f"{markdown_path.name}: {target}") self.assertEqual( missing, @@ -37,19 +73,8 @@ def test_all_repository_relative_links_exist(self): def test_linked_parameter_examples_use_current_schema(self): repository_root = Path(__file__).resolve().parents[1] - skill_path = repository_root / "skills" / "dpgen-run" / "SKILL.md" - parameter_examples_root = repository_root / "examples" / "run" - examples = set() - for target in LINK_PATTERN.findall(skill_path.read_text()): - target_path = target.split("#", 1)[0] - if not target_path.endswith(".json"): - continue - resolved_target = (skill_path.parent / target_path).resolve() - try: - resolved_target.relative_to(parameter_examples_root) - except ValueError: - continue - examples.add(resolved_target) + examples_root = repository_root / "examples" / "run" + examples = linked_json_examples(repository_root, examples_root) self.assertTrue(examples, "No examples/run/*.json links found") for example in examples: @@ -65,19 +90,8 @@ def test_linked_parameter_examples_use_current_schema(self): def test_linked_machine_examples_use_current_schema(self): repository_root = Path(__file__).resolve().parents[1] - skill_path = repository_root / "skills" / "dpgen-run" / "SKILL.md" - machine_examples_root = repository_root / "examples" / "machine" - examples = set() - for target in LINK_PATTERN.findall(skill_path.read_text()): - target_path = target.split("#", 1)[0] - if not target_path.endswith(".json"): - continue - resolved_target = (skill_path.parent / target_path).resolve() - try: - resolved_target.relative_to(machine_examples_root) - except ValueError: - continue - examples.add(resolved_target) + examples_root = repository_root / "examples" / "machine" + examples = linked_json_examples(repository_root, examples_root) self.assertTrue(examples, "No examples/machine/*.json links found") for example in examples: @@ -112,6 +126,7 @@ def test_linked_machine_examples_use_current_schema(self): "group_size", ): self.assertIn(field, resources) + converted = convert_mdata(machine_data) for stage in ("train", "model_devi", "fp"): self.assertIn(f"{stage}_machine", converted) @@ -126,11 +141,18 @@ def test_linked_machine_examples_use_current_schema(self): self.assertTrue(converted_resources) self.assertIsInstance(converted_command, str) self.assertTrue(converted_command.strip()) - self.assertEqual(converted_machine, stage_values[stage]["machine"]) self.assertEqual( - converted_resources, stage_values[stage]["resources"] + converted_machine, + stage_values[stage]["machine"], + ) + self.assertEqual( + converted_resources, + stage_values[stage]["resources"], + ) + self.assertEqual( + converted_command, + stage_values[stage]["command"], ) - self.assertEqual(converted_command, stage_values[stage]["command"]) if __name__ == "__main__": From d3ef90eefa436f859af53202d98631796e19e6bb Mon Sep 17 00:00:00 2001 From: SchrodingersCattt Date: Sun, 30 Aug 2026 02:49:05 +0000 Subject: [PATCH 13/13] test(skills): harden dpgen-run validation --- skills/dpgen-run/references/machine-json.md | 6 +- .../references/validation-and-run.md | 1 + tests/test_dpgen_run_skill.py | 73 +++++++++++++++++++ 3 files changed, 78 insertions(+), 2 deletions(-) diff --git a/skills/dpgen-run/references/machine-json.md b/skills/dpgen-run/references/machine-json.md index 1ecb12a65..caf68b766 100644 --- a/skills/dpgen-run/references/machine-json.md +++ b/skills/dpgen-run/references/machine-json.md @@ -27,7 +27,9 @@ Keep separate `train`, `model_devi`, and `fp` blocks. For each stage, collect or - `resources.source_list` for scheduler runtime activation - queue, partition, account, and custom scheduler flags when applicable -Typical commands are `dp` for training, `lmp` for LAMMPS exploration, and a backend executable such as `vasp_std`, `cp2k.popt`, `abacus`, `pw.x`, or `g16` for labeling. Preserve known working commands. +`train_backend` does not rewrite the machine command. `dp` selects the DeePMD installation's default backend; a DeePMD-kit 3.x PyTorch installation may require `dp --pt`. Verify the installed entry point with `dp train -h` or `dp --pt train -h`, then make the `train` command agree with `train_backend`. + +Typical later-stage commands are `lmp` for LAMMPS exploration and a backend executable such as `vasp_std`, `cp2k.popt`, `abacus`, `pw.x`, or `g16` for labeling. Preserve known working commands. ## Context and batch compatibility @@ -51,6 +53,6 @@ Patch the closest existing working configuration. Do not transplant site-specifi - Keep CPU, GPU, node, and grouping requests explicit. - Require explicit `source_list` activation for scheduler stages. - Preserve working local omissions and installed-version aliases. -- Ensure commands match the scientific stack selected in `param.json`. +- Ensure the training command selects the same backend as `train_backend` and all commands match the scientific stack selected in `param.json`. External machine reference: https://docs.deepmodeling.com/projects/dpgen/en/latest/run/mdata.html diff --git a/skills/dpgen-run/references/validation-and-run.md b/skills/dpgen-run/references/validation-and-run.md index 8ae59c4ed..60acea824 100644 --- a/skills/dpgen-run/references/validation-and-run.md +++ b/skills/dpgen-run/references/validation-and-run.md @@ -52,6 +52,7 @@ Compare every `type_map.raw` line-for-line with `param.json.type_map`. Verify ev Confirm: - DeePMD-kit version, backend, descriptor, and training-input layout agree +- `train_backend` and the machine `train.command` select the same backend; for DeePMD-kit 3.x PyTorch, verify `dp --pt train -h` and use `dp --pt` when required - each exploration job references valid systems and MD settings - force trust thresholds are ordered and scientifically intentional - FP inputs and support files match `fp_style` diff --git a/tests/test_dpgen_run_skill.py b/tests/test_dpgen_run_skill.py index ab9787110..fc5d85bbf 100644 --- a/tests/test_dpgen_run_skill.py +++ b/tests/test_dpgen_run_skill.py @@ -1,8 +1,15 @@ +import copy import json import re import unittest from pathlib import Path +from dargs.dargs import ( + ArgumentKeyError, + ArgumentTypeError, + ArgumentValueError, +) + from dpgen.generator.arginfo import run_jdata_arginfo from dpgen.remote.decide_machine import convert_mdata from dpgen.util import normalize @@ -88,6 +95,72 @@ def test_linked_parameter_examples_use_current_schema(self): self.assertIsInstance(normalized, dict) self.assertTrue(normalized) + def test_documented_parameter_schema_invariants(self): + repository_root = Path(__file__).resolve().parents[1] + example = ( + repository_root + / "examples" + / "run" + / "dp2.x-lammps-cp2k" + / "param_CH4_deepmd-kit-2.0.1.json" + ) + parameter_data = json.loads(example.read_text()) + normalized = normalize( + run_jdata_arginfo(), + copy.deepcopy(parameter_data), + strict_check=False, + ) + + self.assertEqual(normalized["model_devi_engine"], "lammps") + self.assertEqual(normalized["train_backend"], "tensorflow") + + without_mass_map = copy.deepcopy(parameter_data) + without_mass_map.pop("mass_map", None) + normalized_without_mass_map = normalize( + run_jdata_arginfo(), + without_mass_map, + strict_check=False, + ) + self.assertEqual(normalized_without_mass_map["mass_map"], "auto") + + invalid_fp_style = copy.deepcopy(parameter_data) + invalid_fp_style["fp_style"] = "none" + with self.assertRaises(ArgumentValueError): + normalize( + run_jdata_arginfo(), + invalid_fp_style, + strict_check=False, + ) + + missing_model_devi_skip = copy.deepcopy(parameter_data) + missing_model_devi_skip.pop("model_devi_skip") + with self.assertRaises(ArgumentKeyError): + normalize( + run_jdata_arginfo(), + missing_model_devi_skip, + strict_check=False, + ) + + flat_sys_configs = copy.deepcopy(parameter_data) + flat_sys_configs["sys_configs"] = ["POSCAR"] + with self.assertRaises(ArgumentTypeError): + normalize( + run_jdata_arginfo(), + flat_sys_configs, + strict_check=False, + ) + + for backend in ("tensorflow", "pytorch"): + with self.subTest(backend=backend): + backend_data = copy.deepcopy(parameter_data) + backend_data["train_backend"] = backend + normalized_backend = normalize( + run_jdata_arginfo(), + backend_data, + strict_check=False, + ) + self.assertEqual(normalized_backend["train_backend"], backend) + def test_linked_machine_examples_use_current_schema(self): repository_root = Path(__file__).resolve().parents[1] examples_root = repository_root / "examples" / "machine"