From 34d1b4573d15aa83122b6ca9be12d89308eb9ac5 Mon Sep 17 00:00:00 2001 From: MatMaster Date: Thu, 11 Jun 2026 14:22:58 +0800 Subject: [PATCH 01/23] feat(skills): add dpgen-run skill for concurrent learning workflow --- skills/dpgen-run/SKILL.md | 405 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 405 insertions(+) create mode 100644 skills/dpgen-run/SKILL.md diff --git a/skills/dpgen-run/SKILL.md b/skills/dpgen-run/SKILL.md new file mode 100644 index 000000000..b5daa9a12 --- /dev/null +++ b/skills/dpgen-run/SKILL.md @@ -0,0 +1,405 @@ +--- +name: dpgen-run +description: Prepare, explain, validate, and run DP-GEN concurrent learning workflows for training deep potential models via iterative exploration. Use when the user wants to generate or modify `param.json` and `machine.json` for `dpgen run`, configure training/exploration/labeling iterations, select descriptor types, set trust levels, define model_devi_jobs, or inspect run outputs. +compatibility: Requires a runnable environment with Python and an activated DP-GEN runtime where `dpgen` is available in PATH for the outer run command. Real execution also requires DeePMD-kit, LAMMPS (with DeePMD plugin for model_devi), and any backend-specific software required by the selected `fp_style`. For scheduler execution, each stage environment must be explicitly activated in `resources.source_list`. +license: LGPL-3.0-or-later +metadata: + author: MatMaster + version: 0.1.0 + repository: https://github.com/deepmodeling/dpgen +--- + +# DP-GEN Run (Concurrent Learning) + +Use this skill when the user wants to prepare, explain, validate, or execute the `dpgen run` concurrent learning workflow. + +This skill is for the main DP-GEN iterative loop: train an ensemble of deep potential models, explore configuration space via LAMMPS MD, select uncertain structures, label them with first-principles calculations, and feed new data back into training. + +## Core Rule (Critical) + +DP-GEN run always uses **two parameter classes** and therefore **two JSON files**: + +- **Workflow parameters** -> `param.json` +- **Execution / machine parameters** -> `machine.json` + +Run exactly: + +```bash +dpgen run param.json machine.json +``` + +Environment boundary rule: + +- Outer layer: run `dpgen run param.json machine.json` in an activated environment where `dpgen --version` works. +- Inner layer: for scheduler stages, explicitly activate runtime in `resources.source_list` on the server side. + +## Critical Pitfalls (Must Embed) + +These are verified failure modes discovered through testing. Treat as hard rules: + +1. **`se_atten_v2` spelling** — The descriptor type is `se_atten_v2` (double-t in "atten"). Writing `se_attn_v2` will silently fail or error. Always verify the exact string. + +2. **`remote_root` is mandatory** — dpdispatcher requires `remote_root` even for local Shell execution. Set it to a writable path like `/tmp/dpgen_train`. Omitting it causes runtime errors. + +3. **`batch_type: "Shell"` is case-sensitive** — Must be capitalized `"Shell"`, not `"shell"` or `"SHELL"`. Same for `"Slurm"`, `"PBS"`, etc. + +4. **CP2K `user_fp_params` KIND format** — When multiple element kinds are needed, use `"_": ["H", "O"]` with parallel arrays for `POTENTIAL` and `BASIS_SET`. This maps to repeated KIND sections. + +5. **`type_map.raw` ordering** — The `type_map.raw` in your `init_data_sys` directories must exactly match the `type_map` array in `param.json`. Mismatches cause silent data corruption. + +6. **DeePMD-kit 3.x format** — `default_training_param` uses nested structure: `model.descriptor`, `model.fitting_net`, `learning_rate`, `loss`, `training`. The `training` block includes `training_data.systems` (left empty — dpgen fills it). + +7. **`model_devi_engine` defaults to LAMMPS** — No need to set `model_devi_engine` explicitly when using LAMMPS. Only set it for alternative engines. + +8. **`fp_style: "cp2k"` is native** — dpgen v0.12+ supports CP2K natively via `user_fp_params` dict. No external plugins required. + +9. **`sys_configs` format** — List of lists: outer list = systems, inner list = file paths (POSCAR/xyz) for that system. Example: `[["path/to/POSCAR_1"], ["path/to/POSCAR_2"]]`. + +10. **`init_data_sys` format** — List of paths to directories in deepmd/npy format. Each directory must contain `type_map.raw`, `type.raw`, and `set.000/` with numpy arrays. + +## Agent responsibilities + +When using this skill, the agent should: + +1. confirm that the task is a `dpgen run` concurrent learning workflow +1. check whether existing configs or templates are already available +1. collect only the missing training, exploration, FP, and machine inputs +1. select appropriate descriptor type based on system complexity +1. generate or patch `param.json` +1. generate or patch `machine.json` +1. explain important parameters in plain language when asked +1. validate the workflow before execution +1. provide the exact command for running +1. after execution, summarize outputs and next inspection targets + +## Working policy + +### 1. Ask only for missing inputs + +Do not ask the user for everything if part of the configuration is already available. + +If the user already provides: + +- a partial `param.json` +- a partial `machine.json` +- a known training template +- a known cluster template + +then patch those files instead of rebuilding everything from scratch. + +### 2. Preserve the user's scientific choices + +Do not silently change: + +- descriptor family or type +- fitting net structure +- fp backend +- trust thresholds +- `type_map` ordering +- `model_devi_jobs` schedule +- ensemble or temperature settings + +If a value looks scientifically questionable, explain the concern instead of silently replacing it. + +### 3. Descriptor selection guidance + +dpgen (v0.13.x) only supports the **TensorFlow backend** of DeePMD-kit. It does NOT support PyTorch-only models (such as DPA-2, DPA-3). Do not recommend or generate configs for PyTorch-only descriptors. + +When the user has not specified a descriptor type, recommend based on: + +| Descriptor | Use case | Notes | +|---|---|---| +| `se_e2_a` | Simple systems, fast training, well-tested | Classic two-body embedding, requires explicit `sel` list | +| `se_atten` | Multi-component systems, moderate complexity | Attention-based, supports `sel: "auto"` | +| `se_atten_v2` | Modern default, best accuracy/cost balance | **Spell exactly `se_atten_v2`** (double-t). Supports `sel: "auto"`, `attn_layer: 0` | + +For new projects, prefer `se_atten_v2` unless the user has specific needs. If the user requires DPA-2/DPA-3 or PyTorch models, redirect them to **dpgen2** which supports the PyTorch backend. + +### 4. Keep local and scheduler execution explicit + +If the user wants local execution, produce local-friendly commands. + +If the user wants scheduler execution, produce scheduler-friendly commands and keep queue, partition, and resource requests explicit. + +Do not invent scheduler module names or executable paths. + +### 5. Do not invent environment activation commands + +If the user already has a working activation command such as: + +- `conda activate ...` +- `module load ...` +- `source ...` + +reuse it exactly. + +If execution is requested and the activation method is unknown, ask the user for the precise activation command. + +Do not guess conda environment names, module names, or site-specific paths. + +### 5.1 Outer launcher policy + +Use an activated DP-GEN environment and verify with: + +```bash +dpgen --version +``` + +Do not start run from a shell where `dpgen` is unavailable. + +### 5.2 Outer vs inner runtime boundaries (critical) + +Treat run execution as two separate environment layers: + +1. Outer layer: the shell that launches `dpgen run param.json machine.json` (must have `dpgen` in PATH) +1. Inner layer: stage tasks dispatched by DP-GEN (`train` / `model_devi` / `fp`) on server/runtime side + +Even if the outer layer is correct, inner stage tasks still need explicit runtime setup in `machine.json`. +Do not assume the outer shell environment will be inherited by dispatched stage jobs. +For scheduler-style execution, `resources.source_list` must explicitly activate the required runtime environment. + +### 6. Prefer reproducible output layout + +When generating a run workflow, keep files organized and predictable. + +Recommended structure: + +```text +project/ +├── param.json +├── machine.json +├── init_data/ +│ └── system_000/ +│ ├── type_map.raw +│ ├── type.raw +│ └── set.000/ +├── assets/ +│ └── structures/ +│ └── POSCAR_* +├── cp2k_basis_pp_file/ (if using CP2K) +│ ├── BASIS_MOLOPT +│ └── GTH_POTENTIALS +└── iter.*/ (created by dpgen) +``` + +## Minimum required inputs + +Collect the following information before generating files. + +### System information + +- `type_map` — ordered element symbols +- `mass_map` — atomic masses matching `type_map` order +- `init_data_prefix` — prefix for init_data_sys paths +- `init_data_sys` — list of paths to initial training data (deepmd/npy format) +- `sys_configs_prefix` — prefix for sys_configs paths +- `sys_configs` — list of lists of structure file paths + +### Training setup + +- `numb_models` — number of ensemble models (default: 4) +- `default_training_param` — full DeePMD training input: + - `model.descriptor` — descriptor type and settings + - `model.fitting_net` — fitting network settings + - `learning_rate` — learning rate schedule + - `loss` — loss function weights + - `training` — training steps and output settings + +### Exploration setup (model_devi) + +- `model_devi_dt` — MD timestep in ps (e.g. 0.0005 = 0.5 fs) +- `model_devi_skip` — number of initial frames to skip +- `model_devi_f_trust_lo` — lower force deviation threshold (eV/Å) +- `model_devi_f_trust_hi` — upper force deviation threshold (eV/Å) +- `model_devi_clean_traj` — whether to clean trajectory files after selection +- `model_devi_jobs` — list of exploration job specifications: + - `sys_idx` — which systems to explore + - `temps` — temperatures (K) + - `press` — pressures (bar) + - `trj_freq` — trajectory save frequency + - `nsteps` — number of MD steps + - `ensemble` — `"nvt"` or `"npt"` + +### FP setup + +- `fp_style` — backend: `"vasp"`, `"cp2k"`, `"abacus"`, `"gaussian"`, `"pwscf"`, or `"none"` +- `fp_task_max` — maximum number of FP tasks per iteration +- `fp_task_min` — minimum number to trigger FP +- Backend-specific settings: + - VASP: `fp_pp_path`, `fp_pp_files`, `fp_incar` or `fp_params` + - CP2K: `user_fp_params` (nested dict representing cp2k input) + - ABACUS: `user_fp_params`, `fp_pp_path`, `fp_pp_files`, `fp_orb_files` + - Gaussian: `fp_params` (keywords, nproc, multiplicity) + - PWSCF: `user_fp_params` + +### Execution setup + +For each stage `train`, `model_devi`, and `fp`, collect or preserve: + +- `command` +- `machine.batch_type` +- `machine.context_type` +- `machine.local_root` +- `machine.remote_root` **(mandatory — even for local Shell)** +- `resources.number_node` +- `resources.cpu_per_node` +- `resources.gpu_per_node` +- `resources.group_size` +- `resources.source_list` (required for scheduler jobs; use it to activate environment explicitly) +- any explicit queue / partition / custom scheduler flags if the user already uses them + +Choose a runtime profile first, then fill the matching template: + +- server-local Slurm: `assets/machine.template.server-local-slurm.json` +- pure local shell testing: `assets/machine.template.local-shell.json` + +## How to build `param.json` + +Construct `param.json` around these logical blocks: + +1. element and mass definitions (`type_map`, `mass_map`) +1. data source configuration (`init_data_prefix`, `init_data_sys`, `sys_configs_prefix`, `sys_configs`) +1. model ensemble count (`numb_models`) +1. default DeePMD training parameters (`default_training_param`) +1. exploration settings (`model_devi_dt`, `model_devi_skip`, `model_devi_f_trust_lo/hi`, `model_devi_clean_traj`) +1. exploration job schedule (`model_devi_jobs`) +1. FP backend settings (`fp_style`, `fp_task_max`, `fp_task_min`, backend-specific params) + +Key fields always required: + +- `type_map` +- `mass_map` +- `init_data_sys` +- `sys_configs` +- `numb_models` +- `default_training_param` +- `model_devi_dt` +- `model_devi_f_trust_lo` +- `model_devi_f_trust_hi` +- `model_devi_jobs` +- `fp_style` +- `fp_task_max` +- `fp_task_min` + +Trust level guidance: + +- `model_devi_f_trust_lo`: structures below this are "accurate" — not selected +- `model_devi_f_trust_hi`: structures above this are "failed" — discarded +- Between lo and hi: "candidate" — selected for FP labeling +- Typical starting values: lo=0.05–0.10, hi=0.15–0.30 (eV/Å) +- Tighter thresholds = more FP cost, better accuracy +- System-specific tuning is recommended after initial iterations + +Official reference examples: + +- `assets/param.example.water-cp2k.json` +- `assets/param.example.water-vasp.json` + +## How to build `machine.json` + +Construct `machine.json` with separate stage blocks for: + +- `train` +- `model_devi` +- `fp` + +For each stage, keep the following explicit: + +- `command` — the executable (`dp` for train, `lmp` for model_devi, FP command for fp) +- machine or context configuration +- resources +- queue or partition if needed +- cpu and gpu counts +- custom scheduler flags +- environment activation commands + +Do not merge all stages into one vague machine block. + +Stage-specific commands: + +- `train`: `"dp"` (DeePMD-kit training) +- `model_devi`: `"lmp"` (LAMMPS with DeePMD plugin) +- `fp`: backend-specific (`"vasp_std"`, `"cp2k.popt"`, `"abacus"`, `"pw.x"`, `"g16"`) + +## Validation before run + +Before execution, validate the workflow in this order: + +1. confirm outer-layer `dpgen` is available: + +```bash +dpgen --version +``` + +2. validate JSON syntax: + +```bash +python -m json.tool param.json +python -m json.tool machine.json +``` + +3. verify `init_data_sys` directories exist and contain proper deepmd/npy format: + +```bash +# Each directory must have type_map.raw, type.raw, set.000/ +ls init_data/*/type_map.raw +``` + +4. verify `type_map.raw` content matches `param.json` `type_map` ordering +5. verify `sys_configs` structure files exist +6. verify stage commands match the selected software stack +7. for CP2K: verify basis set and potential files are accessible +8. only then run: + +```bash +dpgen run param.json machine.json +``` + +## Output contract + +Always provide: + +1. final absolute paths to `param.json` and `machine.json` +1. the exact run command (`dpgen run param.json machine.json`) +1. a short pre-run checklist +1. any unresolved required fields +1. if execution was performed, the main output locations and next files to inspect + +## Guardrails + +- Never merge workflow and machine parameters into one file. +- Never run `dpgen run` before both JSON files are present. +- Never hardcode personal cluster, account, queue, or path settings as universal defaults. +- Never silently change the user's scientific choices. +- Keep `type_map` ordering consistent with `init_data_sys` type_map.raw files. +- If required inputs are missing, stop and ask instead of guessing. +- Always spell `se_atten_v2` correctly (not `se_attn_v2`). +- Always include `remote_root` in machine config — it is mandatory even for local execution. +- Always capitalize `batch_type` values: `"Shell"`, `"Slurm"`, `"PBS"`. +- For CP2K KIND sections with multiple elements, use `"_": ["elem1", "elem2"]` array format. +- Do not assume outer-shell activation is inherited by stage jobs; for scheduler execution, require explicit `source_list` per stage. +- If the user already has working templates, patch them rather than overwriting them blindly. +- Do not set `model_devi_engine` unless using a non-LAMMPS engine (it defaults to LAMMPS). +- Leave `training.training_data.systems` as an empty list `[]` in `default_training_param` — dpgen fills this automatically. + +## References and bundled files + +Use these bundled files: + +- `assets/param.template.json` +- `assets/param.example.water-cp2k.json` +- `assets/param.example.water-vasp.json` +- `assets/machine.template.json` +- `assets/machine.template.server-local-slurm.json` +- `assets/machine.template.local-shell.json` +- `references/param-fields.md` +- `references/machine-fields.md` +- `references/workflow-notes.md` +- `references/descriptor-types.md` + +External references: + +- DP-GEN run overview: https://docs.deepmodeling.com/projects/dpgen/en/latest/run/index.html +- run parameter definitions: https://docs.deepmodeling.com/projects/dpgen/en/latest/run/param.html +- run machine definitions: https://docs.deepmodeling.com/projects/dpgen/en/latest/run/mdata.html +- DeePMD-kit model documentation: https://docs.deepmodeling.com/projects/deepmd/en/latest/model/index.html From 841acdaf8744faa96ef7d28c99deb9455a7ecc77 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Thu, 11 Jun 2026 06:51:07 +0000 Subject: [PATCH 02/23] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- skills/dpgen-run/SKILL.md | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/skills/dpgen-run/SKILL.md b/skills/dpgen-run/SKILL.md index b5daa9a12..ba4a66a17 100644 --- a/skills/dpgen-run/SKILL.md +++ b/skills/dpgen-run/SKILL.md @@ -39,23 +39,23 @@ These are verified failure modes discovered through testing. Treat as hard rules 1. **`se_atten_v2` spelling** — The descriptor type is `se_atten_v2` (double-t in "atten"). Writing `se_attn_v2` will silently fail or error. Always verify the exact string. -2. **`remote_root` is mandatory** — dpdispatcher requires `remote_root` even for local Shell execution. Set it to a writable path like `/tmp/dpgen_train`. Omitting it causes runtime errors. +1. **`remote_root` is mandatory** — dpdispatcher requires `remote_root` even for local Shell execution. Set it to a writable path like `/tmp/dpgen_train`. Omitting it causes runtime errors. -3. **`batch_type: "Shell"` is case-sensitive** — Must be capitalized `"Shell"`, not `"shell"` or `"SHELL"`. Same for `"Slurm"`, `"PBS"`, etc. +1. **`batch_type: "Shell"` is case-sensitive** — Must be capitalized `"Shell"`, not `"shell"` or `"SHELL"`. Same for `"Slurm"`, `"PBS"`, etc. -4. **CP2K `user_fp_params` KIND format** — When multiple element kinds are needed, use `"_": ["H", "O"]` with parallel arrays for `POTENTIAL` and `BASIS_SET`. This maps to repeated KIND sections. +1. **CP2K `user_fp_params` KIND format** — When multiple element kinds are needed, use `"_": ["H", "O"]` with parallel arrays for `POTENTIAL` and `BASIS_SET`. This maps to repeated KIND sections. -5. **`type_map.raw` ordering** — The `type_map.raw` in your `init_data_sys` directories must exactly match the `type_map` array in `param.json`. Mismatches cause silent data corruption. +1. **`type_map.raw` ordering** — The `type_map.raw` in your `init_data_sys` directories must exactly match the `type_map` array in `param.json`. Mismatches cause silent data corruption. -6. **DeePMD-kit 3.x format** — `default_training_param` uses nested structure: `model.descriptor`, `model.fitting_net`, `learning_rate`, `loss`, `training`. The `training` block includes `training_data.systems` (left empty — dpgen fills it). +1. **DeePMD-kit 3.x format** — `default_training_param` uses nested structure: `model.descriptor`, `model.fitting_net`, `learning_rate`, `loss`, `training`. The `training` block includes `training_data.systems` (left empty — dpgen fills it). -7. **`model_devi_engine` defaults to LAMMPS** — No need to set `model_devi_engine` explicitly when using LAMMPS. Only set it for alternative engines. +1. **`model_devi_engine` defaults to LAMMPS** — No need to set `model_devi_engine` explicitly when using LAMMPS. Only set it for alternative engines. -8. **`fp_style: "cp2k"` is native** — dpgen v0.12+ supports CP2K natively via `user_fp_params` dict. No external plugins required. +1. **`fp_style: "cp2k"` is native** — dpgen v0.12+ supports CP2K natively via `user_fp_params` dict. No external plugins required. -9. **`sys_configs` format** — List of lists: outer list = systems, inner list = file paths (POSCAR/xyz) for that system. Example: `[["path/to/POSCAR_1"], ["path/to/POSCAR_2"]]`. +1. **`sys_configs` format** — List of lists: outer list = systems, inner list = file paths (POSCAR/xyz) for that system. Example: `[["path/to/POSCAR_1"], ["path/to/POSCAR_2"]]`. -10. **`init_data_sys` format** — List of paths to directories in deepmd/npy format. Each directory must contain `type_map.raw`, `type.raw`, and `set.000/` with numpy arrays. +1. **`init_data_sys` format** — List of paths to directories in deepmd/npy format. Each directory must contain `type_map.raw`, `type.raw`, and `set.000/` with numpy arrays. ## Agent responsibilities @@ -107,11 +107,11 @@ dpgen (v0.13.x) only supports the **TensorFlow backend** of DeePMD-kit. It does When the user has not specified a descriptor type, recommend based on: -| Descriptor | Use case | Notes | -|---|---|---| -| `se_e2_a` | Simple systems, fast training, well-tested | Classic two-body embedding, requires explicit `sel` list | -| `se_atten` | Multi-component systems, moderate complexity | Attention-based, supports `sel: "auto"` | -| `se_atten_v2` | Modern default, best accuracy/cost balance | **Spell exactly `se_atten_v2`** (double-t). Supports `sel: "auto"`, `attn_layer: 0` | +| Descriptor | Use case | Notes | +| ------------- | -------------------------------------------- | ----------------------------------------------------------------------------------- | +| `se_e2_a` | Simple systems, fast training, well-tested | Classic two-body embedding, requires explicit `sel` list | +| `se_atten` | Multi-component systems, moderate complexity | Attention-based, supports `sel: "auto"` | +| `se_atten_v2` | Modern default, best accuracy/cost balance | **Spell exactly `se_atten_v2`** (double-t). Supports `sel: "auto"`, `attn_layer: 0` | For new projects, prefer `se_atten_v2` unless the user has specific needs. If the user requires DPA-2/DPA-3 or PyTorch models, redirect them to **dpgen2** which supports the PyTorch backend. @@ -346,10 +346,10 @@ ls init_data/*/type_map.raw ``` 4. verify `type_map.raw` content matches `param.json` `type_map` ordering -5. verify `sys_configs` structure files exist -6. verify stage commands match the selected software stack -7. for CP2K: verify basis set and potential files are accessible -8. only then run: +1. verify `sys_configs` structure files exist +1. verify stage commands match the selected software stack +1. for CP2K: verify basis set and potential files are accessible +1. only then run: ```bash dpgen run param.json machine.json From cd01b79ee05f015f6479a6226aded8aafc35206a Mon Sep 17 00:00:00 2001 From: SchrodingersCattt Date: Sun, 14 Jun 2026 07:04:12 +0000 Subject: [PATCH 03/23] docs: address dpgen-run skill review --- skills/dpgen-run/SKILL.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/skills/dpgen-run/SKILL.md b/skills/dpgen-run/SKILL.md index ba4a66a17..a964424c5 100644 --- a/skills/dpgen-run/SKILL.md +++ b/skills/dpgen-run/SKILL.md @@ -103,7 +103,7 @@ If a value looks scientifically questionable, explain the concern instead of sil ### 3. Descriptor selection guidance -dpgen (v0.13.x) only supports the **TensorFlow backend** of DeePMD-kit. It does NOT support PyTorch-only models (such as DPA-2, DPA-3). Do not recommend or generate configs for PyTorch-only descriptors. +dpgen includes `train_backend` options for both `"tensorflow"` and `"pytorch"`, so backend support may include PyTorch depending on the installed DeePMD-kit stack. If backend or descriptor compatibility is unclear, verify it against the repo's supported `train_backend` values (`"tensorflow"`, `"pytorch"`) and the user's installed software before generating configs. When the user has not specified a descriptor type, recommend based on: @@ -113,7 +113,7 @@ When the user has not specified a descriptor type, recommend based on: | `se_atten` | Multi-component systems, moderate complexity | Attention-based, supports `sel: "auto"` | | `se_atten_v2` | Modern default, best accuracy/cost balance | **Spell exactly `se_atten_v2`** (double-t). Supports `sel: "auto"`, `attn_layer: 0` | -For new projects, prefer `se_atten_v2` unless the user has specific needs. If the user requires DPA-2/DPA-3 or PyTorch models, redirect them to **dpgen2** which supports the PyTorch backend. +For new projects, prefer `se_atten_v2` unless the user has specific needs. If the user requires PyTorch-only models or DPA-2/DPA-3, prefer **dpgen2** unless the current dpgen and DeePMD-kit installation are verified to support the requested model. ### 4. Keep local and scheduler execution explicit @@ -346,10 +346,10 @@ ls init_data/*/type_map.raw ``` 4. verify `type_map.raw` content matches `param.json` `type_map` ordering -1. verify `sys_configs` structure files exist -1. verify stage commands match the selected software stack -1. for CP2K: verify basis set and potential files are accessible -1. only then run: +5. verify `sys_configs` structure files exist +6. verify stage commands match the selected software stack +7. for CP2K: verify basis set and potential files are accessible +8. only then run: ```bash dpgen run param.json machine.json @@ -380,7 +380,7 @@ Always provide: - Do not assume outer-shell activation is inherited by stage jobs; for scheduler execution, require explicit `source_list` per stage. - If the user already has working templates, patch them rather than overwriting them blindly. - Do not set `model_devi_engine` unless using a non-LAMMPS engine (it defaults to LAMMPS). -- Leave `training.training_data.systems` as an empty list `[]` in `default_training_param` — dpgen fills this automatically. +- In `default_training_param`, leave `training.training_data.systems` unset or empty — `dpgen/generator/run.py` fills it from `init_data_sys` automatically. ## References and bundled files From 39adc8fc5a980dd9e8b7424fff14bc0cd8a684e1 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Sun, 14 Jun 2026 07:05:02 +0000 Subject: [PATCH 04/23] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- skills/dpgen-run/SKILL.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/skills/dpgen-run/SKILL.md b/skills/dpgen-run/SKILL.md index a964424c5..2aa70ee81 100644 --- a/skills/dpgen-run/SKILL.md +++ b/skills/dpgen-run/SKILL.md @@ -346,10 +346,10 @@ ls init_data/*/type_map.raw ``` 4. verify `type_map.raw` content matches `param.json` `type_map` ordering -5. verify `sys_configs` structure files exist -6. verify stage commands match the selected software stack -7. for CP2K: verify basis set and potential files are accessible -8. only then run: +1. verify `sys_configs` structure files exist +1. verify stage commands match the selected software stack +1. for CP2K: verify basis set and potential files are accessible +1. only then run: ```bash dpgen run param.json machine.json From 8397b836323d941b91702f4f6aa40f75df12a4ce Mon Sep 17 00:00:00 2001 From: SchrodingersCattt Date: Sun, 14 Jun 2026 07:09:10 +0000 Subject: [PATCH 05/23] docs: clarify dpgen backend compatibility --- skills/dpgen-run/SKILL.md | 48 +++++++++++++++++++-------------------- 1 file changed, 24 insertions(+), 24 deletions(-) diff --git a/skills/dpgen-run/SKILL.md b/skills/dpgen-run/SKILL.md index 2aa70ee81..779b56515 100644 --- a/skills/dpgen-run/SKILL.md +++ b/skills/dpgen-run/SKILL.md @@ -47,7 +47,7 @@ These are verified failure modes discovered through testing. Treat as hard rules 1. **`type_map.raw` ordering** — The `type_map.raw` in your `init_data_sys` directories must exactly match the `type_map` array in `param.json`. Mismatches cause silent data corruption. -1. **DeePMD-kit 3.x format** — `default_training_param` uses nested structure: `model.descriptor`, `model.fitting_net`, `learning_rate`, `loss`, `training`. The `training` block includes `training_data.systems` (left empty — dpgen fills it). +1. **DeePMD-kit input format** — `default_training_param` uses nested structure: `model.descriptor`, `model.fitting_net`, `learning_rate`, `loss`, `training`. The `training` block may leave `training_data.systems` unset or empty — dpgen fills it. Check the configured `deepmd_version` and installed DeePMD-kit before using 3.x-specific features. 1. **`model_devi_engine` defaults to LAMMPS** — No need to set `model_devi_engine` explicitly when using LAMMPS. Only set it for alternative engines. @@ -103,7 +103,7 @@ If a value looks scientifically questionable, explain the concern instead of sil ### 3. Descriptor selection guidance -dpgen includes `train_backend` options for both `"tensorflow"` and `"pytorch"`, so backend support may include PyTorch depending on the installed DeePMD-kit stack. If backend or descriptor compatibility is unclear, verify it against the repo's supported `train_backend` values (`"tensorflow"`, `"pytorch"`) and the user's installed software before generating configs. +dpgen exposes `train_backend` options for `"tensorflow"` and `"pytorch"`, but that option alone does not guarantee every PyTorch descriptor or every DeePMD-kit 3.x feature works in the user's installed stack. If backend or descriptor compatibility is unclear, verify it against the repo's `train_backend` options, the configured `deepmd_version`, and the installed DeePMD-kit version before generating configs. When the user has not specified a descriptor type, recommend based on: @@ -325,35 +325,35 @@ Stage-specific commands: Before execution, validate the workflow in this order: -1. confirm outer-layer `dpgen` is available: +- Step 1: confirm outer-layer `dpgen` is available: -```bash -dpgen --version -``` + ```bash + dpgen --version + ``` -2. validate JSON syntax: +- Step 2: validate JSON syntax: -```bash -python -m json.tool param.json -python -m json.tool machine.json -``` + ```bash + python -m json.tool param.json + python -m json.tool machine.json + ``` -3. verify `init_data_sys` directories exist and contain proper deepmd/npy format: +- Step 3: verify `init_data_sys` directories exist and contain proper deepmd/npy format: -```bash -# Each directory must have type_map.raw, type.raw, set.000/ -ls init_data/*/type_map.raw -``` + ```bash + # Each directory must have type_map.raw, type.raw, set.000/ + ls init_data/*/type_map.raw + ``` -4. verify `type_map.raw` content matches `param.json` `type_map` ordering -1. verify `sys_configs` structure files exist -1. verify stage commands match the selected software stack -1. for CP2K: verify basis set and potential files are accessible -1. only then run: +- Step 4: verify `type_map.raw` content matches `param.json` `type_map` ordering. +- Step 5: verify `sys_configs` structure files exist. +- Step 6: verify stage commands match the selected software stack. +- Step 7: for CP2K, verify basis set and potential files are accessible. +- Step 8: only then run: -```bash -dpgen run param.json machine.json -``` + ```bash + dpgen run param.json machine.json + ``` ## Output contract From f1ee9580323d186470ec8f720274d8bba78b0a2d Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Sun, 14 Jun 2026 07:11:55 +0000 Subject: [PATCH 06/23] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- skills/dpgen-run/SKILL.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/skills/dpgen-run/SKILL.md b/skills/dpgen-run/SKILL.md index 779b56515..6089f68d3 100644 --- a/skills/dpgen-run/SKILL.md +++ b/skills/dpgen-run/SKILL.md @@ -346,9 +346,13 @@ Before execution, validate the workflow in this order: ``` - Step 4: verify `type_map.raw` content matches `param.json` `type_map` ordering. + - Step 5: verify `sys_configs` structure files exist. + - Step 6: verify stage commands match the selected software stack. + - Step 7: for CP2K, verify basis set and potential files are accessible. + - Step 8: only then run: ```bash From ae2c3457b458a81534c9da4d419760a8c1776b19 Mon Sep 17 00:00:00 2001 From: SchrodingersCattt Date: Wed, 12 Aug 2026 04:43:56 +0000 Subject: [PATCH 07/23] docs(skills): complete dpgen-run guardrails --- skills/dpgen-run/SKILL.md | 56 ++++++++++++++++++++++------------- tests/test_dpgen_run_skill.py | 29 ++++++++++++++++++ 2 files changed, 64 insertions(+), 21 deletions(-) create mode 100644 tests/test_dpgen_run_skill.py diff --git a/skills/dpgen-run/SKILL.md b/skills/dpgen-run/SKILL.md index 6089f68d3..41759fd60 100644 --- a/skills/dpgen-run/SKILL.md +++ b/skills/dpgen-run/SKILL.md @@ -39,9 +39,14 @@ These are verified failure modes discovered through testing. Treat as hard rules 1. **`se_atten_v2` spelling** — The descriptor type is `se_atten_v2` (double-t in "atten"). Writing `se_attn_v2` will silently fail or error. Always verify the exact string. -1. **`remote_root` is mandatory** — dpdispatcher requires `remote_root` even for local Shell execution. Set it to a writable path like `/tmp/dpgen_train`. Omitting it causes runtime errors. +1. **`remote_root` is context-dependent** — require it for remote or scheduler + contexts whose installed dpdispatcher schema needs a remote working root. + Lazy/local contexts can omit it. Preserve an existing working omission and + validate against the installed dpdispatcher version. -1. **`batch_type: "Shell"` is case-sensitive** — Must be capitalized `"Shell"`, not `"shell"` or `"SHELL"`. Same for `"Slurm"`, `"PBS"`, etc. +1. **`batch_type` must be registered** — canonical names and lowercase aliases + can both be valid. Preserve an existing working value and validate new values + against the installed dpdispatcher version. 1. **CP2K `user_fp_params` KIND format** — When multiple element kinds are needed, use `"_": ["H", "O"]` with parallel arrays for `POTENTIAL` and `BASIS_SET`. This maps to repeated KIND sections. @@ -74,6 +79,13 @@ When using this skill, the agent should: ## Working policy +### 0. Treat execution as separately authorized + +Prepare, explain, and validate by default. Execute `dpgen run` only when the +user has explicitly requested execution and, after validation, confirmed the +exact command. Supplying configuration files, passing validation, or showing the +command does not by itself authorize an HPC or first-principles workload. + ### 1. Ask only for missing inputs Do not ask the user for everything if part of the configuration is already available. @@ -240,7 +252,8 @@ For each stage `train`, `model_devi`, and `fp`, collect or preserve: - `machine.batch_type` - `machine.context_type` - `machine.local_root` -- `machine.remote_root` **(mandatory — even for local Shell)** +- `machine.remote_root` when required by the selected context; preserve a + working omission for lazy/local execution - `resources.number_node` - `resources.cpu_per_node` - `resources.gpu_per_node` @@ -250,8 +263,10 @@ For each stage `train`, `model_devi`, and `fp`, collect or preserve: Choose a runtime profile first, then fill the matching template: -- server-local Slurm: `assets/machine.template.server-local-slurm.json` -- pure local shell testing: `assets/machine.template.local-shell.json` +- server-local scheduler: + [existing scheduler example](../../examples/machine/DeePMD-kit-1.x/machine-lsf-slurm-cp2k.json) +- pure local shell testing: + [existing local example](../../examples/run/ch4/machine.json) ## How to build `param.json` @@ -292,8 +307,10 @@ Trust level guidance: Official reference examples: -- `assets/param.example.water-cp2k.json` -- `assets/param.example.water-vasp.json` +- CP2K: + [methane example](../../examples/run/dp2.x-lammps-cp2k/param_CH4_deepmd-kit-2.0.1.json) +- VASP: + [CH4 example](../../examples/run/dp2.x-lammps-vasp/CH4/param_CH4_deepmd-kit-2.x.json) ## How to build `machine.json` @@ -378,28 +395,25 @@ Always provide: - Keep `type_map` ordering consistent with `init_data_sys` type_map.raw files. - If required inputs are missing, stop and ask instead of guessing. - Always spell `se_atten_v2` correctly (not `se_attn_v2`). -- Always include `remote_root` in machine config — it is mandatory even for local execution. -- Always capitalize `batch_type` values: `"Shell"`, `"Slurm"`, `"PBS"`. +- Include `remote_root` when the selected context requires it; do not add it + solely to rewrite a working lazy/local configuration. +- Use a `batch_type` accepted by the installed dpdispatcher version and + preserve an existing working canonical name or alias. - For CP2K KIND sections with multiple elements, use `"_": ["elem1", "elem2"]` array format. - Do not assume outer-shell activation is inherited by stage jobs; for scheduler execution, require explicit `source_list` per stage. - If the user already has working templates, patch them rather than overwriting them blindly. - Do not set `model_devi_engine` unless using a non-LAMMPS engine (it defaults to LAMMPS). - In `default_training_param`, leave `training.training_data.systems` unset or empty — `dpgen/generator/run.py` fills it from `init_data_sys` automatically. -## References and bundled files +## Repository examples and references -Use these bundled files: +Use these checked-in repository examples as starting points: -- `assets/param.template.json` -- `assets/param.example.water-cp2k.json` -- `assets/param.example.water-vasp.json` -- `assets/machine.template.json` -- `assets/machine.template.server-local-slurm.json` -- `assets/machine.template.local-shell.json` -- `references/param-fields.md` -- `references/machine-fields.md` -- `references/workflow-notes.md` -- `references/descriptor-types.md` +- [local CH4 parameter example](../../examples/run/ch4/param.json) +- [local CH4 machine example](../../examples/run/ch4/machine.json) +- [CP2K parameter example](../../examples/run/dp2.x-lammps-cp2k/param_CH4_deepmd-kit-2.0.1.json) +- [VASP parameter example](../../examples/run/dp2.x-lammps-vasp/CH4/param_CH4_deepmd-kit-2.x.json) +- [scheduler machine example](../../examples/machine/DeePMD-kit-1.x/machine-lsf-slurm-cp2k.json) External references: diff --git a/tests/test_dpgen_run_skill.py b/tests/test_dpgen_run_skill.py new file mode 100644 index 000000000..cc783c953 --- /dev/null +++ b/tests/test_dpgen_run_skill.py @@ -0,0 +1,29 @@ +import re +import unittest +from pathlib import Path + + +class TestDPGenRunSkill(unittest.TestCase): + def test_all_repository_relative_links_exist(self): + repository_root = Path(__file__).resolve().parents[1] + skill_path = repository_root / "skills" / "dpgen-run" / "SKILL.md" + link_pattern = re.compile(r"\[[^]]+\]\(([^)]+)\)") + + missing = [] + for target in link_pattern.findall(skill_path.read_text()): + if "://" in target or target.startswith("#"): + continue + target_path = target.split("#", 1)[0] + if not (skill_path.parent / target_path).resolve().exists(): + missing.append(target) + + self.assertEqual( + missing, + [], + "Missing repository-relative links in dpgen-run skill: " + + ", ".join(missing), + ) + + +if __name__ == "__main__": + unittest.main() From 730249a672e2ad89350bca94b1b0fcbad58eb039 Mon Sep 17 00:00:00 2001 From: SchrodingersCattt Date: Wed, 12 Aug 2026 05:24:40 +0000 Subject: [PATCH 08/23] docs(skills): validate current machine examples --- skills/dpgen-run/SKILL.md | 4 ++-- tests/test_dpgen_run_skill.py | 38 +++++++++++++++++++++++++++++++++-- 2 files changed, 38 insertions(+), 4 deletions(-) diff --git a/skills/dpgen-run/SKILL.md b/skills/dpgen-run/SKILL.md index 41759fd60..3375aa5c3 100644 --- a/skills/dpgen-run/SKILL.md +++ b/skills/dpgen-run/SKILL.md @@ -266,7 +266,7 @@ Choose a runtime profile first, then fill the matching template: - server-local scheduler: [existing scheduler example](../../examples/machine/DeePMD-kit-1.x/machine-lsf-slurm-cp2k.json) - pure local shell testing: - [existing local example](../../examples/run/ch4/machine.json) + [existing local example](../../examples/machine/DeePMD-kit-1.x/machine-local.json) ## How to build `param.json` @@ -410,7 +410,7 @@ Always provide: Use these checked-in repository examples as starting points: - [local CH4 parameter example](../../examples/run/ch4/param.json) -- [local CH4 machine example](../../examples/run/ch4/machine.json) +- [local machine example](../../examples/machine/DeePMD-kit-1.x/machine-local.json) - [CP2K parameter example](../../examples/run/dp2.x-lammps-cp2k/param_CH4_deepmd-kit-2.0.1.json) - [VASP parameter example](../../examples/run/dp2.x-lammps-vasp/CH4/param_CH4_deepmd-kit-2.x.json) - [scheduler machine example](../../examples/machine/DeePMD-kit-1.x/machine-lsf-slurm-cp2k.json) diff --git a/tests/test_dpgen_run_skill.py b/tests/test_dpgen_run_skill.py index cc783c953..31fdaf6db 100644 --- a/tests/test_dpgen_run_skill.py +++ b/tests/test_dpgen_run_skill.py @@ -1,7 +1,10 @@ +import json import re import unittest from pathlib import Path +from dpgen.remote.decide_machine import convert_mdata + class TestDPGenRunSkill(unittest.TestCase): def test_all_repository_relative_links_exist(self): @@ -13,9 +16,14 @@ def test_all_repository_relative_links_exist(self): for target in link_pattern.findall(skill_path.read_text()): if "://" in target or target.startswith("#"): continue - target_path = target.split("#", 1)[0] - if not (skill_path.parent / target_path).resolve().exists(): + resolved_target = (skill_path.parent / target.split("#", 1)[0]).resolve() + try: + resolved_target.relative_to(repository_root) + except ValueError: missing.append(target) + else: + if not resolved_target.exists(): + missing.append(target) self.assertEqual( missing, @@ -24,6 +32,32 @@ def test_all_repository_relative_links_exist(self): + ", ".join(missing), ) + def test_linked_machine_examples_use_current_schema(self): + repository_root = Path(__file__).resolve().parents[1] + examples = [ + repository_root + / "examples" + / "machine" + / "DeePMD-kit-1.x" + / "machine-local.json", + repository_root + / "examples" + / "machine" + / "DeePMD-kit-1.x" + / "machine-lsf-slurm-cp2k.json", + ] + + for example in examples: + with self.subTest(example=example): + machine_data = json.loads(example.read_text()) + for stage in ("train", "model_devi", "fp"): + self.assertIn(stage, machine_data) + converted = convert_mdata(machine_data) + for stage in ("train", "model_devi", "fp"): + self.assertIn(f"{stage}_machine", converted) + self.assertIn(f"{stage}_resources", converted) + self.assertIn(f"{stage}_command", converted) + if __name__ == "__main__": unittest.main() From 91ecc4275bd8c3a9f98ca6d714ea38dc24316c76 Mon Sep 17 00:00:00 2001 From: SchrodingersCattt Date: Wed, 12 Aug 2026 05:55:35 +0000 Subject: [PATCH 09/23] test: validate documented machine examples --- tests/test_dpgen_run_skill.py | 72 ++++++++++++++++++++++++++++------- 1 file changed, 58 insertions(+), 14 deletions(-) diff --git a/tests/test_dpgen_run_skill.py b/tests/test_dpgen_run_skill.py index 31fdaf6db..c5f9abbef 100644 --- a/tests/test_dpgen_run_skill.py +++ b/tests/test_dpgen_run_skill.py @@ -5,15 +5,16 @@ from dpgen.remote.decide_machine import convert_mdata +LINK_PATTERN = re.compile(r"\[[^]]+\]\(([^)]+)\)") + class TestDPGenRunSkill(unittest.TestCase): def test_all_repository_relative_links_exist(self): repository_root = Path(__file__).resolve().parents[1] skill_path = repository_root / "skills" / "dpgen-run" / "SKILL.md" - link_pattern = re.compile(r"\[[^]]+\]\(([^)]+)\)") missing = [] - for target in link_pattern.findall(skill_path.read_text()): + for target in LINK_PATTERN.findall(skill_path.read_text()): if "://" in target or target.startswith("#"): continue resolved_target = (skill_path.parent / target.split("#", 1)[0]).resolve() @@ -34,29 +35,72 @@ def test_all_repository_relative_links_exist(self): def test_linked_machine_examples_use_current_schema(self): repository_root = Path(__file__).resolve().parents[1] - examples = [ - repository_root - / "examples" - / "machine" - / "DeePMD-kit-1.x" - / "machine-local.json", - repository_root - / "examples" - / "machine" - / "DeePMD-kit-1.x" - / "machine-lsf-slurm-cp2k.json", - ] + skill_path = repository_root / "skills" / "dpgen-run" / "SKILL.md" + machine_examples_root = repository_root / "examples" / "machine" + examples = set() + for target in LINK_PATTERN.findall(skill_path.read_text()): + target_path = target.split("#", 1)[0] + if not target_path.endswith(".json"): + continue + resolved_target = (skill_path.parent / target_path).resolve() + try: + resolved_target.relative_to(machine_examples_root) + except ValueError: + continue + examples.add(resolved_target) + self.assertTrue(examples, "No examples/machine/*.json links found") for example in examples: with self.subTest(example=example): machine_data = json.loads(example.read_text()) + stage_values = {} for stage in ("train", "model_devi", "fp"): self.assertIn(stage, machine_data) + stage_data = machine_data[stage] + if isinstance(stage_data, list): + self.assertTrue(stage_data) + stage_data = stage_data[0] + self.assertIsInstance(stage_data, dict) + stage_values[stage] = stage_data + + machine = stage_data.get("machine") + resources = stage_data.get("resources") + command = stage_data.get("command") + self.assertIsInstance(machine, dict) + self.assertTrue(machine) + self.assertIsInstance(resources, dict) + self.assertTrue(resources) + self.assertIsInstance(command, str) + self.assertTrue(command.strip()) + for field in ("batch_type", "context_type", "local_root"): + self.assertIn(field, machine) + self.assertTrue(machine[field]) + for field in ( + "number_node", + "cpu_per_node", + "gpu_per_node", + "group_size", + ): + self.assertIn(field, resources) converted = convert_mdata(machine_data) for stage in ("train", "model_devi", "fp"): self.assertIn(f"{stage}_machine", converted) self.assertIn(f"{stage}_resources", converted) self.assertIn(f"{stage}_command", converted) + converted_machine = converted[f"{stage}_machine"] + converted_resources = converted[f"{stage}_resources"] + converted_command = converted[f"{stage}_command"] + self.assertIsInstance(converted_machine, dict) + self.assertTrue(converted_machine) + self.assertIsInstance(converted_resources, dict) + self.assertTrue(converted_resources) + self.assertIsInstance(converted_command, str) + self.assertTrue(converted_command.strip()) + self.assertEqual(converted_machine, stage_values[stage]["machine"]) + self.assertEqual( + converted_resources, stage_values[stage]["resources"] + ) + self.assertEqual(converted_command, stage_values[stage]["command"]) if __name__ == "__main__": From 90e1be0804a0577916b90ec6d54a05b8f838d57f Mon Sep 17 00:00:00 2001 From: njzjz-bot Date: Sun, 30 Aug 2026 00:21:51 +0800 Subject: [PATCH 10/23] docs: correct DP-GEN skill validation Coding-Agent: Codex Codex-Version: codex-cli 0.151.0 Model: gpt-5.6-sol Reasoning-Effort: xhigh --- skills/dpgen-run/SKILL.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/skills/dpgen-run/SKILL.md b/skills/dpgen-run/SKILL.md index 3375aa5c3..f232a6e35 100644 --- a/skills/dpgen-run/SKILL.md +++ b/skills/dpgen-run/SKILL.md @@ -30,7 +30,7 @@ dpgen run param.json machine.json Environment boundary rule: -- Outer layer: run `dpgen run param.json machine.json` in an activated environment where `dpgen --version` works. +- Outer layer: run `dpgen run param.json machine.json` in an activated environment where `dpgen -h` works. - Inner layer: for scheduler stages, explicitly activate runtime in `resources.source_list` on the server side. ## Critical Pitfalls (Must Embed) @@ -154,7 +154,7 @@ Do not guess conda environment names, module names, or site-specific paths. Use an activated DP-GEN environment and verify with: ```bash -dpgen --version +dpgen -h ``` Do not start run from a shell where `dpgen` is unavailable. @@ -238,7 +238,7 @@ Collect the following information before generating files. - `fp_task_max` — maximum number of FP tasks per iteration - `fp_task_min` — minimum number to trigger FP - Backend-specific settings: - - VASP: `fp_pp_path`, `fp_pp_files`, `fp_incar` or `fp_params` + - VASP: `fp_pp_path`, `fp_pp_files`, and `fp_incar` - CP2K: `user_fp_params` (nested dict representing cp2k input) - ABACUS: `user_fp_params`, `fp_pp_path`, `fp_pp_files`, `fp_orb_files` - Gaussian: `fp_params` (keywords, nproc, multiplicity) @@ -345,7 +345,7 @@ Before execution, validate the workflow in this order: - Step 1: confirm outer-layer `dpgen` is available: ```bash - dpgen --version + dpgen -h ``` - Step 2: validate JSON syntax: From 5ae7c0e2696525b069065b7510b6316d8e0b42fe Mon Sep 17 00:00:00 2001 From: SchrodingersCattt Date: Sun, 30 Aug 2026 01:36:14 +0000 Subject: [PATCH 11/23] fix(skills): validate dpgen-run examples --- skills/dpgen-run/SKILL.md | 33 +++++++++++++++++++-------------- tests/test_dpgen_run_skill.py | 30 ++++++++++++++++++++++++++++++ 2 files changed, 49 insertions(+), 14 deletions(-) diff --git a/skills/dpgen-run/SKILL.md b/skills/dpgen-run/SKILL.md index f232a6e35..d30cec8e6 100644 --- a/skills/dpgen-run/SKILL.md +++ b/skills/dpgen-run/SKILL.md @@ -1,7 +1,6 @@ --- name: dpgen-run description: Prepare, explain, validate, and run DP-GEN concurrent learning workflows for training deep potential models via iterative exploration. Use when the user wants to generate or modify `param.json` and `machine.json` for `dpgen run`, configure training/exploration/labeling iterations, select descriptor types, set trust levels, define model_devi_jobs, or inspect run outputs. -compatibility: Requires a runnable environment with Python and an activated DP-GEN runtime where `dpgen` is available in PATH for the outer run command. Real execution also requires DeePMD-kit, LAMMPS (with DeePMD plugin for model_devi), and any backend-specific software required by the selected `fp_style`. For scheduler execution, each stage environment must be explicitly activated in `resources.source_list`. license: LGPL-3.0-or-later metadata: author: MatMaster @@ -15,6 +14,13 @@ Use this skill when the user wants to prepare, explain, validate, or execute the This skill is for the main DP-GEN iterative loop: train an ensemble of deep potential models, explore configuration space via LAMMPS MD, select uncertain structures, label them with first-principles calculations, and feed new data back into training. +## Requirements + +Preparation and validation require Python and DP-GEN. Actual workflow execution +also requires DeePMD-kit, a compatible exploration engine such as LAMMPS, and +the software selected by `fp_style`. Scheduler stages must activate their +runtime explicitly in `resources.source_list`. + ## Core Rule (Critical) DP-GEN run always uses **two parameter classes** and therefore **two JSON files**: @@ -52,7 +58,7 @@ These are verified failure modes discovered through testing. Treat as hard rules 1. **`type_map.raw` ordering** — The `type_map.raw` in your `init_data_sys` directories must exactly match the `type_map` array in `param.json`. Mismatches cause silent data corruption. -1. **DeePMD-kit input format** — `default_training_param` uses nested structure: `model.descriptor`, `model.fitting_net`, `learning_rate`, `loss`, `training`. The `training` block may leave `training_data.systems` unset or empty — dpgen fills it. Check the configured `deepmd_version` and installed DeePMD-kit before using 3.x-specific features. +1. **DeePMD-kit input format** — `default_training_param` uses `model`, `learning_rate`, `loss`, and `training` blocks. DeePMD-kit 2.x–3.x place systems under `training.training_data`, while 1.x uses `training.systems`; dpgen fills the version-appropriate systems field. Check the configured `deepmd_version` and installed DeePMD-kit before generating the rest of the training input or using 3.x-specific features. 1. **`model_devi_engine` defaults to LAMMPS** — No need to set `model_devi_engine` explicitly when using LAMMPS. Only set it for alternative engines. @@ -201,15 +207,15 @@ Collect the following information before generating files. ### System information - `type_map` — ordered element symbols -- `mass_map` — atomic masses matching `type_map` order -- `init_data_prefix` — prefix for init_data_sys paths +- `mass_map` — optional atomic masses matching `type_map` order; defaults to `"auto"` +- `init_data_prefix` — optional prefix for `init_data_sys` paths - `init_data_sys` — list of paths to initial training data (deepmd/npy format) -- `sys_configs_prefix` — prefix for sys_configs paths +- `sys_configs_prefix` — optional prefix for `sys_configs` paths - `sys_configs` — list of lists of structure file paths ### Training setup -- `numb_models` — number of ensemble models (default: 4) +- `numb_models` — number of ensemble models (4 is recommended) - `default_training_param` — full DeePMD training input: - `model.descriptor` — descriptor type and settings - `model.fitting_net` — fitting network settings @@ -234,7 +240,7 @@ Collect the following information before generating files. ### FP setup -- `fp_style` — backend: `"vasp"`, `"cp2k"`, `"abacus"`, `"gaussian"`, `"pwscf"`, or `"none"` +- `fp_style` — a labeling backend registered in the [run argument schema](../../dpgen/generator/arginfo.py), such as `"vasp"`, `"cp2k"`, `"abacus"`, `"gaussian"`, or `"pwscf"`; `"none"` is not a valid current choice - `fp_task_max` — maximum number of FP tasks per iteration - `fp_task_min` — minimum number to trigger FP - Backend-specific settings: @@ -283,12 +289,12 @@ Construct `param.json` around these logical blocks: Key fields always required: - `type_map` -- `mass_map` - `init_data_sys` - `sys_configs` - `numb_models` - `default_training_param` - `model_devi_dt` +- `model_devi_skip` - `model_devi_f_trust_lo` - `model_devi_f_trust_hi` - `model_devi_jobs` @@ -357,10 +363,9 @@ Before execution, validate the workflow in this order: - Step 3: verify `init_data_sys` directories exist and contain proper deepmd/npy format: - ```bash - # Each directory must have type_map.raw, type.raw, set.000/ - ls init_data/*/type_map.raw - ``` + Resolve each `init_data_sys` entry against `init_data_prefix` when a prefix + is set; do not assume the literal directory `init_data/`. Each NumPy system + must contain `type_map.raw`, `type.raw`, and `set.000/`. - Step 4: verify `type_map.raw` content matches `param.json` `type_map` ordering. @@ -395,6 +400,7 @@ Always provide: - Keep `type_map` ordering consistent with `init_data_sys` type_map.raw files. - If required inputs are missing, stop and ask instead of guessing. - Always spell `se_atten_v2` correctly (not `se_attn_v2`). +- Do not use `fp_style: "none"`; stop before `dpgen run` when only preparation or validation was authorized. - Include `remote_root` when the selected context requires it; do not add it solely to rewrite a working lazy/local configuration. - Use a `batch_type` accepted by the installed dpdispatcher version and @@ -403,13 +409,12 @@ Always provide: - Do not assume outer-shell activation is inherited by stage jobs; for scheduler execution, require explicit `source_list` per stage. - If the user already has working templates, patch them rather than overwriting them blindly. - Do not set `model_devi_engine` unless using a non-LAMMPS engine (it defaults to LAMMPS). -- In `default_training_param`, leave `training.training_data.systems` unset or empty — `dpgen/generator/run.py` fills it from `init_data_sys` automatically. +- In `default_training_param`, leave `training.training_data.systems` unset or empty for DeePMD-kit 2.x–3.x, or `training.systems` unset or empty for 1.x — `dpgen/generator/run.py` fills it from `init_data_sys` automatically. ## Repository examples and references Use these checked-in repository examples as starting points: -- [local CH4 parameter example](../../examples/run/ch4/param.json) - [local machine example](../../examples/machine/DeePMD-kit-1.x/machine-local.json) - [CP2K parameter example](../../examples/run/dp2.x-lammps-cp2k/param_CH4_deepmd-kit-2.0.1.json) - [VASP parameter example](../../examples/run/dp2.x-lammps-vasp/CH4/param_CH4_deepmd-kit-2.x.json) diff --git a/tests/test_dpgen_run_skill.py b/tests/test_dpgen_run_skill.py index c5f9abbef..baef98b85 100644 --- a/tests/test_dpgen_run_skill.py +++ b/tests/test_dpgen_run_skill.py @@ -3,7 +3,9 @@ import unittest from pathlib import Path +from dpgen.generator.arginfo import run_jdata_arginfo from dpgen.remote.decide_machine import convert_mdata +from dpgen.util import normalize LINK_PATTERN = re.compile(r"\[[^]]+\]\(([^)]+)\)") @@ -33,6 +35,34 @@ def test_all_repository_relative_links_exist(self): + ", ".join(missing), ) + def test_linked_parameter_examples_use_current_schema(self): + repository_root = Path(__file__).resolve().parents[1] + skill_path = repository_root / "skills" / "dpgen-run" / "SKILL.md" + parameter_examples_root = repository_root / "examples" / "run" + examples = set() + for target in LINK_PATTERN.findall(skill_path.read_text()): + target_path = target.split("#", 1)[0] + if not target_path.endswith(".json"): + continue + resolved_target = (skill_path.parent / target_path).resolve() + try: + resolved_target.relative_to(parameter_examples_root) + except ValueError: + continue + examples.add(resolved_target) + self.assertTrue(examples, "No examples/run/*.json links found") + + for example in examples: + with self.subTest(example=example): + parameter_data = json.loads(example.read_text()) + normalized = normalize( + run_jdata_arginfo(), + parameter_data, + strict_check=False, + ) + self.assertIsInstance(normalized, dict) + self.assertTrue(normalized) + def test_linked_machine_examples_use_current_schema(self): repository_root = Path(__file__).resolve().parents[1] skill_path = repository_root / "skills" / "dpgen-run" / "SKILL.md" From b7c2a8fe5d64acdece7f7c305368faa00ed60db3 Mon Sep 17 00:00:00 2001 From: SchrodingersCattt Date: Sun, 30 Aug 2026 01:52:35 +0000 Subject: [PATCH 12/23] refactor(skills): load dpgen-run references progressively --- skills/dpgen-run/SKILL.md | 437 ++---------------- skills/dpgen-run/references/machine-json.md | 56 +++ skills/dpgen-run/references/param-json.md | 107 +++++ .../references/validation-and-run.md | 85 ++++ skills/dpgen-run/references/workflow.md | 97 ++++ tests/test_dpgen_run_skill.py | 106 +++-- 6 files changed, 436 insertions(+), 452 deletions(-) create mode 100644 skills/dpgen-run/references/machine-json.md create mode 100644 skills/dpgen-run/references/param-json.md create mode 100644 skills/dpgen-run/references/validation-and-run.md create mode 100644 skills/dpgen-run/references/workflow.md diff --git a/skills/dpgen-run/SKILL.md b/skills/dpgen-run/SKILL.md index d30cec8e6..039108a35 100644 --- a/skills/dpgen-run/SKILL.md +++ b/skills/dpgen-run/SKILL.md @@ -1,428 +1,45 @@ --- name: dpgen-run -description: Prepare, explain, validate, and run DP-GEN concurrent learning workflows for training deep potential models via iterative exploration. Use when the user wants to generate or modify `param.json` and `machine.json` for `dpgen run`, configure training/exploration/labeling iterations, select descriptor types, set trust levels, define model_devi_jobs, or inspect run outputs. +description: Prepare, explain, validate, and run DP-GEN concurrent-learning workflows. Use for dpgen run param.json or machine.json, training and exploration settings, labeling backends, trust levels, model_devi_jobs, execution, or output inspection. license: LGPL-3.0-or-later metadata: author: MatMaster - version: 0.1.0 + version: 0.2.0 repository: https://github.com/deepmodeling/dpgen --- -# DP-GEN Run (Concurrent Learning) +# DP-GEN Run -Use this skill when the user wants to prepare, explain, validate, or execute the `dpgen run` concurrent learning workflow. +Use this skill for the iterative DP-GEN loop: train an ensemble, explore with MD, select uncertain structures, label them, and add the labels to training data. -This skill is for the main DP-GEN iterative loop: train an ensemble of deep potential models, explore configuration space via LAMMPS MD, select uncertain structures, label them with first-principles calculations, and feed new data back into training. +## Core contract -## Requirements +- Keep workflow settings in `param.json` and execution settings in `machine.json`. +- The exact launcher is `dpgen run param.json machine.json`. +- Treat the launcher shell and dispatched stage environments as separate layers. +- Prepare and validate by default. Execute only when the user explicitly requests it and confirms the exact command after validation. -Preparation and validation require Python and DP-GEN. Actual workflow execution -also requires DeePMD-kit, a compatible exploration engine such as LAMMPS, and -the software selected by `fp_style`. Scheduler stages must activate their -runtime explicitly in `resources.source_list`. +## Workflow -## Core Rule (Critical) +1. Confirm this is a `dpgen run` task and inspect existing files first. +1. Collect only missing inputs; patch working configurations instead of rebuilding them. +1. Load only the references needed for the current task. +1. Preserve scientific and site-specific choices unless the user approves changes. +1. Validate files, data, commands, and environments before proposing execution. +1. Report exact paths, the command, unresolved fields, and post-run inspection targets. -DP-GEN run always uses **two parameter classes** and therefore **two JSON files**: +## Progressive references -- **Workflow parameters** -> `param.json` -- **Execution / machine parameters** -> `machine.json` +Read only the relevant files: -Run exactly: +- [Workflow policy](references/workflow.md): scope, requirements, missing-input policy, layout, and reporting. +- [param.json guide](references/param-json.md): systems, training, exploration, FP fields, descriptors, thresholds, and examples. +- [machine.json guide](references/machine-json.md): stage commands, contexts, resources, scheduler environments, and examples. +- [Validation and execution](references/validation-and-run.md): schema checks, data checks, authorization, launch, and output inspection. -```bash -dpgen run param.json machine.json -``` +## Non-negotiable guardrails -Environment boundary rule: - -- Outer layer: run `dpgen run param.json machine.json` in an activated environment where `dpgen -h` works. -- Inner layer: for scheduler stages, explicitly activate runtime in `resources.source_list` on the server side. - -## Critical Pitfalls (Must Embed) - -These are verified failure modes discovered through testing. Treat as hard rules: - -1. **`se_atten_v2` spelling** — The descriptor type is `se_atten_v2` (double-t in "atten"). Writing `se_attn_v2` will silently fail or error. Always verify the exact string. - -1. **`remote_root` is context-dependent** — require it for remote or scheduler - contexts whose installed dpdispatcher schema needs a remote working root. - Lazy/local contexts can omit it. Preserve an existing working omission and - validate against the installed dpdispatcher version. - -1. **`batch_type` must be registered** — canonical names and lowercase aliases - can both be valid. Preserve an existing working value and validate new values - against the installed dpdispatcher version. - -1. **CP2K `user_fp_params` KIND format** — When multiple element kinds are needed, use `"_": ["H", "O"]` with parallel arrays for `POTENTIAL` and `BASIS_SET`. This maps to repeated KIND sections. - -1. **`type_map.raw` ordering** — The `type_map.raw` in your `init_data_sys` directories must exactly match the `type_map` array in `param.json`. Mismatches cause silent data corruption. - -1. **DeePMD-kit input format** — `default_training_param` uses `model`, `learning_rate`, `loss`, and `training` blocks. DeePMD-kit 2.x–3.x place systems under `training.training_data`, while 1.x uses `training.systems`; dpgen fills the version-appropriate systems field. Check the configured `deepmd_version` and installed DeePMD-kit before generating the rest of the training input or using 3.x-specific features. - -1. **`model_devi_engine` defaults to LAMMPS** — No need to set `model_devi_engine` explicitly when using LAMMPS. Only set it for alternative engines. - -1. **`fp_style: "cp2k"` is native** — dpgen v0.12+ supports CP2K natively via `user_fp_params` dict. No external plugins required. - -1. **`sys_configs` format** — List of lists: outer list = systems, inner list = file paths (POSCAR/xyz) for that system. Example: `[["path/to/POSCAR_1"], ["path/to/POSCAR_2"]]`. - -1. **`init_data_sys` format** — List of paths to directories in deepmd/npy format. Each directory must contain `type_map.raw`, `type.raw`, and `set.000/` with numpy arrays. - -## Agent responsibilities - -When using this skill, the agent should: - -1. confirm that the task is a `dpgen run` concurrent learning workflow -1. check whether existing configs or templates are already available -1. collect only the missing training, exploration, FP, and machine inputs -1. select appropriate descriptor type based on system complexity -1. generate or patch `param.json` -1. generate or patch `machine.json` -1. explain important parameters in plain language when asked -1. validate the workflow before execution -1. provide the exact command for running -1. after execution, summarize outputs and next inspection targets - -## Working policy - -### 0. Treat execution as separately authorized - -Prepare, explain, and validate by default. Execute `dpgen run` only when the -user has explicitly requested execution and, after validation, confirmed the -exact command. Supplying configuration files, passing validation, or showing the -command does not by itself authorize an HPC or first-principles workload. - -### 1. Ask only for missing inputs - -Do not ask the user for everything if part of the configuration is already available. - -If the user already provides: - -- a partial `param.json` -- a partial `machine.json` -- a known training template -- a known cluster template - -then patch those files instead of rebuilding everything from scratch. - -### 2. Preserve the user's scientific choices - -Do not silently change: - -- descriptor family or type -- fitting net structure -- fp backend -- trust thresholds -- `type_map` ordering -- `model_devi_jobs` schedule -- ensemble or temperature settings - -If a value looks scientifically questionable, explain the concern instead of silently replacing it. - -### 3. Descriptor selection guidance - -dpgen exposes `train_backend` options for `"tensorflow"` and `"pytorch"`, but that option alone does not guarantee every PyTorch descriptor or every DeePMD-kit 3.x feature works in the user's installed stack. If backend or descriptor compatibility is unclear, verify it against the repo's `train_backend` options, the configured `deepmd_version`, and the installed DeePMD-kit version before generating configs. - -When the user has not specified a descriptor type, recommend based on: - -| Descriptor | Use case | Notes | -| ------------- | -------------------------------------------- | ----------------------------------------------------------------------------------- | -| `se_e2_a` | Simple systems, fast training, well-tested | Classic two-body embedding, requires explicit `sel` list | -| `se_atten` | Multi-component systems, moderate complexity | Attention-based, supports `sel: "auto"` | -| `se_atten_v2` | Modern default, best accuracy/cost balance | **Spell exactly `se_atten_v2`** (double-t). Supports `sel: "auto"`, `attn_layer: 0` | - -For new projects, prefer `se_atten_v2` unless the user has specific needs. If the user requires PyTorch-only models or DPA-2/DPA-3, prefer **dpgen2** unless the current dpgen and DeePMD-kit installation are verified to support the requested model. - -### 4. Keep local and scheduler execution explicit - -If the user wants local execution, produce local-friendly commands. - -If the user wants scheduler execution, produce scheduler-friendly commands and keep queue, partition, and resource requests explicit. - -Do not invent scheduler module names or executable paths. - -### 5. Do not invent environment activation commands - -If the user already has a working activation command such as: - -- `conda activate ...` -- `module load ...` -- `source ...` - -reuse it exactly. - -If execution is requested and the activation method is unknown, ask the user for the precise activation command. - -Do not guess conda environment names, module names, or site-specific paths. - -### 5.1 Outer launcher policy - -Use an activated DP-GEN environment and verify with: - -```bash -dpgen -h -``` - -Do not start run from a shell where `dpgen` is unavailable. - -### 5.2 Outer vs inner runtime boundaries (critical) - -Treat run execution as two separate environment layers: - -1. Outer layer: the shell that launches `dpgen run param.json machine.json` (must have `dpgen` in PATH) -1. Inner layer: stage tasks dispatched by DP-GEN (`train` / `model_devi` / `fp`) on server/runtime side - -Even if the outer layer is correct, inner stage tasks still need explicit runtime setup in `machine.json`. -Do not assume the outer shell environment will be inherited by dispatched stage jobs. -For scheduler-style execution, `resources.source_list` must explicitly activate the required runtime environment. - -### 6. Prefer reproducible output layout - -When generating a run workflow, keep files organized and predictable. - -Recommended structure: - -```text -project/ -├── param.json -├── machine.json -├── init_data/ -│ └── system_000/ -│ ├── type_map.raw -│ ├── type.raw -│ └── set.000/ -├── assets/ -│ └── structures/ -│ └── POSCAR_* -├── cp2k_basis_pp_file/ (if using CP2K) -│ ├── BASIS_MOLOPT -│ └── GTH_POTENTIALS -└── iter.*/ (created by dpgen) -``` - -## Minimum required inputs - -Collect the following information before generating files. - -### System information - -- `type_map` — ordered element symbols -- `mass_map` — optional atomic masses matching `type_map` order; defaults to `"auto"` -- `init_data_prefix` — optional prefix for `init_data_sys` paths -- `init_data_sys` — list of paths to initial training data (deepmd/npy format) -- `sys_configs_prefix` — optional prefix for `sys_configs` paths -- `sys_configs` — list of lists of structure file paths - -### Training setup - -- `numb_models` — number of ensemble models (4 is recommended) -- `default_training_param` — full DeePMD training input: - - `model.descriptor` — descriptor type and settings - - `model.fitting_net` — fitting network settings - - `learning_rate` — learning rate schedule - - `loss` — loss function weights - - `training` — training steps and output settings - -### Exploration setup (model_devi) - -- `model_devi_dt` — MD timestep in ps (e.g. 0.0005 = 0.5 fs) -- `model_devi_skip` — number of initial frames to skip -- `model_devi_f_trust_lo` — lower force deviation threshold (eV/Å) -- `model_devi_f_trust_hi` — upper force deviation threshold (eV/Å) -- `model_devi_clean_traj` — whether to clean trajectory files after selection -- `model_devi_jobs` — list of exploration job specifications: - - `sys_idx` — which systems to explore - - `temps` — temperatures (K) - - `press` — pressures (bar) - - `trj_freq` — trajectory save frequency - - `nsteps` — number of MD steps - - `ensemble` — `"nvt"` or `"npt"` - -### FP setup - -- `fp_style` — a labeling backend registered in the [run argument schema](../../dpgen/generator/arginfo.py), such as `"vasp"`, `"cp2k"`, `"abacus"`, `"gaussian"`, or `"pwscf"`; `"none"` is not a valid current choice -- `fp_task_max` — maximum number of FP tasks per iteration -- `fp_task_min` — minimum number to trigger FP -- Backend-specific settings: - - VASP: `fp_pp_path`, `fp_pp_files`, and `fp_incar` - - CP2K: `user_fp_params` (nested dict representing cp2k input) - - ABACUS: `user_fp_params`, `fp_pp_path`, `fp_pp_files`, `fp_orb_files` - - Gaussian: `fp_params` (keywords, nproc, multiplicity) - - PWSCF: `user_fp_params` - -### Execution setup - -For each stage `train`, `model_devi`, and `fp`, collect or preserve: - -- `command` -- `machine.batch_type` -- `machine.context_type` -- `machine.local_root` -- `machine.remote_root` when required by the selected context; preserve a - working omission for lazy/local execution -- `resources.number_node` -- `resources.cpu_per_node` -- `resources.gpu_per_node` -- `resources.group_size` -- `resources.source_list` (required for scheduler jobs; use it to activate environment explicitly) -- any explicit queue / partition / custom scheduler flags if the user already uses them - -Choose a runtime profile first, then fill the matching template: - -- server-local scheduler: - [existing scheduler example](../../examples/machine/DeePMD-kit-1.x/machine-lsf-slurm-cp2k.json) -- pure local shell testing: - [existing local example](../../examples/machine/DeePMD-kit-1.x/machine-local.json) - -## How to build `param.json` - -Construct `param.json` around these logical blocks: - -1. element and mass definitions (`type_map`, `mass_map`) -1. data source configuration (`init_data_prefix`, `init_data_sys`, `sys_configs_prefix`, `sys_configs`) -1. model ensemble count (`numb_models`) -1. default DeePMD training parameters (`default_training_param`) -1. exploration settings (`model_devi_dt`, `model_devi_skip`, `model_devi_f_trust_lo/hi`, `model_devi_clean_traj`) -1. exploration job schedule (`model_devi_jobs`) -1. FP backend settings (`fp_style`, `fp_task_max`, `fp_task_min`, backend-specific params) - -Key fields always required: - -- `type_map` -- `init_data_sys` -- `sys_configs` -- `numb_models` -- `default_training_param` -- `model_devi_dt` -- `model_devi_skip` -- `model_devi_f_trust_lo` -- `model_devi_f_trust_hi` -- `model_devi_jobs` -- `fp_style` -- `fp_task_max` -- `fp_task_min` - -Trust level guidance: - -- `model_devi_f_trust_lo`: structures below this are "accurate" — not selected -- `model_devi_f_trust_hi`: structures above this are "failed" — discarded -- Between lo and hi: "candidate" — selected for FP labeling -- Typical starting values: lo=0.05–0.10, hi=0.15–0.30 (eV/Å) -- Tighter thresholds = more FP cost, better accuracy -- System-specific tuning is recommended after initial iterations - -Official reference examples: - -- CP2K: - [methane example](../../examples/run/dp2.x-lammps-cp2k/param_CH4_deepmd-kit-2.0.1.json) -- VASP: - [CH4 example](../../examples/run/dp2.x-lammps-vasp/CH4/param_CH4_deepmd-kit-2.x.json) - -## How to build `machine.json` - -Construct `machine.json` with separate stage blocks for: - -- `train` -- `model_devi` -- `fp` - -For each stage, keep the following explicit: - -- `command` — the executable (`dp` for train, `lmp` for model_devi, FP command for fp) -- machine or context configuration -- resources -- queue or partition if needed -- cpu and gpu counts -- custom scheduler flags -- environment activation commands - -Do not merge all stages into one vague machine block. - -Stage-specific commands: - -- `train`: `"dp"` (DeePMD-kit training) -- `model_devi`: `"lmp"` (LAMMPS with DeePMD plugin) -- `fp`: backend-specific (`"vasp_std"`, `"cp2k.popt"`, `"abacus"`, `"pw.x"`, `"g16"`) - -## Validation before run - -Before execution, validate the workflow in this order: - -- Step 1: confirm outer-layer `dpgen` is available: - - ```bash - dpgen -h - ``` - -- Step 2: validate JSON syntax: - - ```bash - python -m json.tool param.json - python -m json.tool machine.json - ``` - -- Step 3: verify `init_data_sys` directories exist and contain proper deepmd/npy format: - - Resolve each `init_data_sys` entry against `init_data_prefix` when a prefix - is set; do not assume the literal directory `init_data/`. Each NumPy system - must contain `type_map.raw`, `type.raw`, and `set.000/`. - -- Step 4: verify `type_map.raw` content matches `param.json` `type_map` ordering. - -- Step 5: verify `sys_configs` structure files exist. - -- Step 6: verify stage commands match the selected software stack. - -- Step 7: for CP2K, verify basis set and potential files are accessible. - -- Step 8: only then run: - - ```bash - dpgen run param.json machine.json - ``` - -## Output contract - -Always provide: - -1. final absolute paths to `param.json` and `machine.json` -1. the exact run command (`dpgen run param.json machine.json`) -1. a short pre-run checklist -1. any unresolved required fields -1. if execution was performed, the main output locations and next files to inspect - -## Guardrails - -- Never merge workflow and machine parameters into one file. -- Never run `dpgen run` before both JSON files are present. -- Never hardcode personal cluster, account, queue, or path settings as universal defaults. -- Never silently change the user's scientific choices. -- Keep `type_map` ordering consistent with `init_data_sys` type_map.raw files. -- If required inputs are missing, stop and ask instead of guessing. -- Always spell `se_atten_v2` correctly (not `se_attn_v2`). -- Do not use `fp_style: "none"`; stop before `dpgen run` when only preparation or validation was authorized. -- Include `remote_root` when the selected context requires it; do not add it - solely to rewrite a working lazy/local configuration. -- Use a `batch_type` accepted by the installed dpdispatcher version and - preserve an existing working canonical name or alias. -- For CP2K KIND sections with multiple elements, use `"_": ["elem1", "elem2"]` array format. -- Do not assume outer-shell activation is inherited by stage jobs; for scheduler execution, require explicit `source_list` per stage. -- If the user already has working templates, patch them rather than overwriting them blindly. -- Do not set `model_devi_engine` unless using a non-LAMMPS engine (it defaults to LAMMPS). -- In `default_training_param`, leave `training.training_data.systems` unset or empty for DeePMD-kit 2.x–3.x, or `training.systems` unset or empty for 1.x — `dpgen/generator/run.py` fills it from `init_data_sys` automatically. - -## Repository examples and references - -Use these checked-in repository examples as starting points: - -- [local machine example](../../examples/machine/DeePMD-kit-1.x/machine-local.json) -- [CP2K parameter example](../../examples/run/dp2.x-lammps-cp2k/param_CH4_deepmd-kit-2.0.1.json) -- [VASP parameter example](../../examples/run/dp2.x-lammps-vasp/CH4/param_CH4_deepmd-kit-2.x.json) -- [scheduler machine example](../../examples/machine/DeePMD-kit-1.x/machine-lsf-slurm-cp2k.json) - -External references: - -- DP-GEN run overview: https://docs.deepmodeling.com/projects/dpgen/en/latest/run/index.html -- run parameter definitions: https://docs.deepmodeling.com/projects/dpgen/en/latest/run/param.html -- run machine definitions: https://docs.deepmodeling.com/projects/dpgen/en/latest/run/mdata.html -- DeePMD-kit model documentation: https://docs.deepmodeling.com/projects/deepmd/en/latest/model/index.html +- Never merge the two JSON files or invent cluster paths, queues, modules, or activation commands. +- Preserve descriptor, fitting network, FP backend, thresholds, schedules, ensemble settings, and `type_map` ordering. +- Spell `se_atten_v2` exactly; do not use `fp_style: "none"` for current `dpgen run`. +- Stop and ask when required scientific or execution inputs cannot be discovered safely. diff --git a/skills/dpgen-run/references/machine-json.md b/skills/dpgen-run/references/machine-json.md new file mode 100644 index 000000000..1ecb12a65 --- /dev/null +++ b/skills/dpgen-run/references/machine-json.md @@ -0,0 +1,56 @@ +# DP-GEN Run machine.json Guide + +Load this reference when creating, patching, explaining, or reviewing `machine.json`. + +## Runtime boundary + +Treat execution as two environments: + +1. the outer shell that launches `dpgen run param.json machine.json` +1. the dispatched `train`, `model_devi`, and `fp` tasks + +The outer shell must resolve `dpgen`. Scheduler jobs must activate their own software in `resources.source_list`; outer activation is not inherited reliably. + +## Stage blocks + +Keep separate `train`, `model_devi`, and `fp` blocks. For each stage, collect or preserve: + +- `command` +- `machine.batch_type` +- `machine.context_type` +- `machine.local_root` +- `machine.remote_root` when the selected context requires it +- `resources.number_node` +- `resources.cpu_per_node` +- `resources.gpu_per_node` +- `resources.group_size` +- `resources.source_list` for scheduler runtime activation +- queue, partition, account, and custom scheduler flags when applicable + +Typical commands are `dp` for training, `lmp` for LAMMPS exploration, and a backend executable such as `vasp_std`, `cp2k.popt`, `abacus`, `pw.x`, or `g16` for labeling. Preserve known working commands. + +## Context and batch compatibility + +`remote_root` is context-dependent. Require it for remote or scheduler contexts whose installed dpdispatcher schema needs a remote working root. Preserve a working omission for lazy or local execution and validate against the installed dpdispatcher version. + +`batch_type` must be registered by the installed dpdispatcher. Canonical names and lowercase aliases can both be valid. Preserve a working value; validate newly introduced values rather than normalizing them by assumption. + +## Starting profiles + +Choose a profile that matches where the outer command runs: + +- server-local scheduler: [scheduler example](../../../examples/machine/DeePMD-kit-1.x/machine-lsf-slurm-cp2k.json) +- pure local shell: [local example](../../../examples/machine/DeePMD-kit-1.x/machine-local.json) + +Patch the closest existing working configuration. Do not transplant site-specific paths or scheduler settings blindly. + +## Guardrails + +- Never combine all stages into one vague block. +- Never invent executables, modules, activation commands, queues, accounts, or paths. +- Keep CPU, GPU, node, and grouping requests explicit. +- Require explicit `source_list` activation for scheduler stages. +- Preserve working local omissions and installed-version aliases. +- Ensure commands match the scientific stack selected in `param.json`. + +External machine reference: https://docs.deepmodeling.com/projects/dpgen/en/latest/run/mdata.html diff --git a/skills/dpgen-run/references/param-json.md b/skills/dpgen-run/references/param-json.md new file mode 100644 index 000000000..0efc5bdec --- /dev/null +++ b/skills/dpgen-run/references/param-json.md @@ -0,0 +1,107 @@ +# DP-GEN Run param.json Guide + +Load this reference when creating, patching, explaining, or reviewing `param.json`. + +The current authoritative schema is [dpgen/generator/arginfo.py](../../../dpgen/generator/arginfo.py). + +## System and data fields + +Collect or preserve: + +- `type_map`: ordered element symbols +- `mass_map`: optional; defaults to `"auto"` +- `init_data_prefix`: optional prefix for `init_data_sys` +- `init_data_sys`: DeepMD NumPy-system directories +- `sys_configs_prefix`: optional prefix for `sys_configs` +- `sys_configs`: list of systems, each represented by a list of structure paths + +`init_data_sys` entries must resolve to directories containing `type_map.raw`, `type.raw`, and `set.000/`. The ordering in every `type_map.raw` must match `type_map`. + +`sys_configs` is a list of lists. The outer list separates systems; each inner list contains structure files for that system. + +## Training fields + +Required core inputs include: + +- `numb_models`: ensemble size; four is a common recommendation, not a schema default +- `default_training_param.model` +- `default_training_param.learning_rate` +- `default_training_param.loss` +- `default_training_param.training` + +DeePMD-kit 1.x uses `training.systems`. DeePMD-kit 2.x and 3.x use `training.training_data.systems`. Leave the version-appropriate systems value unset or empty because DP-GEN fills it from `init_data_sys`. Verify `deepmd_version` and installed DeePMD-kit before using version-specific features. + +### Descriptor selection + +Do not change a user-selected descriptor silently. + +| Descriptor | Typical use | Notes | +| ------------- | --------------------------------------------- | ------------------------------------- | +| `se_e2_a` | simple systems and conservative compatibility | explicit `sel` list | +| `se_atten` | multicomponent systems | supports `sel: "auto"` | +| `se_atten_v2` | modern attention workflow | spell exactly; supports `sel: "auto"` | + +DP-GEN exposes `train_backend` values `"tensorflow"` and `"pytorch"`, but this alone does not guarantee every descriptor or DeePMD-kit feature works in the installed stack. Verify backend, `deepmd_version`, and DeePMD-kit compatibility. Prefer dpgen2 when a requested DPA-2/DPA-3 or PyTorch-only workflow is not verified with this DP-GEN installation. + +## Exploration fields + +Collect or preserve: + +- `model_devi_dt`: MD timestep in ps +- `model_devi_skip`: initial frames to skip +- `model_devi_f_trust_lo` +- `model_devi_f_trust_hi` +- `model_devi_clean_traj` +- `model_devi_jobs` + +Each `model_devi_jobs` item normally defines `sys_idx`, `temps`, `press`, `trj_freq`, `nsteps`, and `ensemble`. + +Do not set `model_devi_engine` for ordinary LAMMPS exploration; LAMMPS is the default. Set it only for a verified alternative engine. + +### Trust-window interpretation + +- below `model_devi_f_trust_lo`: accurate, normally not selected +- between low and high: candidate for labeling +- above `model_devi_f_trust_hi`: failed or unreliable, normally discarded + +Values such as 0.05-0.10 eV/Angstrom for the low threshold and 0.15-0.30 eV/Angstrom for the high threshold are starting ranges, not universal defaults. Tighter thresholds increase labeling cost and require system-specific justification. + +## First-principles fields + +`fp_style` must be a backend accepted by the current schema, for example `"vasp"`, `"cp2k"`, `"abacus"`, `"gaussian"`, or `"pwscf"`. The value `"none"` is not valid for current `dpgen run`. + +Always collect: + +- `fp_style` +- `fp_task_max` +- `fp_task_min` + +Backend-specific inputs include: + +- VASP: `fp_pp_path`, `fp_pp_files`, `fp_incar` +- CP2K: `user_fp_params` +- ABACUS: `user_fp_params`, `fp_pp_path`, `fp_pp_files`, `fp_orb_files` +- Gaussian: `fp_params` +- PWSCF: `user_fp_params` + +CP2K is supported natively. For multiple KIND sections in `user_fp_params`, use `"_": ["H", "O"]` with parallel `POTENTIAL` and `BASIS_SET` arrays. + +## Logical construction order + +1. elements and masses +1. initial data and exploration structures +1. model ensemble and DeePMD training input +1. model-deviation controls and job schedule +1. first-principles backend and task limits + +Do not assume a field is optional from an old example. Validate against the current schema. + +## Checked repository examples + +- [CP2K methane parameters](../../../examples/run/dp2.x-lammps-cp2k/param_CH4_deepmd-kit-2.0.1.json) +- [VASP CH4 parameters](../../../examples/run/dp2.x-lammps-vasp/CH4/param_CH4_deepmd-kit-2.x.json) + +External references: + +- https://docs.deepmodeling.com/projects/dpgen/en/latest/run/param.html +- https://docs.deepmodeling.com/projects/deepmd/en/latest/model/index.html diff --git a/skills/dpgen-run/references/validation-and-run.md b/skills/dpgen-run/references/validation-and-run.md new file mode 100644 index 000000000..8ae59c4ed --- /dev/null +++ b/skills/dpgen-run/references/validation-and-run.md @@ -0,0 +1,85 @@ +# DP-GEN Run Validation and Execution + +Load this reference before claiming a configuration is valid or before executing `dpgen run`. + +## 1. Verify the outer environment + +```bash +dpgen -h +``` + +Stop if the launcher is unavailable. Actual execution also requires DeePMD-kit, the exploration engine, the selected first-principles software, and scheduler access where applicable. + +## 2. Check JSON syntax + +```bash +python -m json.tool param.json +python -m json.tool machine.json +``` + +## 3. Validate against installed schemas + +Use the installed DP-GEN and dpdispatcher code, not documentation alone. + +```python +import json + +from dpgen.generator.arginfo import run_jdata_arginfo +from dpgen.remote.decide_machine import convert_mdata +from dpgen.util import normalize + +with open("param.json") as stream: + normalize(run_jdata_arginfo(), json.load(stream), strict_check=False) + +with open("machine.json") as stream: + convert_mdata(json.load(stream)) +``` + +Schema normalization is necessary but not sufficient; it does not prove paths, executables, scheduler permissions, or scientific choices are correct. + +## 4. Validate data and structures + +Resolve every `init_data_sys` entry against `init_data_prefix` when set. Verify each NumPy system contains: + +- `type_map.raw` +- `type.raw` +- `set.000/` with the expected arrays + +Compare every `type_map.raw` line-for-line with `param.json.type_map`. Verify every structure referenced through `sys_configs_prefix` and `sys_configs` exists. + +## 5. Validate scientific stages + +Confirm: + +- DeePMD-kit version, backend, descriptor, and training-input layout agree +- each exploration job references valid systems and MD settings +- force trust thresholds are ordered and scientifically intentional +- FP inputs and support files match `fp_style` +- CP2K basis and potential files are accessible when CP2K is selected +- `fp_task_min` and `fp_task_max` reflect intended cost limits + +## 6. Validate execution stages + +For `train`, `model_devi`, and `fp`, verify: + +- commands exist in the dispatched environment +- context and batch types are accepted by installed dpdispatcher +- required local and remote roots exist +- scheduler resources and flags are valid +- `resources.source_list` activates the correct inner runtime + +A successful outer `dpgen -h` does not validate dispatched environments. + +## 7. Confirm before launch + +Show the exact files, validation summary, unresolved risks, and command: + +```bash +dpgen run param.json machine.json +``` + +Execute only after the user explicitly requests the run and confirms this exact validated command. If the user requested only preparation or validation, stop here. + +## 8. Inspect outputs + +After launch, report the current `iter.*` directory and the status of training, model-deviation, and FP tasks. Summarize failures from stage logs and report selected and labeled structure counts when available. diff --git a/skills/dpgen-run/references/workflow.md b/skills/dpgen-run/references/workflow.md new file mode 100644 index 000000000..0d9f69e81 --- /dev/null +++ b/skills/dpgen-run/references/workflow.md @@ -0,0 +1,97 @@ +# DP-GEN Run Workflow Policy + +Load this reference for end-to-end preparation, task scoping, project layout, or result reporting. + +## Scope and requirements + +`dpgen run` implements the concurrent-learning loop: + +1. train an ensemble of Deep Potential models +1. explore configuration space, normally with LAMMPS MD +1. select configurations in the model-deviation candidate window +1. label selected structures with the configured first-principles backend +1. add labeled data to the next training iteration + +Preparation and validation require Python and DP-GEN. Execution additionally requires DeePMD-kit, a compatible exploration engine, the software selected by `fp_style`, and any scheduler runtime. + +The workflow always uses: + +- `param.json` for scientific and workflow parameters +- `machine.json` for commands, contexts, and resources +- `dpgen run param.json machine.json` as the launcher + +## Working policy + +### Inspect before asking + +Look for existing configuration files, training inputs, dataset metadata, and machine templates. Ask only for values that cannot be discovered. Patch existing working files rather than rebuilding them without need. + +### Preserve scientific choices + +Do not silently change: + +- descriptor family or settings +- fitting network +- training backend +- first-principles backend +- trust thresholds +- `type_map` ordering +- `model_devi_jobs` schedule +- ensemble size, temperatures, pressures, or MD ensembles + +Explain a concern and request direction before changing a scientific choice. + +### Preserve site-specific choices + +Reuse known activation commands, executable names, queues, partitions, accounts, paths, and scheduler flags exactly. Never guess conda environments, modules, personal paths, or cluster policy. + +### Keep execution separately authorized + +Preparing files, validating them, or displaying a command does not authorize an HPC, MD, training, or first-principles workload. Run only when the user explicitly requests execution and confirms the exact validated command. + +## Recommended layout + +```text +project/ +|-- param.json +|-- machine.json +|-- init_data/ +| `-- system_000/ +| |-- type_map.raw +| |-- type.raw +| `-- set.000/ +|-- assets/ +| `-- structures/ +|-- cp2k_basis_pp_file/ # only when CP2K needs it +`-- iter.*/ # created by DP-GEN +``` + +Keep repeated experiments in separate, clearly named directories derived from one reviewed base configuration. + +## Result contract + +Before execution, report: + +1. absolute paths to `param.json` and `machine.json` +1. exact launcher command +1. validation results +1. unresolved required inputs +1. expected cost-bearing stages + +After execution, report: + +1. run and current iteration status +1. failed or pending stages +1. main output and log locations +1. candidate and labeled counts when available +1. the next files that need inspection + +## General guardrails + +- Never run before both JSON files exist and pass validation. +- Keep `type_map` consistent from data through training and labeling. +- Do not overwrite working user templates blindly. +- Do not assume outer-shell activation reaches dispatched jobs. +- Stop rather than guess a missing scientific or site-specific value. + +Official overview: https://docs.deepmodeling.com/projects/dpgen/en/latest/run/index.html diff --git a/tests/test_dpgen_run_skill.py b/tests/test_dpgen_run_skill.py index baef98b85..ab9787110 100644 --- a/tests/test_dpgen_run_skill.py +++ b/tests/test_dpgen_run_skill.py @@ -8,25 +8,61 @@ from dpgen.util import normalize LINK_PATTERN = re.compile(r"\[[^]]+\]\(([^)]+)\)") +SKILL_RELATIVE_PATH = Path("skills") / "dpgen-run" / "SKILL.md" -class TestDPGenRunSkill(unittest.TestCase): - def test_all_repository_relative_links_exist(self): - repository_root = Path(__file__).resolve().parents[1] - skill_path = repository_root / "skills" / "dpgen-run" / "SKILL.md" +def skill_markdown_files(repository_root): + skill_root = repository_root / SKILL_RELATIVE_PATH.parent + return sorted(skill_root.rglob("*.md")) - missing = [] - for target in LINK_PATTERN.findall(skill_path.read_text()): - if "://" in target or target.startswith("#"): + +def linked_json_examples(repository_root, examples_root): + examples = set() + for markdown_path in skill_markdown_files(repository_root): + for target in LINK_PATTERN.findall(markdown_path.read_text()): + target_path = target.split("#", 1)[0] + if not target_path.endswith(".json"): continue - resolved_target = (skill_path.parent / target.split("#", 1)[0]).resolve() + resolved_target = (markdown_path.parent / target_path).resolve() try: - resolved_target.relative_to(repository_root) + resolved_target.relative_to(examples_root) except ValueError: - missing.append(target) - else: - if not resolved_target.exists(): - missing.append(target) + continue + examples.add(resolved_target) + return examples + + +class TestDPGenRunSkill(unittest.TestCase): + def test_main_skill_is_small_progressive_router(self): + repository_root = Path(__file__).resolve().parents[1] + skill_path = repository_root / SKILL_RELATIVE_PATH + skill_text = skill_path.read_text() + + self.assertLessEqual(len(skill_text.splitlines()), 50) + reference_targets = { + target + for target in LINK_PATTERN.findall(skill_text) + if target.startswith("references/") + } + self.assertEqual(len(reference_targets), 4) + + def test_all_repository_relative_links_exist(self): + repository_root = Path(__file__).resolve().parents[1] + missing = [] + for markdown_path in skill_markdown_files(repository_root): + for target in LINK_PATTERN.findall(markdown_path.read_text()): + if "://" in target or target.startswith("#"): + continue + resolved_target = ( + markdown_path.parent / target.split("#", 1)[0] + ).resolve() + try: + resolved_target.relative_to(repository_root) + except ValueError: + missing.append(f"{markdown_path.name}: {target}") + else: + if not resolved_target.exists(): + missing.append(f"{markdown_path.name}: {target}") self.assertEqual( missing, @@ -37,19 +73,8 @@ def test_all_repository_relative_links_exist(self): def test_linked_parameter_examples_use_current_schema(self): repository_root = Path(__file__).resolve().parents[1] - skill_path = repository_root / "skills" / "dpgen-run" / "SKILL.md" - parameter_examples_root = repository_root / "examples" / "run" - examples = set() - for target in LINK_PATTERN.findall(skill_path.read_text()): - target_path = target.split("#", 1)[0] - if not target_path.endswith(".json"): - continue - resolved_target = (skill_path.parent / target_path).resolve() - try: - resolved_target.relative_to(parameter_examples_root) - except ValueError: - continue - examples.add(resolved_target) + examples_root = repository_root / "examples" / "run" + examples = linked_json_examples(repository_root, examples_root) self.assertTrue(examples, "No examples/run/*.json links found") for example in examples: @@ -65,19 +90,8 @@ def test_linked_parameter_examples_use_current_schema(self): def test_linked_machine_examples_use_current_schema(self): repository_root = Path(__file__).resolve().parents[1] - skill_path = repository_root / "skills" / "dpgen-run" / "SKILL.md" - machine_examples_root = repository_root / "examples" / "machine" - examples = set() - for target in LINK_PATTERN.findall(skill_path.read_text()): - target_path = target.split("#", 1)[0] - if not target_path.endswith(".json"): - continue - resolved_target = (skill_path.parent / target_path).resolve() - try: - resolved_target.relative_to(machine_examples_root) - except ValueError: - continue - examples.add(resolved_target) + examples_root = repository_root / "examples" / "machine" + examples = linked_json_examples(repository_root, examples_root) self.assertTrue(examples, "No examples/machine/*.json links found") for example in examples: @@ -112,6 +126,7 @@ def test_linked_machine_examples_use_current_schema(self): "group_size", ): self.assertIn(field, resources) + converted = convert_mdata(machine_data) for stage in ("train", "model_devi", "fp"): self.assertIn(f"{stage}_machine", converted) @@ -126,11 +141,18 @@ def test_linked_machine_examples_use_current_schema(self): self.assertTrue(converted_resources) self.assertIsInstance(converted_command, str) self.assertTrue(converted_command.strip()) - self.assertEqual(converted_machine, stage_values[stage]["machine"]) self.assertEqual( - converted_resources, stage_values[stage]["resources"] + converted_machine, + stage_values[stage]["machine"], + ) + self.assertEqual( + converted_resources, + stage_values[stage]["resources"], + ) + self.assertEqual( + converted_command, + stage_values[stage]["command"], ) - self.assertEqual(converted_command, stage_values[stage]["command"]) if __name__ == "__main__": From d3ef90eefa436f859af53202d98631796e19e6bb Mon Sep 17 00:00:00 2001 From: SchrodingersCattt Date: Sun, 30 Aug 2026 02:49:05 +0000 Subject: [PATCH 13/23] test(skills): harden dpgen-run validation --- skills/dpgen-run/references/machine-json.md | 6 +- .../references/validation-and-run.md | 1 + tests/test_dpgen_run_skill.py | 73 +++++++++++++++++++ 3 files changed, 78 insertions(+), 2 deletions(-) diff --git a/skills/dpgen-run/references/machine-json.md b/skills/dpgen-run/references/machine-json.md index 1ecb12a65..caf68b766 100644 --- a/skills/dpgen-run/references/machine-json.md +++ b/skills/dpgen-run/references/machine-json.md @@ -27,7 +27,9 @@ Keep separate `train`, `model_devi`, and `fp` blocks. For each stage, collect or - `resources.source_list` for scheduler runtime activation - queue, partition, account, and custom scheduler flags when applicable -Typical commands are `dp` for training, `lmp` for LAMMPS exploration, and a backend executable such as `vasp_std`, `cp2k.popt`, `abacus`, `pw.x`, or `g16` for labeling. Preserve known working commands. +`train_backend` does not rewrite the machine command. `dp` selects the DeePMD installation's default backend; a DeePMD-kit 3.x PyTorch installation may require `dp --pt`. Verify the installed entry point with `dp train -h` or `dp --pt train -h`, then make the `train` command agree with `train_backend`. + +Typical later-stage commands are `lmp` for LAMMPS exploration and a backend executable such as `vasp_std`, `cp2k.popt`, `abacus`, `pw.x`, or `g16` for labeling. Preserve known working commands. ## Context and batch compatibility @@ -51,6 +53,6 @@ Patch the closest existing working configuration. Do not transplant site-specifi - Keep CPU, GPU, node, and grouping requests explicit. - Require explicit `source_list` activation for scheduler stages. - Preserve working local omissions and installed-version aliases. -- Ensure commands match the scientific stack selected in `param.json`. +- Ensure the training command selects the same backend as `train_backend` and all commands match the scientific stack selected in `param.json`. External machine reference: https://docs.deepmodeling.com/projects/dpgen/en/latest/run/mdata.html diff --git a/skills/dpgen-run/references/validation-and-run.md b/skills/dpgen-run/references/validation-and-run.md index 8ae59c4ed..60acea824 100644 --- a/skills/dpgen-run/references/validation-and-run.md +++ b/skills/dpgen-run/references/validation-and-run.md @@ -52,6 +52,7 @@ Compare every `type_map.raw` line-for-line with `param.json.type_map`. Verify ev Confirm: - DeePMD-kit version, backend, descriptor, and training-input layout agree +- `train_backend` and the machine `train.command` select the same backend; for DeePMD-kit 3.x PyTorch, verify `dp --pt train -h` and use `dp --pt` when required - each exploration job references valid systems and MD settings - force trust thresholds are ordered and scientifically intentional - FP inputs and support files match `fp_style` diff --git a/tests/test_dpgen_run_skill.py b/tests/test_dpgen_run_skill.py index ab9787110..fc5d85bbf 100644 --- a/tests/test_dpgen_run_skill.py +++ b/tests/test_dpgen_run_skill.py @@ -1,8 +1,15 @@ +import copy import json import re import unittest from pathlib import Path +from dargs.dargs import ( + ArgumentKeyError, + ArgumentTypeError, + ArgumentValueError, +) + from dpgen.generator.arginfo import run_jdata_arginfo from dpgen.remote.decide_machine import convert_mdata from dpgen.util import normalize @@ -88,6 +95,72 @@ def test_linked_parameter_examples_use_current_schema(self): self.assertIsInstance(normalized, dict) self.assertTrue(normalized) + def test_documented_parameter_schema_invariants(self): + repository_root = Path(__file__).resolve().parents[1] + example = ( + repository_root + / "examples" + / "run" + / "dp2.x-lammps-cp2k" + / "param_CH4_deepmd-kit-2.0.1.json" + ) + parameter_data = json.loads(example.read_text()) + normalized = normalize( + run_jdata_arginfo(), + copy.deepcopy(parameter_data), + strict_check=False, + ) + + self.assertEqual(normalized["model_devi_engine"], "lammps") + self.assertEqual(normalized["train_backend"], "tensorflow") + + without_mass_map = copy.deepcopy(parameter_data) + without_mass_map.pop("mass_map", None) + normalized_without_mass_map = normalize( + run_jdata_arginfo(), + without_mass_map, + strict_check=False, + ) + self.assertEqual(normalized_without_mass_map["mass_map"], "auto") + + invalid_fp_style = copy.deepcopy(parameter_data) + invalid_fp_style["fp_style"] = "none" + with self.assertRaises(ArgumentValueError): + normalize( + run_jdata_arginfo(), + invalid_fp_style, + strict_check=False, + ) + + missing_model_devi_skip = copy.deepcopy(parameter_data) + missing_model_devi_skip.pop("model_devi_skip") + with self.assertRaises(ArgumentKeyError): + normalize( + run_jdata_arginfo(), + missing_model_devi_skip, + strict_check=False, + ) + + flat_sys_configs = copy.deepcopy(parameter_data) + flat_sys_configs["sys_configs"] = ["POSCAR"] + with self.assertRaises(ArgumentTypeError): + normalize( + run_jdata_arginfo(), + flat_sys_configs, + strict_check=False, + ) + + for backend in ("tensorflow", "pytorch"): + with self.subTest(backend=backend): + backend_data = copy.deepcopy(parameter_data) + backend_data["train_backend"] = backend + normalized_backend = normalize( + run_jdata_arginfo(), + backend_data, + strict_check=False, + ) + self.assertEqual(normalized_backend["train_backend"], backend) + def test_linked_machine_examples_use_current_schema(self): repository_root = Path(__file__).resolve().parents[1] examples_root = repository_root / "examples" / "machine" From d6dbf135394ba307008431b9484e152b64741423 Mon Sep 17 00:00:00 2001 From: Ming-Yu Guo Date: Tue, 1 Sep 2026 16:12:42 +0800 Subject: [PATCH 14/23] docs(skills): guard DP-GEN stage recovery --- skills/dpgen-run/references/machine-json.md | 3 ++ .../references/validation-and-run.md | 47 ++++++++++++++++++- 2 files changed, 48 insertions(+), 2 deletions(-) diff --git a/skills/dpgen-run/references/machine-json.md b/skills/dpgen-run/references/machine-json.md index caf68b766..d88fd96cd 100644 --- a/skills/dpgen-run/references/machine-json.md +++ b/skills/dpgen-run/references/machine-json.md @@ -54,5 +54,8 @@ Patch the closest existing working configuration. Do not transplant site-specifi - Require explicit `source_list` activation for scheduler stages. - Preserve working local omissions and installed-version aliases. - Ensure the training command selects the same backend as `train_backend` and all commands match the scientific stack selected in `param.json`. +- Do not treat a mid-stage `machine.json` edit as an in-place scheduler update. + Before restarting an existing iteration after any stage-machine or resource + change, follow the [restart and recovery checks](validation-and-run.md#7-restart-and-recovery-safety). External machine reference: https://docs.deepmodeling.com/projects/dpgen/en/latest/run/mdata.html diff --git a/skills/dpgen-run/references/validation-and-run.md b/skills/dpgen-run/references/validation-and-run.md index 60acea824..2e2bf6812 100644 --- a/skills/dpgen-run/references/validation-and-run.md +++ b/skills/dpgen-run/references/validation-and-run.md @@ -71,7 +71,50 @@ For `train`, `model_devi`, and `fp`, verify: A successful outer `dpgen -h` does not validate dispatched environments. -## 7. Confirm before launch +## 7. Restart and recovery safety + +Treat a restart from an existing `record.dpgen` as a potentially cost-bearing +operation, especially after changing `machine.json`. + +Current DPDispatcher submission identity includes the resolved machine, +resources, common files, work path, and generated jobs. Job identity includes +the resources and tasks. Consequently, changes such as command, context, +remote root, queue, node count, `group_size`, scheduler flags, environment +activation, or `strategy.ratio_unfinished` can produce a new submission or +job hash. SSH contexts place work beneath a submission-hash-specific remote +directory, so a new identity does not recover the old submission JSON, job IDs, +or completion tags and may submit the stage as fresh work. + +`ratio_unfinished` controls how many tasks may remain unfinished while +evaluating one submission. It does not select only unfinished tasks from a +different submission identity and does not migrate completion state between +hashes. + +Before restarting: + +1. Stop or rule out another active `dpgen run` controller for the same work + directory. +1. Record the current `record.dpgen`, scheduler job IDs, submission records, + and the old and proposed stage configuration. A file checksum is useful for + provenance, but inspect the resolved stage values to judge identity changes. +1. Query the scheduler before retrying an ambiguous submission. Do not infer + that a timeout or disconnected SSH session means nothing was submitted. +1. If the old submission is active, either let it finish under the exact old + configuration or deliberately cancel/migrate it after preserving job IDs + and logs. Do not start a second controller against the edited stage. +1. If enough outputs already exist, verify task counts, completion markers, + parser-readable results, and the configured failure policy before moving to + post-processing. Prefer completing retrieval with the old configuration. + Advance or repair `record.dpgen` only when the installed DP-GEN stage + semantics are verified and the recovery is documented. +1. If a new submission is intentional, independently enumerate its exact task + set and confirm it contains only the work meant to be rerun. Do not assume + DPDispatcher will infer this from existing local outputs. + +Apply machine changes intended for later iterations only after the current +cost-bearing stage has advanced beyond them. + +## 8. Confirm before launch Show the exact files, validation summary, unresolved risks, and command: @@ -81,6 +124,6 @@ dpgen run param.json machine.json Execute only after the user explicitly requests the run and confirms this exact validated command. If the user requested only preparation or validation, stop here. -## 8. Inspect outputs +## 9. Inspect outputs After launch, report the current `iter.*` directory and the status of training, model-deviation, and FP tasks. Summarize failures from stage logs and report selected and labeled structure counts when available. From a00486ac2e7ca30744f00468384e1f8ffa8d8324 Mon Sep 17 00:00:00 2001 From: SchrodingersCattt Date: Tue, 1 Sep 2026 18:24:48 +0800 Subject: [PATCH 15/23] docs(skills): clarify restart submission identity --- skills/dpgen-run/SKILL.md | 2 +- .../references/validation-and-run.md | 23 +++++++++++++++++++ 2 files changed, 24 insertions(+), 1 deletion(-) diff --git a/skills/dpgen-run/SKILL.md b/skills/dpgen-run/SKILL.md index 039108a35..78d1db364 100644 --- a/skills/dpgen-run/SKILL.md +++ b/skills/dpgen-run/SKILL.md @@ -4,7 +4,7 @@ description: Prepare, explain, validate, and run DP-GEN concurrent-learning work license: LGPL-3.0-or-later metadata: author: MatMaster - version: 0.2.0 + version: 0.2.1 repository: https://github.com/deepmodeling/dpgen --- diff --git a/skills/dpgen-run/references/validation-and-run.md b/skills/dpgen-run/references/validation-and-run.md index 2e2bf6812..a1012527b 100644 --- a/skills/dpgen-run/references/validation-and-run.md +++ b/skills/dpgen-run/references/validation-and-run.md @@ -85,6 +85,29 @@ job hash. SSH contexts place work beneath a submission-hash-specific remote directory, so a new identity does not recover the old submission JSON, job IDs, or completion tags and may submit the stage as fresh work. +### What the machine-file checksum means + +DPDispatcher does not hash the raw bytes of `machine.json`. The checksum is +useful provenance, but restart behavior follows the resolved current-stage +submission identity: + +| Restart situation | Expected behavior | +| --- | --- | +| `machine.json` is byte-identical, the DP-GEN/DPDispatcher versions are unchanged, and the same stage generates the same tasks and common-file lists | The same submission hash is expected, so DPDispatcher can recover the existing submission JSON and scheduler job IDs. | +| The file checksum changed, but the resolved block for the active stage did not (for example, only `fp` changed while `train` is active) | The active-stage submission hash may remain unchanged and recover normally. The raw checksum alone cannot decide this. | +| A resolved machine/resource field for the active stage changed, or the generated tasks/common-file lists changed | A new submission hash is expected. DP-GEN re-enters the stage named by `record.dpgen`, but DPDispatcher treats its generated task list as a fresh submission rather than importing completion state from the old identity. | + +An unchanged machine file is therefore not sufficient if `param.json`, task +generation, common files, or serialization behavior changed. Conversely, a +changed file checksum does not necessarily restart the active stage. + +After restart, require positive evidence of recovery: the log should report +`Find old submission; recover submission from json file` with the previous +submission hash, the scheduler job IDs should be unchanged, and no new +`was submitted` message should appear. `record.dpgen` is a workflow-stage +pointer; it is not a scheduler completion record and does not cancel an older +active job when a new submission identity is created. + `ratio_unfinished` controls how many tasks may remain unfinished while evaluating one submission. It does not select only unfinished tasks from a different submission identity and does not migrate completion state between From fa7a47d7593f88cddf908bd41a398aa8d7b29ca0 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Tue, 1 Sep 2026 10:25:03 +0000 Subject: [PATCH 16/23] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- skills/dpgen-run/references/validation-and-run.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/skills/dpgen-run/references/validation-and-run.md b/skills/dpgen-run/references/validation-and-run.md index a1012527b..a79cfb4b3 100644 --- a/skills/dpgen-run/references/validation-and-run.md +++ b/skills/dpgen-run/references/validation-and-run.md @@ -91,11 +91,11 @@ DPDispatcher does not hash the raw bytes of `machine.json`. The checksum is useful provenance, but restart behavior follows the resolved current-stage submission identity: -| Restart situation | Expected behavior | -| --- | --- | -| `machine.json` is byte-identical, the DP-GEN/DPDispatcher versions are unchanged, and the same stage generates the same tasks and common-file lists | The same submission hash is expected, so DPDispatcher can recover the existing submission JSON and scheduler job IDs. | -| The file checksum changed, but the resolved block for the active stage did not (for example, only `fp` changed while `train` is active) | The active-stage submission hash may remain unchanged and recover normally. The raw checksum alone cannot decide this. | -| A resolved machine/resource field for the active stage changed, or the generated tasks/common-file lists changed | A new submission hash is expected. DP-GEN re-enters the stage named by `record.dpgen`, but DPDispatcher treats its generated task list as a fresh submission rather than importing completion state from the old identity. | +| Restart situation | Expected behavior | +| --------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `machine.json` is byte-identical, the DP-GEN/DPDispatcher versions are unchanged, and the same stage generates the same tasks and common-file lists | The same submission hash is expected, so DPDispatcher can recover the existing submission JSON and scheduler job IDs. | +| The file checksum changed, but the resolved block for the active stage did not (for example, only `fp` changed while `train` is active) | The active-stage submission hash may remain unchanged and recover normally. The raw checksum alone cannot decide this. | +| A resolved machine/resource field for the active stage changed, or the generated tasks/common-file lists changed | A new submission hash is expected. DP-GEN re-enters the stage named by `record.dpgen`, but DPDispatcher treats its generated task list as a fresh submission rather than importing completion state from the old identity. | An unchanged machine file is therefore not sufficient if `param.json`, task generation, common files, or serialization behavior changed. Conversely, a From 1ea68ab5825fb44347bea1bbe4aeb343e44f1296 Mon Sep 17 00:00:00 2001 From: SchrodingersCattt Date: Thu, 3 Sep 2026 14:34:45 +0000 Subject: [PATCH 17/23] refactor(skills): merge config references and add monitoring guide --- skills/dpgen-run/SKILL.md | 47 ++--- skills/dpgen-run/references/machine-json.md | 61 ------ skills/dpgen-run/references/monitoring.md | 45 +++++ skills/dpgen-run/references/param-json.md | 107 ---------- .../references/validation-and-run.md | 191 ++++++------------ skills/dpgen-run/references/workflow.md | 110 ++++------ tests/test_dpgen_run_skill.py | 74 +++++-- 7 files changed, 217 insertions(+), 418 deletions(-) delete mode 100644 skills/dpgen-run/references/machine-json.md create mode 100644 skills/dpgen-run/references/monitoring.md delete mode 100644 skills/dpgen-run/references/param-json.md diff --git a/skills/dpgen-run/SKILL.md b/skills/dpgen-run/SKILL.md index 78d1db364..d267684b2 100644 --- a/skills/dpgen-run/SKILL.md +++ b/skills/dpgen-run/SKILL.md @@ -1,45 +1,40 @@ --- name: dpgen-run -description: Prepare, explain, validate, and run DP-GEN concurrent-learning workflows. Use for dpgen run param.json or machine.json, training and exploration settings, labeling backends, trust levels, model_devi_jobs, execution, or output inspection. +description: Prepare, explain, validate, and run DP-GEN concurrent-learning workflows using param.json and machine.json. license: LGPL-3.0-or-later metadata: author: MatMaster - version: 0.2.1 + version: 0.3.0 repository: https://github.com/deepmodeling/dpgen --- # DP-GEN Run -Use this skill for the iterative DP-GEN loop: train an ensemble, explore with MD, select uncertain structures, label them, and add the labels to training data. +Use this skill for the iterative train, explore, label, and retrain workflow. -## Core contract +## Contract -- Keep workflow settings in `param.json` and execution settings in `machine.json`. -- The exact launcher is `dpgen run param.json machine.json`. -- Treat the launcher shell and dispatched stage environments as separate layers. -- Prepare and validate by default. Execute only when the user explicitly requests it and confirms the exact command after validation. +- Keep scientific settings in `param.json` and execution settings in `machine.json`. +- Run exactly `dpgen run param.json machine.json`. +- Prepare and validate by default; execute only after explicit confirmation. ## Workflow -1. Confirm this is a `dpgen run` task and inspect existing files first. -1. Collect only missing inputs; patch working configurations instead of rebuilding them. -1. Load only the references needed for the current task. -1. Preserve scientific and site-specific choices unless the user approves changes. -1. Validate files, data, commands, and environments before proposing execution. -1. Report exact paths, the command, unresolved fields, and post-run inspection targets. +1. Inspect existing data and configurations before asking for inputs. +1. Patch working files and preserve scientific/site-specific choices. +1. Load only the references needed for the task. +1. Validate data, schemas, commands, environments, and monitoring signals. +1. Report paths, command, risks, iteration status, and next checks. -## Progressive references +## References -Read only the relevant files: +- [Workflow guidance](references/workflow.md): scope, layout, policy, and reporting. +- [Configuration, validation, and execution](references/validation-and-run.md): both JSON files and safe launch/restart checks. +- [Monitoring and troubleshooting](references/monitoring.md): accuracy trends and diagnosis. -- [Workflow policy](references/workflow.md): scope, requirements, missing-input policy, layout, and reporting. -- [param.json guide](references/param-json.md): systems, training, exploration, FP fields, descriptors, thresholds, and examples. -- [machine.json guide](references/machine-json.md): stage commands, contexts, resources, scheduler environments, and examples. -- [Validation and execution](references/validation-and-run.md): schema checks, data checks, authorization, launch, and output inspection. +## Guardrails -## Non-negotiable guardrails - -- Never merge the two JSON files or invent cluster paths, queues, modules, or activation commands. -- Preserve descriptor, fitting network, FP backend, thresholds, schedules, ensemble settings, and `type_map` ordering. -- Spell `se_atten_v2` exactly; do not use `fp_style: "none"` for current `dpgen run`. -- Stop and ask when required scientific or execution inputs cannot be discovered safely. +- Never merge the JSON files or invent paths, queues, modules, or commands. +- Preserve descriptor, backend, thresholds, schedules, ensemble, and `type_map` choices. +- Spell `se_atten_v2` exactly; `fp_style: "none"` is invalid for current `dpgen run`. +- Stop and ask when required scientific or execution inputs are undiscoverable. diff --git a/skills/dpgen-run/references/machine-json.md b/skills/dpgen-run/references/machine-json.md deleted file mode 100644 index d88fd96cd..000000000 --- a/skills/dpgen-run/references/machine-json.md +++ /dev/null @@ -1,61 +0,0 @@ -# DP-GEN Run machine.json Guide - -Load this reference when creating, patching, explaining, or reviewing `machine.json`. - -## Runtime boundary - -Treat execution as two environments: - -1. the outer shell that launches `dpgen run param.json machine.json` -1. the dispatched `train`, `model_devi`, and `fp` tasks - -The outer shell must resolve `dpgen`. Scheduler jobs must activate their own software in `resources.source_list`; outer activation is not inherited reliably. - -## Stage blocks - -Keep separate `train`, `model_devi`, and `fp` blocks. For each stage, collect or preserve: - -- `command` -- `machine.batch_type` -- `machine.context_type` -- `machine.local_root` -- `machine.remote_root` when the selected context requires it -- `resources.number_node` -- `resources.cpu_per_node` -- `resources.gpu_per_node` -- `resources.group_size` -- `resources.source_list` for scheduler runtime activation -- queue, partition, account, and custom scheduler flags when applicable - -`train_backend` does not rewrite the machine command. `dp` selects the DeePMD installation's default backend; a DeePMD-kit 3.x PyTorch installation may require `dp --pt`. Verify the installed entry point with `dp train -h` or `dp --pt train -h`, then make the `train` command agree with `train_backend`. - -Typical later-stage commands are `lmp` for LAMMPS exploration and a backend executable such as `vasp_std`, `cp2k.popt`, `abacus`, `pw.x`, or `g16` for labeling. Preserve known working commands. - -## Context and batch compatibility - -`remote_root` is context-dependent. Require it for remote or scheduler contexts whose installed dpdispatcher schema needs a remote working root. Preserve a working omission for lazy or local execution and validate against the installed dpdispatcher version. - -`batch_type` must be registered by the installed dpdispatcher. Canonical names and lowercase aliases can both be valid. Preserve a working value; validate newly introduced values rather than normalizing them by assumption. - -## Starting profiles - -Choose a profile that matches where the outer command runs: - -- server-local scheduler: [scheduler example](../../../examples/machine/DeePMD-kit-1.x/machine-lsf-slurm-cp2k.json) -- pure local shell: [local example](../../../examples/machine/DeePMD-kit-1.x/machine-local.json) - -Patch the closest existing working configuration. Do not transplant site-specific paths or scheduler settings blindly. - -## Guardrails - -- Never combine all stages into one vague block. -- Never invent executables, modules, activation commands, queues, accounts, or paths. -- Keep CPU, GPU, node, and grouping requests explicit. -- Require explicit `source_list` activation for scheduler stages. -- Preserve working local omissions and installed-version aliases. -- Ensure the training command selects the same backend as `train_backend` and all commands match the scientific stack selected in `param.json`. -- Do not treat a mid-stage `machine.json` edit as an in-place scheduler update. - Before restarting an existing iteration after any stage-machine or resource - change, follow the [restart and recovery checks](validation-and-run.md#7-restart-and-recovery-safety). - -External machine reference: https://docs.deepmodeling.com/projects/dpgen/en/latest/run/mdata.html diff --git a/skills/dpgen-run/references/monitoring.md b/skills/dpgen-run/references/monitoring.md new file mode 100644 index 000000000..a7adfdb82 --- /dev/null +++ b/skills/dpgen-run/references/monitoring.md @@ -0,0 +1,45 @@ +# DP-GEN Monitoring and Troubleshooting + +Load this reference after each iteration or whenever training accuracy stalls. +Here "accuracy" means energy/force train and validation RMSE; model deviation +measures sampling uncertainty, not accuracy. + +## Record every iteration + +- train/validation energy and force RMSE from `lcurve.out` or equivalent logs; +- model-deviation distributions and the low/high trust-window counts; +- candidate, FP-success, FP-failure, and newly labeled structure counts; +- current `iter.*`, `record.dpgen`, pending jobs, and whether the next iteration + actually contains the new data. + +## Diagnose in this order + +| Symptom | First checks | Focused adjustment | +| --- | --- | --- | +| Train and validation RMSE both stay high | Units, labels, `type_map`, data coverage, descriptor capacity | Fix data/schema first; then review descriptor, fitting net, learning rate, and steps | +| Train RMSE falls but validation RMSE stalls or rises | Split leakage, duplicates, outliers, validation coverage | Deduplicate/re-split; add representative labels; reduce overfitting only after data is sound | +| RMSE is good but model deviation is high | Unseen compositions/geometries, temperature/pressure range, initial structures | Broaden `model_devi_jobs`, temperatures, ensembles, or initial structures; label the candidate window | +| Candidate count is zero | Trust thresholds, MD stability, trajectory frequency, model diversity | Check thresholds and sampling settings before changing the model | +| Candidate count is excessive or mostly failed FP | Thresholds too loose, unstable MD, bad structures, FP convergence | Tighten/repair sampling and FP inputs; do not flood training with failed labels | +| New labels do not improve the next round | FP failures, wrong paths, stale `record.dpgen`, data not appended | Verify stage completion, merged systems, and restart identity | + +## Data checks + +Check units and signs, atom ordering, `type_map.raw`, duplicate structures, +energy/force/stress consistency, abnormal magnitudes, and FP convergence. +Ensure train/validation data cover every composition, geometry, and target +temperature/pressure rather than only adding more near-duplicates. + +## Training checks + +Compare train versus validation curves, loss components, learning-rate decay, +step budget, batch size, model capacity, ensemble seeds, and backend/version +compatibility. Change one major factor at a time and retain the previous curve +for comparison. + +## Sampling and labeling checks + +Inspect MD stability, temperatures, pressures, ensembles, `nsteps`, `trj_freq`, +and trust-window placement. Confirm selected structures are labeled successfully, +new data are included in the next training systems, and old/new data remain +balanced. Record conclusions in the project journal; never retune silently. diff --git a/skills/dpgen-run/references/param-json.md b/skills/dpgen-run/references/param-json.md deleted file mode 100644 index 0efc5bdec..000000000 --- a/skills/dpgen-run/references/param-json.md +++ /dev/null @@ -1,107 +0,0 @@ -# DP-GEN Run param.json Guide - -Load this reference when creating, patching, explaining, or reviewing `param.json`. - -The current authoritative schema is [dpgen/generator/arginfo.py](../../../dpgen/generator/arginfo.py). - -## System and data fields - -Collect or preserve: - -- `type_map`: ordered element symbols -- `mass_map`: optional; defaults to `"auto"` -- `init_data_prefix`: optional prefix for `init_data_sys` -- `init_data_sys`: DeepMD NumPy-system directories -- `sys_configs_prefix`: optional prefix for `sys_configs` -- `sys_configs`: list of systems, each represented by a list of structure paths - -`init_data_sys` entries must resolve to directories containing `type_map.raw`, `type.raw`, and `set.000/`. The ordering in every `type_map.raw` must match `type_map`. - -`sys_configs` is a list of lists. The outer list separates systems; each inner list contains structure files for that system. - -## Training fields - -Required core inputs include: - -- `numb_models`: ensemble size; four is a common recommendation, not a schema default -- `default_training_param.model` -- `default_training_param.learning_rate` -- `default_training_param.loss` -- `default_training_param.training` - -DeePMD-kit 1.x uses `training.systems`. DeePMD-kit 2.x and 3.x use `training.training_data.systems`. Leave the version-appropriate systems value unset or empty because DP-GEN fills it from `init_data_sys`. Verify `deepmd_version` and installed DeePMD-kit before using version-specific features. - -### Descriptor selection - -Do not change a user-selected descriptor silently. - -| Descriptor | Typical use | Notes | -| ------------- | --------------------------------------------- | ------------------------------------- | -| `se_e2_a` | simple systems and conservative compatibility | explicit `sel` list | -| `se_atten` | multicomponent systems | supports `sel: "auto"` | -| `se_atten_v2` | modern attention workflow | spell exactly; supports `sel: "auto"` | - -DP-GEN exposes `train_backend` values `"tensorflow"` and `"pytorch"`, but this alone does not guarantee every descriptor or DeePMD-kit feature works in the installed stack. Verify backend, `deepmd_version`, and DeePMD-kit compatibility. Prefer dpgen2 when a requested DPA-2/DPA-3 or PyTorch-only workflow is not verified with this DP-GEN installation. - -## Exploration fields - -Collect or preserve: - -- `model_devi_dt`: MD timestep in ps -- `model_devi_skip`: initial frames to skip -- `model_devi_f_trust_lo` -- `model_devi_f_trust_hi` -- `model_devi_clean_traj` -- `model_devi_jobs` - -Each `model_devi_jobs` item normally defines `sys_idx`, `temps`, `press`, `trj_freq`, `nsteps`, and `ensemble`. - -Do not set `model_devi_engine` for ordinary LAMMPS exploration; LAMMPS is the default. Set it only for a verified alternative engine. - -### Trust-window interpretation - -- below `model_devi_f_trust_lo`: accurate, normally not selected -- between low and high: candidate for labeling -- above `model_devi_f_trust_hi`: failed or unreliable, normally discarded - -Values such as 0.05-0.10 eV/Angstrom for the low threshold and 0.15-0.30 eV/Angstrom for the high threshold are starting ranges, not universal defaults. Tighter thresholds increase labeling cost and require system-specific justification. - -## First-principles fields - -`fp_style` must be a backend accepted by the current schema, for example `"vasp"`, `"cp2k"`, `"abacus"`, `"gaussian"`, or `"pwscf"`. The value `"none"` is not valid for current `dpgen run`. - -Always collect: - -- `fp_style` -- `fp_task_max` -- `fp_task_min` - -Backend-specific inputs include: - -- VASP: `fp_pp_path`, `fp_pp_files`, `fp_incar` -- CP2K: `user_fp_params` -- ABACUS: `user_fp_params`, `fp_pp_path`, `fp_pp_files`, `fp_orb_files` -- Gaussian: `fp_params` -- PWSCF: `user_fp_params` - -CP2K is supported natively. For multiple KIND sections in `user_fp_params`, use `"_": ["H", "O"]` with parallel `POTENTIAL` and `BASIS_SET` arrays. - -## Logical construction order - -1. elements and masses -1. initial data and exploration structures -1. model ensemble and DeePMD training input -1. model-deviation controls and job schedule -1. first-principles backend and task limits - -Do not assume a field is optional from an old example. Validate against the current schema. - -## Checked repository examples - -- [CP2K methane parameters](../../../examples/run/dp2.x-lammps-cp2k/param_CH4_deepmd-kit-2.0.1.json) -- [VASP CH4 parameters](../../../examples/run/dp2.x-lammps-vasp/CH4/param_CH4_deepmd-kit-2.x.json) - -External references: - -- https://docs.deepmodeling.com/projects/dpgen/en/latest/run/param.html -- https://docs.deepmodeling.com/projects/deepmd/en/latest/model/index.html diff --git a/skills/dpgen-run/references/validation-and-run.md b/skills/dpgen-run/references/validation-and-run.md index a79cfb4b3..03b161ee8 100644 --- a/skills/dpgen-run/references/validation-and-run.md +++ b/skills/dpgen-run/references/validation-and-run.md @@ -1,152 +1,79 @@ -# DP-GEN Run Validation and Execution +# DP-GEN Configuration, Validation, and Execution -Load this reference before claiming a configuration is valid or before executing `dpgen run`. +Load this reference when creating, reviewing, validating, launching, or +recovering a `dpgen run` configuration. -## 1. Verify the outer environment +## 1. `param.json` essentials -```bash -dpgen -h -``` +Preserve or collect: -Stop if the launcher is unavailable. Actual execution also requires DeePMD-kit, the exploration engine, the selected first-principles software, and scheduler access where applicable. +- `type_map` (ordered symbols), optional `mass_map`, `init_data_prefix`, and `init_data_sys`; +- `sys_configs_prefix` and `sys_configs` (a list of systems, each a list of structures); +- `numb_models` and complete `default_training_param.model`, `learning_rate`, `loss`, and `training`; +- `model_devi_dt`, `model_devi_skip`, trust thresholds, cleanup flag, and `model_devi_jobs`; +- `fp_style`, `fp_task_min`, `fp_task_max`, and backend-specific FP inputs. -## 2. Check JSON syntax +DeepMD-kit 1.x uses `training.systems`; 2.x/3.x use +`training.training_data.systems`. Leave the version-appropriate value empty so +DP-GEN can fill it from `init_data_sys`. Verify `deepmd_version` and the +installed stack before using version-specific descriptors or features. -```bash -python -m json.tool param.json -python -m json.tool machine.json -``` +`train_backend` accepts `tensorflow` and `pytorch`, but compatibility still +depends on the installed DeePMD-kit and descriptor. Verify the combination; +spell `se_atten_v2` exactly. Ordinary LAMMPS exploration uses the default +`model_devi_engine`; set an alternative only when verified. Current `dpgen run` +does not accept `fp_style: "none"`. -## 3. Validate against installed schemas +Each `init_data_sys` directory must contain `type_map.raw`, `type.raw`, and +`set.000/`; every `type_map.raw` must match `param.json.type_map` exactly. -Use the installed DP-GEN and dpdispatcher code, not documentation alone. +## 2. `machine.json` essentials -```python -import json +Keep separate `train`, `model_devi`, and `fp` blocks. Each block supplies a +`command`, `machine` (`batch_type`, `context_type`, `local_root`, and any +required `remote_root`/profile), and `resources` (`number_node`, CPU/GPU counts, +`group_size`, queue/partition flags, and scheduler `source_list`). See the +[repository machine schema walkthrough](../../../doc/run/example-of-machine.md) +for generic local, scheduler, and SSH shapes. -from dpgen.generator.arginfo import run_jdata_arginfo -from dpgen.remote.decide_machine import convert_mdata -from dpgen.util import normalize +The launcher shell and dispatched tasks are different environments. Scheduler +jobs must activate their own runtime through `source_list`; outer activation is +not inherited reliably. The training command must select the same backend as +`train_backend` (for DeePMD-kit 3.x PyTorch this may be `dp --pt`). Validate +commands, context types, batch aliases, roots, and resources against installed +DPDispatcher rather than normalizing by assumption. -with open("param.json") as stream: - normalize(run_jdata_arginfo(), json.load(stream), strict_check=False) +## 3. Validate before launch -with open("machine.json") as stream: - convert_mdata(json.load(stream)) +```bash +dpgen -h +python -m json.tool param.json +python -m json.tool machine.json ``` -Schema normalization is necessary but not sufficient; it does not prove paths, executables, scheduler permissions, or scientific choices are correct. - -## 4. Validate data and structures - -Resolve every `init_data_sys` entry against `init_data_prefix` when set. Verify each NumPy system contains: - -- `type_map.raw` -- `type.raw` -- `set.000/` with the expected arrays - -Compare every `type_map.raw` line-for-line with `param.json.type_map`. Verify every structure referenced through `sys_configs_prefix` and `sys_configs` exists. - -## 5. Validate scientific stages - -Confirm: - -- DeePMD-kit version, backend, descriptor, and training-input layout agree -- `train_backend` and the machine `train.command` select the same backend; for DeePMD-kit 3.x PyTorch, verify `dp --pt train -h` and use `dp --pt` when required -- each exploration job references valid systems and MD settings -- force trust thresholds are ordered and scientifically intentional -- FP inputs and support files match `fp_style` -- CP2K basis and potential files are accessible when CP2K is selected -- `fp_task_min` and `fp_task_max` reflect intended cost limits +Normalize `param.json` with `run_jdata_arginfo()` and convert `machine.json` with +`convert_mdata()`. Then resolve all data/structure paths, compare type maps, +check executables and scheduler access, and verify FP inputs and cost limits. -## 6. Validate execution stages +## 4. Restart and recovery -For `train`, `model_devi`, and `fp`, verify: +Treat a restart from `record.dpgen` as cost-bearing. A changed active-stage +command, context, remote root, queue, resources, `group_size`, flags, +activation, task list, or common files can create a new DPDispatcher submission +identity. It will not import completion state from the old identity. -- commands exist in the dispatched environment -- context and batch types are accepted by installed dpdispatcher -- required local and remote roots exist -- scheduler resources and flags are valid -- `resources.source_list` activates the correct inner runtime - -A successful outer `dpgen -h` does not validate dispatched environments. - -## 7. Restart and recovery safety - -Treat a restart from an existing `record.dpgen` as a potentially cost-bearing -operation, especially after changing `machine.json`. - -Current DPDispatcher submission identity includes the resolved machine, -resources, common files, work path, and generated jobs. Job identity includes -the resources and tasks. Consequently, changes such as command, context, -remote root, queue, node count, `group_size`, scheduler flags, environment -activation, or `strategy.ratio_unfinished` can produce a new submission or -job hash. SSH contexts place work beneath a submission-hash-specific remote -directory, so a new identity does not recover the old submission JSON, job IDs, -or completion tags and may submit the stage as fresh work. - -### What the machine-file checksum means - -DPDispatcher does not hash the raw bytes of `machine.json`. The checksum is -useful provenance, but restart behavior follows the resolved current-stage -submission identity: - -| Restart situation | Expected behavior | -| --------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `machine.json` is byte-identical, the DP-GEN/DPDispatcher versions are unchanged, and the same stage generates the same tasks and common-file lists | The same submission hash is expected, so DPDispatcher can recover the existing submission JSON and scheduler job IDs. | -| The file checksum changed, but the resolved block for the active stage did not (for example, only `fp` changed while `train` is active) | The active-stage submission hash may remain unchanged and recover normally. The raw checksum alone cannot decide this. | -| A resolved machine/resource field for the active stage changed, or the generated tasks/common-file lists changed | A new submission hash is expected. DP-GEN re-enters the stage named by `record.dpgen`, but DPDispatcher treats its generated task list as a fresh submission rather than importing completion state from the old identity. | - -An unchanged machine file is therefore not sufficient if `param.json`, task -generation, common files, or serialization behavior changed. Conversely, a -changed file checksum does not necessarily restart the active stage. - -After restart, require positive evidence of recovery: the log should report -`Find old submission; recover submission from json file` with the previous -submission hash, the scheduler job IDs should be unchanged, and no new -`was submitted` message should appear. `record.dpgen` is a workflow-stage -pointer; it is not a scheduler completion record and does not cancel an older -active job when a new submission identity is created. - -`ratio_unfinished` controls how many tasks may remain unfinished while -evaluating one submission. It does not select only unfinished tasks from a -different submission identity and does not migrate completion state between -hashes. - -Before restarting: - -1. Stop or rule out another active `dpgen run` controller for the same work - directory. -1. Record the current `record.dpgen`, scheduler job IDs, submission records, - and the old and proposed stage configuration. A file checksum is useful for - provenance, but inspect the resolved stage values to judge identity changes. -1. Query the scheduler before retrying an ambiguous submission. Do not infer - that a timeout or disconnected SSH session means nothing was submitted. -1. If the old submission is active, either let it finish under the exact old - configuration or deliberately cancel/migrate it after preserving job IDs - and logs. Do not start a second controller against the edited stage. -1. If enough outputs already exist, verify task counts, completion markers, - parser-readable results, and the configured failure policy before moving to - post-processing. Prefer completing retrieval with the old configuration. - Advance or repair `record.dpgen` only when the installed DP-GEN stage - semantics are verified and the recovery is documented. -1. If a new submission is intentional, independently enumerate its exact task - set and confirm it contains only the work meant to be rerun. Do not assume - DPDispatcher will infer this from existing local outputs. - -Apply machine changes intended for later iterations only after the current -cost-bearing stage has advanced beyond them. - -## 8. Confirm before launch - -Show the exact files, validation summary, unresolved risks, and command: - -```bash -dpgen run param.json machine.json -``` +Before retrying, stop or rule out another controller, record stage/job IDs and +old/new settings, query the scheduler, and preserve logs. Positive recovery +evidence is the old submission hash, unchanged job IDs, and a recovery message; +absence of a new-submission message is required. `record.dpgen` is a workflow +pointer, not a scheduler completion record. Apply later-iteration machine +changes only after the current cost-bearing stage advances. -Execute only after the user explicitly requests the run and confirms this exact validated command. If the user requested only preparation or validation, stop here. +## 5. Confirm and inspect -## 9. Inspect outputs +Show the exact files, validation summary, unresolved risks, and +`dpgen run param.json machine.json`. Execute only after explicit confirmation. +After launch, inspect `iter.*`, stage logs, failure/pending states, selected and +labeled counts, and [monitoring outputs](monitoring.md). -After launch, report the current `iter.*` directory and the status of training, model-deviation, and FP tasks. Summarize failures from stage logs and report selected and labeled structure counts when available. +External references: https://docs.deepmodeling.com/projects/dpgen/en/latest/run/index.html diff --git a/skills/dpgen-run/references/workflow.md b/skills/dpgen-run/references/workflow.md index 0d9f69e81..c4def4c38 100644 --- a/skills/dpgen-run/references/workflow.md +++ b/skills/dpgen-run/references/workflow.md @@ -1,97 +1,67 @@ -# DP-GEN Run Workflow Policy +# DP-GEN Run Workflow Guidance -Load this reference for end-to-end preparation, task scoping, project layout, or result reporting. +Load this reference for scope, preparation policy, project layout, or reporting. -## Scope and requirements +## Loop -`dpgen run` implements the concurrent-learning loop: +`dpgen run` repeats five activities: -1. train an ensemble of Deep Potential models -1. explore configuration space, normally with LAMMPS MD -1. select configurations in the model-deviation candidate window -1. label selected structures with the configured first-principles backend -1. add labeled data to the next training iteration +1. train an ensemble of Deep Potential models; +1. explore configurations, normally with LAMMPS MD; +1. select structures in the model-deviation candidate window; +1. label selected structures with the configured first-principles backend; +1. add labels to the next training iteration. -Preparation and validation require Python and DP-GEN. Execution additionally requires DeePMD-kit, a compatible exploration engine, the software selected by `fp_style`, and any scheduler runtime. +Use `param.json` for scientific/workflow settings, `machine.json` for commands, +contexts, and resources, and `dpgen run param.json machine.json` as the launcher. +Preparation needs Python and DP-GEN. Execution also needs DeePMD-kit, the +exploration engine, the selected FP software, and any scheduler runtime. -The workflow always uses: +## Preparation policy -- `param.json` for scientific and workflow parameters -- `machine.json` for commands, contexts, and resources -- `dpgen run param.json machine.json` as the launcher +Inspect existing configuration files, training inputs, dataset metadata, and +machine templates first. Ask only for values that cannot be discovered. Patch +working files instead of rebuilding them. -## Working policy +Preserve descriptor and fitting settings, training and FP backends, thresholds, +`type_map` ordering, `model_devi_jobs`, ensemble size, temperatures, pressures, +and MD ensembles. Explain concerns and request direction before changing them. -### Inspect before asking +Reuse known activation commands, executables, queues, partitions, accounts, +paths, and scheduler flags exactly. Never guess site policy or conda/module setup. -Look for existing configuration files, training inputs, dataset metadata, and machine templates. Ask only for values that cannot be discovered. Patch existing working files rather than rebuilding them without need. - -### Preserve scientific choices - -Do not silently change: - -- descriptor family or settings -- fitting network -- training backend -- first-principles backend -- trust thresholds -- `type_map` ordering -- `model_devi_jobs` schedule -- ensemble size, temperatures, pressures, or MD ensembles - -Explain a concern and request direction before changing a scientific choice. - -### Preserve site-specific choices - -Reuse known activation commands, executable names, queues, partitions, accounts, paths, and scheduler flags exactly. Never guess conda environments, modules, personal paths, or cluster policy. - -### Keep execution separately authorized - -Preparing files, validating them, or displaying a command does not authorize an HPC, MD, training, or first-principles workload. Run only when the user explicitly requests execution and confirms the exact validated command. - -## Recommended layout +## Generic layout ```text project/ |-- param.json |-- machine.json |-- init_data/ -| `-- system_000/ -| |-- type_map.raw -| |-- type.raw -| `-- set.000/ -|-- assets/ -| `-- structures/ -|-- cp2k_basis_pp_file/ # only when CP2K needs it -`-- iter.*/ # created by DP-GEN +| `-- raw_xxx/ # DeepMD NumPy system(s) +|-- assets/ # structures and runtime support files +`-- iter.*/ # created by DP-GEN ``` -Keep repeated experiments in separate, clearly named directories derived from one reviewed base configuration. - -## Result contract - -Before execution, report: +Keep repeated experiments in separate, clearly named directories derived from a +reviewed base configuration. Backend-specific files belong under `assets/` or +the path explicitly required by the selected backend. -1. absolute paths to `param.json` and `machine.json` -1. exact launcher command -1. validation results -1. unresolved required inputs -1. expected cost-bearing stages +## Reporting contract -After execution, report: +Before execution, report absolute JSON paths, the exact command, validation +results, unresolved inputs, and cost-bearing stages. Execute only after the user +confirms that exact validated command. -1. run and current iteration status -1. failed or pending stages -1. main output and log locations -1. candidate and labeled counts when available -1. the next files that need inspection +After execution, report the current iteration and stage status, failed/pending +tasks, main logs and outputs, candidate and labeled counts, accuracy trends, and +the next files to inspect. Use [monitoring guidance](monitoring.md) when an +iteration stalls or accuracy fails to improve. -## General guardrails +## Guardrails - Never run before both JSON files exist and pass validation. -- Keep `type_map` consistent from data through training and labeling. -- Do not overwrite working user templates blindly. -- Do not assume outer-shell activation reaches dispatched jobs. +- Keep `type_map` consistent through data, training, and labeling. +- Do not overwrite working templates blindly or assume outer activation reaches jobs. - Stop rather than guess a missing scientific or site-specific value. Official overview: https://docs.deepmodeling.com/projects/dpgen/en/latest/run/index.html diff --git a/tests/test_dpgen_run_skill.py b/tests/test_dpgen_run_skill.py index fc5d85bbf..253c2f60d 100644 --- a/tests/test_dpgen_run_skill.py +++ b/tests/test_dpgen_run_skill.py @@ -45,13 +45,38 @@ def test_main_skill_is_small_progressive_router(self): skill_path = repository_root / SKILL_RELATIVE_PATH skill_text = skill_path.read_text() - self.assertLessEqual(len(skill_text.splitlines()), 50) + self.assertLessEqual(len(skill_text.splitlines()), 40) reference_targets = { target for target in LINK_PATTERN.findall(skill_text) if target.startswith("references/") } - self.assertEqual(len(reference_targets), 4) + self.assertEqual(len(reference_targets), 3) + + def test_reference_files_are_small_and_monitoring_is_actionable(self): + repository_root = Path(__file__).resolve().parents[1] + for markdown_path in skill_markdown_files(repository_root): + with self.subTest(markdown_path=markdown_path): + self.assertLessEqual(len(markdown_path.read_text().splitlines()), 100) + + monitoring = ( + (repository_root / "skills" / "dpgen-run" / "references" / "monitoring.md") + .read_text() + .lower() + ) + for term in ( + "data", + "training", + "sampling", + "labeling", + "rmse", + "model deviation", + ): + self.assertIn(term, monitoring) + + references = repository_root / "skills" / "dpgen-run" / "references" + self.assertFalse((references / "param-json.md").exists()) + self.assertFalse((references / "machine-json.md").exists()) def test_all_repository_relative_links_exist(self): repository_root = Path(__file__).resolve().parents[1] @@ -78,11 +103,21 @@ def test_all_repository_relative_links_exist(self): + ", ".join(missing), ) - def test_linked_parameter_examples_use_current_schema(self): + def test_parameter_examples_use_current_schema(self): repository_root = Path(__file__).resolve().parents[1] - examples_root = repository_root / "examples" / "run" - examples = linked_json_examples(repository_root, examples_root) - self.assertTrue(examples, "No examples/run/*.json links found") + examples = [ + repository_root + / "examples" + / "run" + / "dp2.x-lammps-cp2k" + / "param_CH4_deepmd-kit-2.0.1.json", + repository_root + / "examples" + / "run" + / "dp2.x-lammps-vasp" + / "CH4" + / "param_CH4_deepmd-kit-2.x.json", + ] for example in examples: with self.subTest(example=example): @@ -161,11 +196,15 @@ def test_documented_parameter_schema_invariants(self): ) self.assertEqual(normalized_backend["train_backend"], backend) - def test_linked_machine_examples_use_current_schema(self): + def test_current_machine_example_uses_current_schema(self): repository_root = Path(__file__).resolve().parents[1] - examples_root = repository_root / "examples" / "machine" - examples = linked_json_examples(repository_root, examples_root) - self.assertTrue(examples, "No examples/machine/*.json links found") + examples = [ + repository_root + / "examples" + / "run" + / "dp2.x-lammps-gaussian" + / "machine.json", + ] for example in examples: with self.subTest(example=example): @@ -205,25 +244,16 @@ def test_linked_machine_examples_use_current_schema(self): self.assertIn(f"{stage}_machine", converted) self.assertIn(f"{stage}_resources", converted) self.assertIn(f"{stage}_command", converted) - converted_machine = converted[f"{stage}_machine"] - converted_resources = converted[f"{stage}_resources"] - converted_command = converted[f"{stage}_command"] - self.assertIsInstance(converted_machine, dict) - self.assertTrue(converted_machine) - self.assertIsInstance(converted_resources, dict) - self.assertTrue(converted_resources) - self.assertIsInstance(converted_command, str) - self.assertTrue(converted_command.strip()) self.assertEqual( - converted_machine, + converted[f"{stage}_machine"], stage_values[stage]["machine"], ) self.assertEqual( - converted_resources, + converted[f"{stage}_resources"], stage_values[stage]["resources"], ) self.assertEqual( - converted_command, + converted[f"{stage}_command"], stage_values[stage]["command"], ) From 54a68310e9d6ecb78ecf26b9936cb04665df09c7 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Thu, 3 Sep 2026 16:31:12 +0000 Subject: [PATCH 18/23] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- skills/dpgen-run/references/monitoring.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/skills/dpgen-run/references/monitoring.md b/skills/dpgen-run/references/monitoring.md index a7adfdb82..895f6031f 100644 --- a/skills/dpgen-run/references/monitoring.md +++ b/skills/dpgen-run/references/monitoring.md @@ -14,14 +14,14 @@ measures sampling uncertainty, not accuracy. ## Diagnose in this order -| Symptom | First checks | Focused adjustment | -| --- | --- | --- | -| Train and validation RMSE both stay high | Units, labels, `type_map`, data coverage, descriptor capacity | Fix data/schema first; then review descriptor, fitting net, learning rate, and steps | -| Train RMSE falls but validation RMSE stalls or rises | Split leakage, duplicates, outliers, validation coverage | Deduplicate/re-split; add representative labels; reduce overfitting only after data is sound | -| RMSE is good but model deviation is high | Unseen compositions/geometries, temperature/pressure range, initial structures | Broaden `model_devi_jobs`, temperatures, ensembles, or initial structures; label the candidate window | -| Candidate count is zero | Trust thresholds, MD stability, trajectory frequency, model diversity | Check thresholds and sampling settings before changing the model | -| Candidate count is excessive or mostly failed FP | Thresholds too loose, unstable MD, bad structures, FP convergence | Tighten/repair sampling and FP inputs; do not flood training with failed labels | -| New labels do not improve the next round | FP failures, wrong paths, stale `record.dpgen`, data not appended | Verify stage completion, merged systems, and restart identity | +| Symptom | First checks | Focused adjustment | +| ---------------------------------------------------- | ------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------- | +| Train and validation RMSE both stay high | Units, labels, `type_map`, data coverage, descriptor capacity | Fix data/schema first; then review descriptor, fitting net, learning rate, and steps | +| Train RMSE falls but validation RMSE stalls or rises | Split leakage, duplicates, outliers, validation coverage | Deduplicate/re-split; add representative labels; reduce overfitting only after data is sound | +| RMSE is good but model deviation is high | Unseen compositions/geometries, temperature/pressure range, initial structures | Broaden `model_devi_jobs`, temperatures, ensembles, or initial structures; label the candidate window | +| Candidate count is zero | Trust thresholds, MD stability, trajectory frequency, model diversity | Check thresholds and sampling settings before changing the model | +| Candidate count is excessive or mostly failed FP | Thresholds too loose, unstable MD, bad structures, FP convergence | Tighten/repair sampling and FP inputs; do not flood training with failed labels | +| New labels do not improve the next round | FP failures, wrong paths, stale `record.dpgen`, data not appended | Verify stage completion, merged systems, and restart identity | ## Data checks From 783d5f033632ca911e0c3ff8233397f34c7a0516 Mon Sep 17 00:00:00 2001 From: SchrodingersCattt Date: Thu, 3 Sep 2026 16:48:29 +0000 Subject: [PATCH 19/23] docs(skills): add evidence-based dpgen guardrails --- skills/dpgen-run/SKILL.md | 8 +-- skills/dpgen-run/references/monitoring.md | 69 +++++++++++-------- .../references/validation-and-run.md | 43 ++++++++---- tests/test_dpgen_run_skill.py | 36 ++++++++++ 4 files changed, 108 insertions(+), 48 deletions(-) diff --git a/skills/dpgen-run/SKILL.md b/skills/dpgen-run/SKILL.md index d267684b2..9bde72d27 100644 --- a/skills/dpgen-run/SKILL.md +++ b/skills/dpgen-run/SKILL.md @@ -23,14 +23,14 @@ Use this skill for the iterative train, explore, label, and retrain workflow. 1. Inspect existing data and configurations before asking for inputs. 1. Patch working files and preserve scientific/site-specific choices. 1. Load only the references needed for the task. -1. Validate data, schemas, commands, environments, and monitoring signals. -1. Report paths, command, risks, iteration status, and next checks. +1. Validate schemas, generated task inputs, stage state, and monitoring signals. +1. Report paths, command, per-system evidence, risks, and next checks. ## References - [Workflow guidance](references/workflow.md): scope, layout, policy, and reporting. -- [Configuration, validation, and execution](references/validation-and-run.md): both JSON files and safe launch/restart checks. -- [Monitoring and troubleshooting](references/monitoring.md): accuracy trends and diagnosis. +- [Configuration, validation, and execution](references/validation-and-run.md): both JSON files, generated-input checks, and safe launch/restart checks. +- [Monitoring and troubleshooting](references/monitoring.md): accuracy trends, stage progress, and evidence-based diagnosis. ## Guardrails diff --git a/skills/dpgen-run/references/monitoring.md b/skills/dpgen-run/references/monitoring.md index 895f6031f..2ebb2a630 100644 --- a/skills/dpgen-run/references/monitoring.md +++ b/skills/dpgen-run/references/monitoring.md @@ -1,45 +1,54 @@ # DP-GEN Monitoring and Troubleshooting -Load this reference after each iteration or whenever training accuracy stalls. -Here "accuracy" means energy/force train and validation RMSE; model deviation -measures sampling uncertainty, not accuracy. +Load this reference after each iteration or whenever accuracy or stage progress +stalls. Here "accuracy" means energy/force train and validation RMSE; model +deviation and FP failure ratio are separate signals. -## Record every iteration +## Evidence loop -- train/validation energy and force RMSE from `lcurve.out` or equivalent logs; -- model-deviation distributions and the low/high trust-window counts; -- candidate, FP-success, FP-failure, and newly labeled structure counts; -- current `iter.*`, `record.dpgen`, pending jobs, and whether the next iteration - actually contains the new data. +For each iteration, record the parameter/configuration identity, systems and +temperature/pressure conditions, train/validation RMSE, model-deviation +accurate/candidate/failed counts, FP success/failure counts, current stage, and +whether new labels reached the next training input. Report per-system and +per-condition results before any global aggregate. ## Diagnose in this order -| Symptom | First checks | Focused adjustment | -| ---------------------------------------------------- | ------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------- | -| Train and validation RMSE both stay high | Units, labels, `type_map`, data coverage, descriptor capacity | Fix data/schema first; then review descriptor, fitting net, learning rate, and steps | -| Train RMSE falls but validation RMSE stalls or rises | Split leakage, duplicates, outliers, validation coverage | Deduplicate/re-split; add representative labels; reduce overfitting only after data is sound | -| RMSE is good but model deviation is high | Unseen compositions/geometries, temperature/pressure range, initial structures | Broaden `model_devi_jobs`, temperatures, ensembles, or initial structures; label the candidate window | -| Candidate count is zero | Trust thresholds, MD stability, trajectory frequency, model diversity | Check thresholds and sampling settings before changing the model | -| Candidate count is excessive or mostly failed FP | Thresholds too loose, unstable MD, bad structures, FP convergence | Tighten/repair sampling and FP inputs; do not flood training with failed labels | -| New labels do not improve the next round | FP failures, wrong paths, stale `record.dpgen`, data not appended | Verify stage completion, merged systems, and restart identity | +| Symptom | First checks | Focused adjustment | +| ---------------------------------------------------- | ---------------------------------------------------------------------------- | --------------------------------------------------------------------------- | +| Train and validation RMSE both stay high | Units, labels, type map, coverage, descriptor capacity | Fix data/schema first; then review fitting net, learning rate, and steps | +| Train RMSE falls but validation RMSE stalls or rises | Split leakage, duplicates, outliers, validation coverage | Re-split or deduplicate; add representative labels before changing capacity | +| RMSE is good but model deviation is high | Unseen systems/geometries or conditions, initial structures | Broaden targeted sampling and labeling the candidate window | +| Candidate count is zero | Trust thresholds, MD stability, trajectory frequency, model diversity | Check thresholds and sampling settings before changing training | +| Candidate count is excessive or FP failures rise | Thresholds, unstable MD, malformed structures, FP convergence | Repair inputs/sampling; do not add failed labels | +| Stage does not advance or task count is wrong | record state, schedule length, sys_idx, generated input, submission metadata | Reconcile state and actual tasks; do not restart blindly | +| Effective steps differ from requested steps | Reuse/iteration override in generated input | Compare root and task-level values; fix the override deliberately | -## Data checks +## Data and label checks -Check units and signs, atom ordering, `type_map.raw`, duplicate structures, -energy/force/stress consistency, abnormal magnitudes, and FP convergence. -Ensure train/validation data cover every composition, geometry, and target -temperature/pressure rather than only adding more near-duplicates. +Check units and signs, atom ordering, type maps, duplicates, abnormal +magnitudes, and train/validation coverage. Every training frame must have +finite, parseable energy and force; include virial/stress only when configured +and verified. Reject incomplete, duplicated, or unconverged labels before +merging them into training data. ## Training checks -Compare train versus validation curves, loss components, learning-rate decay, -step budget, batch size, model capacity, ensemble seeds, and backend/version -compatibility. Change one major factor at a time and retain the previous curve -for comparison. +Compare train and validation curves, loss components, learning-rate decay, step +budget, batch size, model capacity, ensemble seeds, and backend/version +compatibility. A good global metric must not hide a failing system or condition. +Change one major factor at a time and retain the previous evidence. -## Sampling and labeling checks +## Sampling and decision checks Inspect MD stability, temperatures, pressures, ensembles, `nsteps`, `trj_freq`, -and trust-window placement. Confirm selected structures are labeled successfully, -new data are included in the next training systems, and old/new data remain -balanced. Record conclusions in the project journal; never retune silently. +trust-window placement, and per-condition distributions. Distinguish a model +problem from missing coverage: good RMSE with high deviation usually calls for +targeted sampling, while high RMSE on covered data calls for data or training +review. + +Monitoring is an evidence-based decision stage, not only status reporting. +Before extending sampling, changing thresholds, increasing steps, or restarting, +record the evidence, exact parameter delta, expected effect, and controller +state. Afterward record the new task identity and outcome; never retune or +restart silently. diff --git a/skills/dpgen-run/references/validation-and-run.md b/skills/dpgen-run/references/validation-and-run.md index 03b161ee8..de979b839 100644 --- a/skills/dpgen-run/references/validation-and-run.md +++ b/skills/dpgen-run/references/validation-and-run.md @@ -43,7 +43,7 @@ not inherited reliably. The training command must select the same backend as commands, context types, batch aliases, roots, and resources against installed DPDispatcher rather than normalizing by assumption. -## 3. Validate before launch +## 3. Validate generated inputs before launch ```bash dpgen -h @@ -52,22 +52,37 @@ python -m json.tool machine.json ``` Normalize `param.json` with `run_jdata_arginfo()` and convert `machine.json` with -`convert_mdata()`. Then resolve all data/structure paths, compare type maps, -check executables and scheduler access, and verify FP inputs and cost limits. +`convert_mdata()`. Resolve all data/structure paths, compare type maps, check +executables and scheduler access, and verify FP inputs and cost limits. + +Before submission, inspect the generated `input.json` for every stage/task. Do +not infer the effective configuration from the root files: check model count, +seed, checkpoint, data systems, `sys_idx`, temperatures/pressures, and final +training steps. When reuse or iteration overrides exist, assert both the default +and effective values. + +Cross-check the intended iteration against `record.dpgen`, stage directories, +DPDispatcher work base/submission metadata, and logs. A stage is not complete +because a directory or job exists. ## 4. Restart and recovery -Treat a restart from `record.dpgen` as cost-bearing. A changed active-stage -command, context, remote root, queue, resources, `group_size`, flags, -activation, task list, or common files can create a new DPDispatcher submission -identity. It will not import completion state from the old identity. - -Before retrying, stop or rule out another controller, record stage/job IDs and -old/new settings, query the scheduler, and preserve logs. Positive recovery -evidence is the old submission hash, unchanged job IDs, and a recovery message; -absence of a new-submission message is required. `record.dpgen` is a workflow -pointer, not a scheduler completion record. Apply later-iteration machine -changes only after the current cost-bearing stage advances. +Treat a restart from `record.dpgen` as cost-bearing. Keep one controller for a +work directory. Before editing state, stop it, back up the state file, record +stage/job IDs and old/new settings, and query the scheduler. + +A changed active-stage command, context, remote root, queue, resources, +`group_size`, flags, activation, task list, or common files can create a new +DPDispatcher submission identity. It will not import completion state from the +old identity. Distinguish recovery of the old submission from an intentional +rerun with a new remote root; never resubmit solely because a connection timed +out. + +Positive recovery evidence is the old submission hash, unchanged job IDs, and a +recovery message; absence of a new-submission message is required. +`record.dpgen` is a workflow pointer, not a scheduler completion record. +Apply later-iteration machine changes only after the current cost-bearing stage +advances. ## 5. Confirm and inspect diff --git a/tests/test_dpgen_run_skill.py b/tests/test_dpgen_run_skill.py index 253c2f60d..d1dc9df55 100644 --- a/tests/test_dpgen_run_skill.py +++ b/tests/test_dpgen_run_skill.py @@ -71,13 +71,49 @@ def test_reference_files_are_small_and_monitoring_is_actionable(self): "labeling", "rmse", "model deviation", + "labeling", + "fp failure", + "system", + "condition", + "override", ): self.assertIn(term, monitoring) + validation = ( + ( + repository_root + / "skills" + / "dpgen-run" + / "references" + / "validation-and-run.md" + ) + .read_text() + .lower() + ) + for term in ( + "record.dpgen", + "input.json", + "submission identity", + "sys_idx", + "generated", + ): + self.assertIn(term, validation) + references = repository_root / "skills" / "dpgen-run" / "references" self.assertFalse((references / "param-json.md").exists()) self.assertFalse((references / "machine-json.md").exists()) + def test_skill_docs_are_project_agnostic(self): + repository_root = Path(__file__).resolve().parents[1] + documents = [ + path.read_text() + for path in skill_markdown_files(repository_root) + if path.name != "test_dpgen_run_skill.py" + ] + text = "\n".join(documents) + self.assertNotRegex(text, r"(?m)(?:^|\s)/(?:[A-Za-z0-9_.-]+/)+") + self.assertNotRegex(text, r"\b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b") + def test_all_repository_relative_links_exist(self): repository_root = Path(__file__).resolve().parents[1] missing = [] From f0c7b33fa963aa043c6231a1c259e3ec8a5eb679 Mon Sep 17 00:00:00 2001 From: SchrodingersCattt Date: Fri, 4 Sep 2026 04:38:57 +0000 Subject: [PATCH 20/23] docs(skills): translate dpgen-run docs to Chinese --- skills/dpgen-run/SKILL.md | 44 +++---- skills/dpgen-run/references/monitoring.md | 98 +++++++------- .../references/validation-and-run.md | 120 ++++++++---------- skills/dpgen-run/references/workflow.md | 79 ++++++------ tests/test_dpgen_run_skill.py | 11 +- 5 files changed, 161 insertions(+), 191 deletions(-) diff --git a/skills/dpgen-run/SKILL.md b/skills/dpgen-run/SKILL.md index 9bde72d27..404634af8 100644 --- a/skills/dpgen-run/SKILL.md +++ b/skills/dpgen-run/SKILL.md @@ -1,40 +1,40 @@ --- name: dpgen-run -description: Prepare, explain, validate, and run DP-GEN concurrent-learning workflows using param.json and machine.json. +description: 使用 param.json 和 machine.json 准备、解释、验证并运行 DP-GEN 并行学习流程。 license: LGPL-3.0-or-later metadata: author: MatMaster - version: 0.3.0 + version: 0.4.0 repository: https://github.com/deepmodeling/dpgen --- # DP-GEN Run -Use this skill for the iterative train, explore, label, and retrain workflow. +用于迭代训练、探索、标注和再训练流程。 -## Contract +## 使用契约 -- Keep scientific settings in `param.json` and execution settings in `machine.json`. -- Run exactly `dpgen run param.json machine.json`. -- Prepare and validate by default; execute only after explicit confirmation. +- 科学参数放在 `param.json`，执行参数放在 `machine.json`。 +- 精确运行命令为 `dpgen run param.json machine.json`。 +- 默认只准备和验证；只有用户明确确认后才执行。 -## Workflow +## 工作流程 -1. Inspect existing data and configurations before asking for inputs. -1. Patch working files and preserve scientific/site-specific choices. -1. Load only the references needed for the task. -1. Validate schemas, generated task inputs, stage state, and monitoring signals. -1. Report paths, command, per-system evidence, risks, and next checks. +1. 先检查已有数据和配置，再询问缺失输入。 +1. 修改现有可用文件，并保留科学及站点选择。 +1. 只加载当前任务需要的 reference。 +1. 验证 schema、生成的任务输入、阶段状态和监控信号。 +1. 汇报路径、命令、分体系证据、风险和后续检查。 -## References +## Reference -- [Workflow guidance](references/workflow.md): scope, layout, policy, and reporting. -- [Configuration, validation, and execution](references/validation-and-run.md): both JSON files, generated-input checks, and safe launch/restart checks. -- [Monitoring and troubleshooting](references/monitoring.md): accuracy trends, stage progress, and evidence-based diagnosis. +- [工作流说明](references/workflow.md)：范围、布局、策略和汇报。 +- [配置、验证与执行](references/validation-and-run.md)：两个 JSON、生成输入检查及安全启动/恢复。 +- [监控与排障](references/monitoring.md)：准确率趋势、阶段进度和证据化诊断。 -## Guardrails +## 防护规则 -- Never merge the JSON files or invent paths, queues, modules, or commands. -- Preserve descriptor, backend, thresholds, schedules, ensemble, and `type_map` choices. -- Spell `se_atten_v2` exactly; `fp_style: "none"` is invalid for current `dpgen run`. -- Stop and ask when required scientific or execution inputs are undiscoverable. +- 不得合并两个 JSON，也不得臆造路径、队列、模块或命令。 +- 保留 descriptor、backend、阈值、计划、ensemble 和 `type_map` 选择。 +- 必须准确写成 `se_atten_v2`；当前 `dpgen run` 不接受 `fp_style: "none"`。 +- 无法安全发现必需的科学或执行输入时，停止并询问。 diff --git a/skills/dpgen-run/references/monitoring.md b/skills/dpgen-run/references/monitoring.md index 2ebb2a630..a7f9385bd 100644 --- a/skills/dpgen-run/references/monitoring.md +++ b/skills/dpgen-run/references/monitoring.md @@ -1,54 +1,44 @@ -# DP-GEN Monitoring and Troubleshooting - -Load this reference after each iteration or whenever accuracy or stage progress -stalls. Here "accuracy" means energy/force train and validation RMSE; model -deviation and FP failure ratio are separate signals. - -## Evidence loop - -For each iteration, record the parameter/configuration identity, systems and -temperature/pressure conditions, train/validation RMSE, model-deviation -accurate/candidate/failed counts, FP success/failure counts, current stage, and -whether new labels reached the next training input. Report per-system and -per-condition results before any global aggregate. - -## Diagnose in this order - -| Symptom | First checks | Focused adjustment | -| ---------------------------------------------------- | ---------------------------------------------------------------------------- | --------------------------------------------------------------------------- | -| Train and validation RMSE both stay high | Units, labels, type map, coverage, descriptor capacity | Fix data/schema first; then review fitting net, learning rate, and steps | -| Train RMSE falls but validation RMSE stalls or rises | Split leakage, duplicates, outliers, validation coverage | Re-split or deduplicate; add representative labels before changing capacity | -| RMSE is good but model deviation is high | Unseen systems/geometries or conditions, initial structures | Broaden targeted sampling and labeling the candidate window | -| Candidate count is zero | Trust thresholds, MD stability, trajectory frequency, model diversity | Check thresholds and sampling settings before changing training | -| Candidate count is excessive or FP failures rise | Thresholds, unstable MD, malformed structures, FP convergence | Repair inputs/sampling; do not add failed labels | -| Stage does not advance or task count is wrong | record state, schedule length, sys_idx, generated input, submission metadata | Reconcile state and actual tasks; do not restart blindly | -| Effective steps differ from requested steps | Reuse/iteration override in generated input | Compare root and task-level values; fix the override deliberately | - -## Data and label checks - -Check units and signs, atom ordering, type maps, duplicates, abnormal -magnitudes, and train/validation coverage. Every training frame must have -finite, parseable energy and force; include virial/stress only when configured -and verified. Reject incomplete, duplicated, or unconverged labels before -merging them into training data. - -## Training checks - -Compare train and validation curves, loss components, learning-rate decay, step -budget, batch size, model capacity, ensemble seeds, and backend/version -compatibility. A good global metric must not hide a failing system or condition. -Change one major factor at a time and retain the previous evidence. - -## Sampling and decision checks - -Inspect MD stability, temperatures, pressures, ensembles, `nsteps`, `trj_freq`, -trust-window placement, and per-condition distributions. Distinguish a model -problem from missing coverage: good RMSE with high deviation usually calls for -targeted sampling, while high RMSE on covered data calls for data or training -review. - -Monitoring is an evidence-based decision stage, not only status reporting. -Before extending sampling, changing thresholds, increasing steps, or restarting, -record the evidence, exact parameter delta, expected effect, and controller -state. Afterward record the new task identity and outcome; never retune or -restart silently. +# DP-GEN 监控与排障 + +在每轮迭代后，或准确率/阶段进度停滞时加载本 reference。“准确率”指能量/力的训练与 +验证 RMSE；model deviation 和 FP failure ratio 是独立信号。 + +## 证据循环 + +每轮记录参数/配置身份、系统及温度/压力条件、训练/验证 RMSE、model-deviation 的 +accurate/candidate/failed 数量、FP 成功/失败数量、当前阶段，以及新标注是否进入下一轮 +训练输入。先汇报分 system、分 condition 结果，再给全局汇总。 + +## 按顺序诊断 + +| 现象 | 首先检查 | 针对性调整 | +| ------------------------------------ | ---------------------------------------------------- | ------------------------------------------------- | +| 训练和验证 RMSE 都长期偏高 | 单位、标签、type map、覆盖范围、descriptor 容量 | 先修数据/schema，再检查 fitting net、学习率和步数 | +| 训练 RMSE 下降但验证 RMSE 停滞或上升 | 数据泄漏、重复样本、异常值、验证覆盖 | 先重新划分/去重并补代表性标注，再改模型容量 | +| RMSE 良好但 model deviation 偏高 | 未见过的体系/构型或条件、初始结构 | 定向扩展采样并标注候选区间 | +| candidate 数为零 | 信任阈值、MD 稳定性、轨迹频率、模型多样性 | 先检查阈值和采样设置，不要直接改训练 | +| candidate 过多或 FP 失败增加 | 阈值、MD 不稳定、结构损坏、FP 收敛 | 修复输入/采样，不要把失败标签加入训练 | +| 阶段不推进或任务数不对 | record 状态、计划长度、sys_idx、生成输入、提交元数据 | 对齐状态与实际任务，禁止盲目重启 | +| 生效步数与请求步数不同 | 生成输入中的 reuse/迭代 override | 对比根配置和任务配置，有意修正 override | + +## 数据与标签检查 + +检查单位和符号、原子顺序、type map、重复样本、异常幅值以及训练/验证覆盖。每个训练帧 +必须有有限且可解析的 energy 和 force；只有在配置并验证后才纳入 virial/stress。合并前拒绝 +缺失、重复或未收敛的标签。 + +## 训练检查 + +比较训练/验证曲线、loss 分量、学习率衰减、步数预算、batch size、模型容量、ensemble seed +和 backend/version 兼容性。全局指标良好也不能掩盖某个 system 或 condition 的失败。一次只 +改变一个主要因素，并保留修改前的证据。 + +## 采样与决策检查 + +检查 MD 稳定性、温度、压力、ensemble、`nsteps`、`trj_freq`、信任区间位置及分 condition +分布。区分模型问题和覆盖不足：RMSE 良好但 deviation 偏高通常需要定向采样；已覆盖区域 +RMSE 仍高则优先检查数据或训练。 + +监控是基于证据的决策阶段，不只是状态播报。扩展采样、修改阈值、增加步数或重启前，先记录 +证据、精确参数差异、预期效果和 controller 状态；之后记录新任务身份与结果。禁止静默调参 +或重启。 diff --git a/skills/dpgen-run/references/validation-and-run.md b/skills/dpgen-run/references/validation-and-run.md index de979b839..887d38d9c 100644 --- a/skills/dpgen-run/references/validation-and-run.md +++ b/skills/dpgen-run/references/validation-and-run.md @@ -1,49 +1,44 @@ -# DP-GEN Configuration, Validation, and Execution +# DP-GEN 配置、验证与执行 -Load this reference when creating, reviewing, validating, launching, or -recovering a `dpgen run` configuration. +在创建、审阅、验证、启动或恢复 `dpgen run` 配置时加载本 reference。 -## 1. `param.json` essentials +## 1. `param.json` 要点 -Preserve or collect: +应保留或收集： -- `type_map` (ordered symbols), optional `mass_map`, `init_data_prefix`, and `init_data_sys`; -- `sys_configs_prefix` and `sys_configs` (a list of systems, each a list of structures); -- `numb_models` and complete `default_training_param.model`, `learning_rate`, `loss`, and `training`; -- `model_devi_dt`, `model_devi_skip`, trust thresholds, cleanup flag, and `model_devi_jobs`; -- `fp_style`, `fp_task_min`, `fp_task_max`, and backend-specific FP inputs. +- `type_map`（有序元素符号）、可选的 `mass_map`、`init_data_prefix` 和 `init_data_sys`； +- `sys_configs_prefix` 与 `sys_configs`（系统列表，每个系统又是结构列表）； +- `numb_models` 以及完整的 `default_training_param.model`、`learning_rate`、`loss` 和 `training`； +- `model_devi_dt`、`model_devi_skip`、信任阈值、清理标志和 `model_devi_jobs`； +- `fp_style`、`fp_task_min`、`fp_task_max` 及后端专用 FP 输入。 -DeepMD-kit 1.x uses `training.systems`; 2.x/3.x use -`training.training_data.systems`. Leave the version-appropriate value empty so -DP-GEN can fill it from `init_data_sys`. Verify `deepmd_version` and the -installed stack before using version-specific descriptors or features. +DeePMD-kit 1.x 使用 `training.systems`，2.x/3.x 使用 +`training.training_data.systems`。将对应版本的 systems 留空，由 DP-GEN 根据 +`init_data_sys` 填充。使用版本相关 descriptor 或功能前，核对 `deepmd_version` +和已安装的软件栈。 -`train_backend` accepts `tensorflow` and `pytorch`, but compatibility still -depends on the installed DeePMD-kit and descriptor. Verify the combination; -spell `se_atten_v2` exactly. Ordinary LAMMPS exploration uses the default -`model_devi_engine`; set an alternative only when verified. Current `dpgen run` -does not accept `fp_style: "none"`. +`train_backend` 支持 `tensorflow` 和 `pytorch`，但实际兼容性仍取决于已安装的 +DeePMD-kit 与 descriptor。核对组合并准确拼写 `se_atten_v2`。普通 LAMMPS 探索使用 +默认 `model_devi_engine`；替代引擎只有在验证后才能设置。当前 `dpgen run` 不接受 +`fp_style: "none"`。 -Each `init_data_sys` directory must contain `type_map.raw`, `type.raw`, and -`set.000/`; every `type_map.raw` must match `param.json.type_map` exactly. +每个 `init_data_sys` 目录必须包含 `type_map.raw`、`type.raw` 和 `set.000/`； +每个 `type_map.raw` 必须与 `param.json.type_map` 完全一致。 -## 2. `machine.json` essentials +## 2. `machine.json` 要点 -Keep separate `train`, `model_devi`, and `fp` blocks. Each block supplies a -`command`, `machine` (`batch_type`, `context_type`, `local_root`, and any -required `remote_root`/profile), and `resources` (`number_node`, CPU/GPU counts, -`group_size`, queue/partition flags, and scheduler `source_list`). See the -[repository machine schema walkthrough](../../../doc/run/example-of-machine.md) -for generic local, scheduler, and SSH shapes. +保持独立的 `train`、`model_devi` 和 `fp` block。每个 block 提供 `command`、 +`machine`（`batch_type`、`context_type`、`local_root` 以及需要时的 +`remote_root`/profile）和 `resources`（节点数、CPU/GPU 数、`group_size`、 +队列/分区参数及调度器 `source_list`）。通用 local、scheduler 和 SSH 结构见 +[machine schema 说明](../../../doc/run/example-of-machine.md)。 -The launcher shell and dispatched tasks are different environments. Scheduler -jobs must activate their own runtime through `source_list`; outer activation is -not inherited reliably. The training command must select the same backend as -`train_backend` (for DeePMD-kit 3.x PyTorch this may be `dp --pt`). Validate -commands, context types, batch aliases, roots, and resources against installed -DPDispatcher rather than normalizing by assumption. +启动 shell 与派发任务处于不同环境。调度器任务必须通过 `source_list` 激活自己的 +运行环境，外层激活不会可靠继承。训练命令必须选择与 `train_backend` 相同的后端 +（DeePMD-kit 3.x 的 PyTorch 可能需要 `dp --pt`）。应根据已安装的 DPDispatcher +验证命令、context、batch 别名、路径和资源，不要凭经验统一改写。 -## 3. Validate generated inputs before launch +## 3. 启动前验证生成输入 ```bash dpgen -h @@ -51,44 +46,35 @@ python -m json.tool param.json python -m json.tool machine.json ``` -Normalize `param.json` with `run_jdata_arginfo()` and convert `machine.json` with -`convert_mdata()`. Resolve all data/structure paths, compare type maps, check -executables and scheduler access, and verify FP inputs and cost limits. +用 `run_jdata_arginfo()` 规范化 `param.json`，用 `convert_mdata()` 转换 +`machine.json`。解析所有数据/结构路径，比较 type map，检查可执行文件和调度器权限， +并核对 FP 输入及成本上限。 -Before submission, inspect the generated `input.json` for every stage/task. Do -not infer the effective configuration from the root files: check model count, -seed, checkpoint, data systems, `sys_idx`, temperatures/pressures, and final -training steps. When reuse or iteration overrides exist, assert both the default -and effective values. +提交前逐个检查每个阶段/任务实际生成的 `input.json`。不能从根目录 JSON 推断最终配置： +核对模型数、seed、checkpoint、数据系统、`sys_idx`、温度/压力和最终训练步数。存在 +reuse 或迭代 override 时，同时断言默认值和生效值。 -Cross-check the intended iteration against `record.dpgen`, stage directories, -DPDispatcher work base/submission metadata, and logs. A stage is not complete -because a directory or job exists. +将目标迭代与 `record.dpgen`、阶段目录、DPDispatcher work base/提交元数据及日志交叉核对。 +仅有目录或 job 并不代表阶段已完成。 -## 4. Restart and recovery +## 4. 恢复与重启 -Treat a restart from `record.dpgen` as cost-bearing. Keep one controller for a -work directory. Before editing state, stop it, back up the state file, record -stage/job IDs and old/new settings, and query the scheduler. +从 `record.dpgen` 恢复属于可能产生费用的操作。同一工作目录只能保留一个 controller。 +编辑状态前先停止 controller、备份状态文件、记录阶段/job ID 及新旧设置，并查询调度器。 -A changed active-stage command, context, remote root, queue, resources, -`group_size`, flags, activation, task list, or common files can create a new -DPDispatcher submission identity. It will not import completion state from the -old identity. Distinguish recovery of the old submission from an intentional -rerun with a new remote root; never resubmit solely because a connection timed -out. +活动阶段的 command、context、remote root、队列、资源、`group_size`、flags、激活方式、 +任务列表或 common files 变化，都可能产生新的 DPDispatcher submission identity。新 identity +不会导入旧 identity 的完成状态。要区分旧提交恢复和使用新 remote root 的有意重跑；连接超时 +本身不是重新提交理由。 -Positive recovery evidence is the old submission hash, unchanged job IDs, and a -recovery message; absence of a new-submission message is required. -`record.dpgen` is a workflow pointer, not a scheduler completion record. -Apply later-iteration machine changes only after the current cost-bearing stage -advances. +恢复的正证据包括旧 submission hash、未变化的 job ID 和恢复日志；同时不能出现新的提交日志。 +`record.dpgen` 是工作流指针，不是调度器完成记录。当前计费阶段推进后，才能应用后续迭代的 +machine 修改。 -## 5. Confirm and inspect +## 5. 确认与检查 -Show the exact files, validation summary, unresolved risks, and -`dpgen run param.json machine.json`. Execute only after explicit confirmation. -After launch, inspect `iter.*`, stage logs, failure/pending states, selected and -labeled counts, and [monitoring outputs](monitoring.md). +展示文件、验证摘要、未解决风险和 `dpgen run param.json machine.json`。只有明确确认后才执行。 +启动后检查 `iter.*`、阶段日志、失败/等待状态、选中和标注数量，以及 +[监控输出](monitoring.md)。 -External references: https://docs.deepmodeling.com/projects/dpgen/en/latest/run/index.html +外部参考：https://docs.deepmodeling.com/projects/dpgen/en/latest/run/index.html diff --git a/skills/dpgen-run/references/workflow.md b/skills/dpgen-run/references/workflow.md index c4def4c38..de38b78ea 100644 --- a/skills/dpgen-run/references/workflow.md +++ b/skills/dpgen-run/references/workflow.md @@ -1,36 +1,34 @@ -# DP-GEN Run Workflow Guidance +# DP-GEN Run 工作流说明 -Load this reference for scope, preparation policy, project layout, or reporting. +在需要了解范围、准备策略、项目布局或结果汇报时加载本 reference。 -## Loop +## 循环 -`dpgen run` repeats five activities: +`dpgen run` 会重复以下五个活动： -1. train an ensemble of Deep Potential models; -1. explore configurations, normally with LAMMPS MD; -1. select structures in the model-deviation candidate window; -1. label selected structures with the configured first-principles backend; -1. add labels to the next training iteration. +1. 训练一组 Deep Potential 模型； +1. 通常使用 LAMMPS MD 探索构型空间； +1. 选取处于 model-deviation 候选区间的构型； +1. 使用配置的第一性原理后端标注选中构型； +1. 将标注加入下一轮训练。 -Use `param.json` for scientific/workflow settings, `machine.json` for commands, -contexts, and resources, and `dpgen run param.json machine.json` as the launcher. -Preparation needs Python and DP-GEN. Execution also needs DeePMD-kit, the -exploration engine, the selected FP software, and any scheduler runtime. +`param.json` 保存科学/工作流设置，`machine.json` 保存命令、上下文和资源， +启动命令为 `dpgen run param.json machine.json`。准备阶段需要 Python 和 DP-GEN； +执行阶段还需要 DeePMD-kit、探索引擎、选定的 FP 软件和调度器运行环境。 -## Preparation policy +## 准备策略 -Inspect existing configuration files, training inputs, dataset metadata, and -machine templates first. Ask only for values that cannot be discovered. Patch -working files instead of rebuilding them. +先检查已有配置、训练输入、数据集元数据和 machine 模板。只询问无法发现的值， +修改现有文件而不是无必要地重建。 -Preserve descriptor and fitting settings, training and FP backends, thresholds, -`type_map` ordering, `model_devi_jobs`, ensemble size, temperatures, pressures, -and MD ensembles. Explain concerns and request direction before changing them. +保留 descriptor 与 fitting 设置、训练和 FP 后端、阈值、`type_map` 顺序、 +`model_devi_jobs`、ensemble 数量、温度、压力和 MD ensemble。发现问题时先说明并 +请求方向，不要静默修改。 -Reuse known activation commands, executables, queues, partitions, accounts, -paths, and scheduler flags exactly. Never guess site policy or conda/module setup. +原有的激活命令、可执行文件、队列、分区、账号、路径和调度器参数必须原样复用。 +不得猜测站点策略、conda 环境或 module 配置。 -## Generic layout +## 通用布局 ```text project/ @@ -38,30 +36,27 @@ project/ |-- machine.json |-- init_data/ | `-- raw_xxx/ # DeepMD NumPy system(s) -|-- assets/ # structures and runtime support files -`-- iter.*/ # created by DP-GEN +|-- assets/ # 构型和运行所需支持文件 +`-- iter.*/ # 由 DP-GEN 创建 ``` -Keep repeated experiments in separate, clearly named directories derived from a -reviewed base configuration. Backend-specific files belong under `assets/` or -the path explicitly required by the selected backend. +重复实验应放在独立且命名清晰的目录中，并从审阅过的基础配置派生。后端专用文件 +放在 `assets/` 或选定后端明确要求的路径下。 -## Reporting contract +## 汇报契约 -Before execution, report absolute JSON paths, the exact command, validation -results, unresolved inputs, and cost-bearing stages. Execute only after the user -confirms that exact validated command. +执行前汇报两个 JSON 的绝对路径、精确命令、验证结果、未解决输入和计费阶段。只有用户 +确认该精确命令后才执行。 -After execution, report the current iteration and stage status, failed/pending -tasks, main logs and outputs, candidate and labeled counts, accuracy trends, and -the next files to inspect. Use [monitoring guidance](monitoring.md) when an -iteration stalls or accuracy fails to improve. +执行后汇报当前迭代和阶段状态、失败/等待任务、主要日志和输出、候选与标注数量、 +准确率趋势以及下一步要检查的文件。迭代停滞或准确率不提升时，使用 +[监控说明](monitoring.md)。 -## Guardrails +## 防护规则 -- Never run before both JSON files exist and pass validation. -- Keep `type_map` consistent through data, training, and labeling. -- Do not overwrite working templates blindly or assume outer activation reaches jobs. -- Stop rather than guess a missing scientific or site-specific value. +- 两个 JSON 都存在且通过验证前不得运行。 +- 数据、训练和标注过程中的 `type_map` 必须一致。 +- 不得盲目覆盖可用模板，也不能假设外层激活会传递到任务环境。 +- 无法发现缺失的科学或站点值时，停止而不是猜测。 -Official overview: https://docs.deepmodeling.com/projects/dpgen/en/latest/run/index.html +官方概览：https://docs.deepmodeling.com/projects/dpgen/en/latest/run/index.html diff --git a/tests/test_dpgen_run_skill.py b/tests/test_dpgen_run_skill.py index d1dc9df55..dea4c4c4f 100644 --- a/tests/test_dpgen_run_skill.py +++ b/tests/test_dpgen_run_skill.py @@ -65,13 +65,12 @@ def test_reference_files_are_small_and_monitoring_is_actionable(self): .lower() ) for term in ( - "data", - "training", - "sampling", - "labeling", + "数据", + "训练", + "采样", + "标注", "rmse", "model deviation", - "labeling", "fp failure", "system", "condition", @@ -95,7 +94,7 @@ def test_reference_files_are_small_and_monitoring_is_actionable(self): "input.json", "submission identity", "sys_idx", - "generated", + "生成", ): self.assertIn(term, validation) From c7820464737115999bede4c9226ab48780d82384 Mon Sep 17 00:00:00 2001 From: SchrodingersCattt Date: Fri, 4 Sep 2026 04:46:29 +0000 Subject: [PATCH 21/23] Revert "docs(skills): translate dpgen-run docs to Chinese" This reverts commit f0c7b33fa963aa043c6231a1c259e3ec8a5eb679. --- skills/dpgen-run/SKILL.md | 44 +++---- skills/dpgen-run/references/monitoring.md | 98 +++++++------- .../references/validation-and-run.md | 120 ++++++++++-------- skills/dpgen-run/references/workflow.md | 79 ++++++------ tests/test_dpgen_run_skill.py | 11 +- 5 files changed, 191 insertions(+), 161 deletions(-) diff --git a/skills/dpgen-run/SKILL.md b/skills/dpgen-run/SKILL.md index 404634af8..9bde72d27 100644 --- a/skills/dpgen-run/SKILL.md +++ b/skills/dpgen-run/SKILL.md @@ -1,40 +1,40 @@ --- name: dpgen-run -description: 使用 param.json 和 machine.json 准备、解释、验证并运行 DP-GEN 并行学习流程。 +description: Prepare, explain, validate, and run DP-GEN concurrent-learning workflows using param.json and machine.json. license: LGPL-3.0-or-later metadata: author: MatMaster - version: 0.4.0 + version: 0.3.0 repository: https://github.com/deepmodeling/dpgen --- # DP-GEN Run -用于迭代训练、探索、标注和再训练流程。 +Use this skill for the iterative train, explore, label, and retrain workflow. -## 使用契约 +## Contract -- 科学参数放在 `param.json`，执行参数放在 `machine.json`。 -- 精确运行命令为 `dpgen run param.json machine.json`。 -- 默认只准备和验证；只有用户明确确认后才执行。 +- Keep scientific settings in `param.json` and execution settings in `machine.json`. +- Run exactly `dpgen run param.json machine.json`. +- Prepare and validate by default; execute only after explicit confirmation. -## 工作流程 +## Workflow -1. 先检查已有数据和配置，再询问缺失输入。 -1. 修改现有可用文件，并保留科学及站点选择。 -1. 只加载当前任务需要的 reference。 -1. 验证 schema、生成的任务输入、阶段状态和监控信号。 -1. 汇报路径、命令、分体系证据、风险和后续检查。 +1. Inspect existing data and configurations before asking for inputs. +1. Patch working files and preserve scientific/site-specific choices. +1. Load only the references needed for the task. +1. Validate schemas, generated task inputs, stage state, and monitoring signals. +1. Report paths, command, per-system evidence, risks, and next checks. -## Reference +## References -- [工作流说明](references/workflow.md)：范围、布局、策略和汇报。 -- [配置、验证与执行](references/validation-and-run.md)：两个 JSON、生成输入检查及安全启动/恢复。 -- [监控与排障](references/monitoring.md)：准确率趋势、阶段进度和证据化诊断。 +- [Workflow guidance](references/workflow.md): scope, layout, policy, and reporting. +- [Configuration, validation, and execution](references/validation-and-run.md): both JSON files, generated-input checks, and safe launch/restart checks. +- [Monitoring and troubleshooting](references/monitoring.md): accuracy trends, stage progress, and evidence-based diagnosis. -## 防护规则 +## Guardrails -- 不得合并两个 JSON，也不得臆造路径、队列、模块或命令。 -- 保留 descriptor、backend、阈值、计划、ensemble 和 `type_map` 选择。 -- 必须准确写成 `se_atten_v2`；当前 `dpgen run` 不接受 `fp_style: "none"`。 -- 无法安全发现必需的科学或执行输入时，停止并询问。 +- Never merge the JSON files or invent paths, queues, modules, or commands. +- Preserve descriptor, backend, thresholds, schedules, ensemble, and `type_map` choices. +- Spell `se_atten_v2` exactly; `fp_style: "none"` is invalid for current `dpgen run`. +- Stop and ask when required scientific or execution inputs are undiscoverable. diff --git a/skills/dpgen-run/references/monitoring.md b/skills/dpgen-run/references/monitoring.md index a7f9385bd..2ebb2a630 100644 --- a/skills/dpgen-run/references/monitoring.md +++ b/skills/dpgen-run/references/monitoring.md @@ -1,44 +1,54 @@ -# DP-GEN 监控与排障 - -在每轮迭代后，或准确率/阶段进度停滞时加载本 reference。“准确率”指能量/力的训练与 -验证 RMSE；model deviation 和 FP failure ratio 是独立信号。 - -## 证据循环 - -每轮记录参数/配置身份、系统及温度/压力条件、训练/验证 RMSE、model-deviation 的 -accurate/candidate/failed 数量、FP 成功/失败数量、当前阶段，以及新标注是否进入下一轮 -训练输入。先汇报分 system、分 condition 结果，再给全局汇总。 - -## 按顺序诊断 - -| 现象 | 首先检查 | 针对性调整 | -| ------------------------------------ | ---------------------------------------------------- | ------------------------------------------------- | -| 训练和验证 RMSE 都长期偏高 | 单位、标签、type map、覆盖范围、descriptor 容量 | 先修数据/schema，再检查 fitting net、学习率和步数 | -| 训练 RMSE 下降但验证 RMSE 停滞或上升 | 数据泄漏、重复样本、异常值、验证覆盖 | 先重新划分/去重并补代表性标注，再改模型容量 | -| RMSE 良好但 model deviation 偏高 | 未见过的体系/构型或条件、初始结构 | 定向扩展采样并标注候选区间 | -| candidate 数为零 | 信任阈值、MD 稳定性、轨迹频率、模型多样性 | 先检查阈值和采样设置，不要直接改训练 | -| candidate 过多或 FP 失败增加 | 阈值、MD 不稳定、结构损坏、FP 收敛 | 修复输入/采样，不要把失败标签加入训练 | -| 阶段不推进或任务数不对 | record 状态、计划长度、sys_idx、生成输入、提交元数据 | 对齐状态与实际任务，禁止盲目重启 | -| 生效步数与请求步数不同 | 生成输入中的 reuse/迭代 override | 对比根配置和任务配置，有意修正 override | - -## 数据与标签检查 - -检查单位和符号、原子顺序、type map、重复样本、异常幅值以及训练/验证覆盖。每个训练帧 -必须有有限且可解析的 energy 和 force；只有在配置并验证后才纳入 virial/stress。合并前拒绝 -缺失、重复或未收敛的标签。 - -## 训练检查 - -比较训练/验证曲线、loss 分量、学习率衰减、步数预算、batch size、模型容量、ensemble seed -和 backend/version 兼容性。全局指标良好也不能掩盖某个 system 或 condition 的失败。一次只 -改变一个主要因素，并保留修改前的证据。 - -## 采样与决策检查 - -检查 MD 稳定性、温度、压力、ensemble、`nsteps`、`trj_freq`、信任区间位置及分 condition -分布。区分模型问题和覆盖不足：RMSE 良好但 deviation 偏高通常需要定向采样；已覆盖区域 -RMSE 仍高则优先检查数据或训练。 - -监控是基于证据的决策阶段，不只是状态播报。扩展采样、修改阈值、增加步数或重启前，先记录 -证据、精确参数差异、预期效果和 controller 状态；之后记录新任务身份与结果。禁止静默调参 -或重启。 +# DP-GEN Monitoring and Troubleshooting + +Load this reference after each iteration or whenever accuracy or stage progress +stalls. Here "accuracy" means energy/force train and validation RMSE; model +deviation and FP failure ratio are separate signals. + +## Evidence loop + +For each iteration, record the parameter/configuration identity, systems and +temperature/pressure conditions, train/validation RMSE, model-deviation +accurate/candidate/failed counts, FP success/failure counts, current stage, and +whether new labels reached the next training input. Report per-system and +per-condition results before any global aggregate. + +## Diagnose in this order + +| Symptom | First checks | Focused adjustment | +| ---------------------------------------------------- | ---------------------------------------------------------------------------- | --------------------------------------------------------------------------- | +| Train and validation RMSE both stay high | Units, labels, type map, coverage, descriptor capacity | Fix data/schema first; then review fitting net, learning rate, and steps | +| Train RMSE falls but validation RMSE stalls or rises | Split leakage, duplicates, outliers, validation coverage | Re-split or deduplicate; add representative labels before changing capacity | +| RMSE is good but model deviation is high | Unseen systems/geometries or conditions, initial structures | Broaden targeted sampling and labeling the candidate window | +| Candidate count is zero | Trust thresholds, MD stability, trajectory frequency, model diversity | Check thresholds and sampling settings before changing training | +| Candidate count is excessive or FP failures rise | Thresholds, unstable MD, malformed structures, FP convergence | Repair inputs/sampling; do not add failed labels | +| Stage does not advance or task count is wrong | record state, schedule length, sys_idx, generated input, submission metadata | Reconcile state and actual tasks; do not restart blindly | +| Effective steps differ from requested steps | Reuse/iteration override in generated input | Compare root and task-level values; fix the override deliberately | + +## Data and label checks + +Check units and signs, atom ordering, type maps, duplicates, abnormal +magnitudes, and train/validation coverage. Every training frame must have +finite, parseable energy and force; include virial/stress only when configured +and verified. Reject incomplete, duplicated, or unconverged labels before +merging them into training data. + +## Training checks + +Compare train and validation curves, loss components, learning-rate decay, step +budget, batch size, model capacity, ensemble seeds, and backend/version +compatibility. A good global metric must not hide a failing system or condition. +Change one major factor at a time and retain the previous evidence. + +## Sampling and decision checks + +Inspect MD stability, temperatures, pressures, ensembles, `nsteps`, `trj_freq`, +trust-window placement, and per-condition distributions. Distinguish a model +problem from missing coverage: good RMSE with high deviation usually calls for +targeted sampling, while high RMSE on covered data calls for data or training +review. + +Monitoring is an evidence-based decision stage, not only status reporting. +Before extending sampling, changing thresholds, increasing steps, or restarting, +record the evidence, exact parameter delta, expected effect, and controller +state. Afterward record the new task identity and outcome; never retune or +restart silently. diff --git a/skills/dpgen-run/references/validation-and-run.md b/skills/dpgen-run/references/validation-and-run.md index 887d38d9c..de979b839 100644 --- a/skills/dpgen-run/references/validation-and-run.md +++ b/skills/dpgen-run/references/validation-and-run.md @@ -1,44 +1,49 @@ -# DP-GEN 配置、验证与执行 +# DP-GEN Configuration, Validation, and Execution -在创建、审阅、验证、启动或恢复 `dpgen run` 配置时加载本 reference。 +Load this reference when creating, reviewing, validating, launching, or +recovering a `dpgen run` configuration. -## 1. `param.json` 要点 +## 1. `param.json` essentials -应保留或收集： +Preserve or collect: -- `type_map`（有序元素符号）、可选的 `mass_map`、`init_data_prefix` 和 `init_data_sys`； -- `sys_configs_prefix` 与 `sys_configs`（系统列表，每个系统又是结构列表）； -- `numb_models` 以及完整的 `default_training_param.model`、`learning_rate`、`loss` 和 `training`； -- `model_devi_dt`、`model_devi_skip`、信任阈值、清理标志和 `model_devi_jobs`； -- `fp_style`、`fp_task_min`、`fp_task_max` 及后端专用 FP 输入。 +- `type_map` (ordered symbols), optional `mass_map`, `init_data_prefix`, and `init_data_sys`; +- `sys_configs_prefix` and `sys_configs` (a list of systems, each a list of structures); +- `numb_models` and complete `default_training_param.model`, `learning_rate`, `loss`, and `training`; +- `model_devi_dt`, `model_devi_skip`, trust thresholds, cleanup flag, and `model_devi_jobs`; +- `fp_style`, `fp_task_min`, `fp_task_max`, and backend-specific FP inputs. -DeePMD-kit 1.x 使用 `training.systems`，2.x/3.x 使用 -`training.training_data.systems`。将对应版本的 systems 留空，由 DP-GEN 根据 -`init_data_sys` 填充。使用版本相关 descriptor 或功能前，核对 `deepmd_version` -和已安装的软件栈。 +DeepMD-kit 1.x uses `training.systems`; 2.x/3.x use +`training.training_data.systems`. Leave the version-appropriate value empty so +DP-GEN can fill it from `init_data_sys`. Verify `deepmd_version` and the +installed stack before using version-specific descriptors or features. -`train_backend` 支持 `tensorflow` 和 `pytorch`，但实际兼容性仍取决于已安装的 -DeePMD-kit 与 descriptor。核对组合并准确拼写 `se_atten_v2`。普通 LAMMPS 探索使用 -默认 `model_devi_engine`；替代引擎只有在验证后才能设置。当前 `dpgen run` 不接受 -`fp_style: "none"`。 +`train_backend` accepts `tensorflow` and `pytorch`, but compatibility still +depends on the installed DeePMD-kit and descriptor. Verify the combination; +spell `se_atten_v2` exactly. Ordinary LAMMPS exploration uses the default +`model_devi_engine`; set an alternative only when verified. Current `dpgen run` +does not accept `fp_style: "none"`. -每个 `init_data_sys` 目录必须包含 `type_map.raw`、`type.raw` 和 `set.000/`； -每个 `type_map.raw` 必须与 `param.json.type_map` 完全一致。 +Each `init_data_sys` directory must contain `type_map.raw`, `type.raw`, and +`set.000/`; every `type_map.raw` must match `param.json.type_map` exactly. -## 2. `machine.json` 要点 +## 2. `machine.json` essentials -保持独立的 `train`、`model_devi` 和 `fp` block。每个 block 提供 `command`、 -`machine`（`batch_type`、`context_type`、`local_root` 以及需要时的 -`remote_root`/profile）和 `resources`（节点数、CPU/GPU 数、`group_size`、 -队列/分区参数及调度器 `source_list`）。通用 local、scheduler 和 SSH 结构见 -[machine schema 说明](../../../doc/run/example-of-machine.md)。 +Keep separate `train`, `model_devi`, and `fp` blocks. Each block supplies a +`command`, `machine` (`batch_type`, `context_type`, `local_root`, and any +required `remote_root`/profile), and `resources` (`number_node`, CPU/GPU counts, +`group_size`, queue/partition flags, and scheduler `source_list`). See the +[repository machine schema walkthrough](../../../doc/run/example-of-machine.md) +for generic local, scheduler, and SSH shapes. -启动 shell 与派发任务处于不同环境。调度器任务必须通过 `source_list` 激活自己的 -运行环境，外层激活不会可靠继承。训练命令必须选择与 `train_backend` 相同的后端 -（DeePMD-kit 3.x 的 PyTorch 可能需要 `dp --pt`）。应根据已安装的 DPDispatcher -验证命令、context、batch 别名、路径和资源，不要凭经验统一改写。 +The launcher shell and dispatched tasks are different environments. Scheduler +jobs must activate their own runtime through `source_list`; outer activation is +not inherited reliably. The training command must select the same backend as +`train_backend` (for DeePMD-kit 3.x PyTorch this may be `dp --pt`). Validate +commands, context types, batch aliases, roots, and resources against installed +DPDispatcher rather than normalizing by assumption. -## 3. 启动前验证生成输入 +## 3. Validate generated inputs before launch ```bash dpgen -h @@ -46,35 +51,44 @@ python -m json.tool param.json python -m json.tool machine.json ``` -用 `run_jdata_arginfo()` 规范化 `param.json`，用 `convert_mdata()` 转换 -`machine.json`。解析所有数据/结构路径，比较 type map，检查可执行文件和调度器权限， -并核对 FP 输入及成本上限。 +Normalize `param.json` with `run_jdata_arginfo()` and convert `machine.json` with +`convert_mdata()`. Resolve all data/structure paths, compare type maps, check +executables and scheduler access, and verify FP inputs and cost limits. -提交前逐个检查每个阶段/任务实际生成的 `input.json`。不能从根目录 JSON 推断最终配置： -核对模型数、seed、checkpoint、数据系统、`sys_idx`、温度/压力和最终训练步数。存在 -reuse 或迭代 override 时，同时断言默认值和生效值。 +Before submission, inspect the generated `input.json` for every stage/task. Do +not infer the effective configuration from the root files: check model count, +seed, checkpoint, data systems, `sys_idx`, temperatures/pressures, and final +training steps. When reuse or iteration overrides exist, assert both the default +and effective values. -将目标迭代与 `record.dpgen`、阶段目录、DPDispatcher work base/提交元数据及日志交叉核对。 -仅有目录或 job 并不代表阶段已完成。 +Cross-check the intended iteration against `record.dpgen`, stage directories, +DPDispatcher work base/submission metadata, and logs. A stage is not complete +because a directory or job exists. -## 4. 恢复与重启 +## 4. Restart and recovery -从 `record.dpgen` 恢复属于可能产生费用的操作。同一工作目录只能保留一个 controller。 -编辑状态前先停止 controller、备份状态文件、记录阶段/job ID 及新旧设置，并查询调度器。 +Treat a restart from `record.dpgen` as cost-bearing. Keep one controller for a +work directory. Before editing state, stop it, back up the state file, record +stage/job IDs and old/new settings, and query the scheduler. -活动阶段的 command、context、remote root、队列、资源、`group_size`、flags、激活方式、 -任务列表或 common files 变化，都可能产生新的 DPDispatcher submission identity。新 identity -不会导入旧 identity 的完成状态。要区分旧提交恢复和使用新 remote root 的有意重跑；连接超时 -本身不是重新提交理由。 +A changed active-stage command, context, remote root, queue, resources, +`group_size`, flags, activation, task list, or common files can create a new +DPDispatcher submission identity. It will not import completion state from the +old identity. Distinguish recovery of the old submission from an intentional +rerun with a new remote root; never resubmit solely because a connection timed +out. -恢复的正证据包括旧 submission hash、未变化的 job ID 和恢复日志；同时不能出现新的提交日志。 -`record.dpgen` 是工作流指针，不是调度器完成记录。当前计费阶段推进后，才能应用后续迭代的 -machine 修改。 +Positive recovery evidence is the old submission hash, unchanged job IDs, and a +recovery message; absence of a new-submission message is required. +`record.dpgen` is a workflow pointer, not a scheduler completion record. +Apply later-iteration machine changes only after the current cost-bearing stage +advances. -## 5. 确认与检查 +## 5. Confirm and inspect -展示文件、验证摘要、未解决风险和 `dpgen run param.json machine.json`。只有明确确认后才执行。 -启动后检查 `iter.*`、阶段日志、失败/等待状态、选中和标注数量，以及 -[监控输出](monitoring.md)。 +Show the exact files, validation summary, unresolved risks, and +`dpgen run param.json machine.json`. Execute only after explicit confirmation. +After launch, inspect `iter.*`, stage logs, failure/pending states, selected and +labeled counts, and [monitoring outputs](monitoring.md). -外部参考：https://docs.deepmodeling.com/projects/dpgen/en/latest/run/index.html +External references: https://docs.deepmodeling.com/projects/dpgen/en/latest/run/index.html diff --git a/skills/dpgen-run/references/workflow.md b/skills/dpgen-run/references/workflow.md index de38b78ea..c4def4c38 100644 --- a/skills/dpgen-run/references/workflow.md +++ b/skills/dpgen-run/references/workflow.md @@ -1,34 +1,36 @@ -# DP-GEN Run 工作流说明 +# DP-GEN Run Workflow Guidance -在需要了解范围、准备策略、项目布局或结果汇报时加载本 reference。 +Load this reference for scope, preparation policy, project layout, or reporting. -## 循环 +## Loop -`dpgen run` 会重复以下五个活动： +`dpgen run` repeats five activities: -1. 训练一组 Deep Potential 模型； -1. 通常使用 LAMMPS MD 探索构型空间； -1. 选取处于 model-deviation 候选区间的构型； -1. 使用配置的第一性原理后端标注选中构型； -1. 将标注加入下一轮训练。 +1. train an ensemble of Deep Potential models; +1. explore configurations, normally with LAMMPS MD; +1. select structures in the model-deviation candidate window; +1. label selected structures with the configured first-principles backend; +1. add labels to the next training iteration. -`param.json` 保存科学/工作流设置，`machine.json` 保存命令、上下文和资源， -启动命令为 `dpgen run param.json machine.json`。准备阶段需要 Python 和 DP-GEN； -执行阶段还需要 DeePMD-kit、探索引擎、选定的 FP 软件和调度器运行环境。 +Use `param.json` for scientific/workflow settings, `machine.json` for commands, +contexts, and resources, and `dpgen run param.json machine.json` as the launcher. +Preparation needs Python and DP-GEN. Execution also needs DeePMD-kit, the +exploration engine, the selected FP software, and any scheduler runtime. -## 准备策略 +## Preparation policy -先检查已有配置、训练输入、数据集元数据和 machine 模板。只询问无法发现的值， -修改现有文件而不是无必要地重建。 +Inspect existing configuration files, training inputs, dataset metadata, and +machine templates first. Ask only for values that cannot be discovered. Patch +working files instead of rebuilding them. -保留 descriptor 与 fitting 设置、训练和 FP 后端、阈值、`type_map` 顺序、 -`model_devi_jobs`、ensemble 数量、温度、压力和 MD ensemble。发现问题时先说明并 -请求方向，不要静默修改。 +Preserve descriptor and fitting settings, training and FP backends, thresholds, +`type_map` ordering, `model_devi_jobs`, ensemble size, temperatures, pressures, +and MD ensembles. Explain concerns and request direction before changing them. -原有的激活命令、可执行文件、队列、分区、账号、路径和调度器参数必须原样复用。 -不得猜测站点策略、conda 环境或 module 配置。 +Reuse known activation commands, executables, queues, partitions, accounts, +paths, and scheduler flags exactly. Never guess site policy or conda/module setup. -## 通用布局 +## Generic layout ```text project/ @@ -36,27 +38,30 @@ project/ |-- machine.json |-- init_data/ | `-- raw_xxx/ # DeepMD NumPy system(s) -|-- assets/ # 构型和运行所需支持文件 -`-- iter.*/ # 由 DP-GEN 创建 +|-- assets/ # structures and runtime support files +`-- iter.*/ # created by DP-GEN ``` -重复实验应放在独立且命名清晰的目录中，并从审阅过的基础配置派生。后端专用文件 -放在 `assets/` 或选定后端明确要求的路径下。 +Keep repeated experiments in separate, clearly named directories derived from a +reviewed base configuration. Backend-specific files belong under `assets/` or +the path explicitly required by the selected backend. -## 汇报契约 +## Reporting contract -执行前汇报两个 JSON 的绝对路径、精确命令、验证结果、未解决输入和计费阶段。只有用户 -确认该精确命令后才执行。 +Before execution, report absolute JSON paths, the exact command, validation +results, unresolved inputs, and cost-bearing stages. Execute only after the user +confirms that exact validated command. -执行后汇报当前迭代和阶段状态、失败/等待任务、主要日志和输出、候选与标注数量、 -准确率趋势以及下一步要检查的文件。迭代停滞或准确率不提升时，使用 -[监控说明](monitoring.md)。 +After execution, report the current iteration and stage status, failed/pending +tasks, main logs and outputs, candidate and labeled counts, accuracy trends, and +the next files to inspect. Use [monitoring guidance](monitoring.md) when an +iteration stalls or accuracy fails to improve. -## 防护规则 +## Guardrails -- 两个 JSON 都存在且通过验证前不得运行。 -- 数据、训练和标注过程中的 `type_map` 必须一致。 -- 不得盲目覆盖可用模板，也不能假设外层激活会传递到任务环境。 -- 无法发现缺失的科学或站点值时，停止而不是猜测。 +- Never run before both JSON files exist and pass validation. +- Keep `type_map` consistent through data, training, and labeling. +- Do not overwrite working templates blindly or assume outer activation reaches jobs. +- Stop rather than guess a missing scientific or site-specific value. -官方概览：https://docs.deepmodeling.com/projects/dpgen/en/latest/run/index.html +Official overview: https://docs.deepmodeling.com/projects/dpgen/en/latest/run/index.html diff --git a/tests/test_dpgen_run_skill.py b/tests/test_dpgen_run_skill.py index dea4c4c4f..d1dc9df55 100644 --- a/tests/test_dpgen_run_skill.py +++ b/tests/test_dpgen_run_skill.py @@ -65,12 +65,13 @@ def test_reference_files_are_small_and_monitoring_is_actionable(self): .lower() ) for term in ( - "数据", - "训练", - "采样", - "标注", + "data", + "training", + "sampling", + "labeling", "rmse", "model deviation", + "labeling", "fp failure", "system", "condition", @@ -94,7 +95,7 @@ def test_reference_files_are_small_and_monitoring_is_actionable(self): "input.json", "submission identity", "sys_idx", - "生成", + "generated", ): self.assertIn(term, validation) From c3ef378d192a0a3fcca5cdf885748652f1893431 Mon Sep 17 00:00:00 2001 From: SchrodingersCattt Date: Fri, 4 Sep 2026 07:35:29 +0000 Subject: [PATCH 22/23] fix(skills): align runtime and input validation guidance ACK: keep the training command as dp because DP-GEN appends backend flags; accept NumPy directories or HDF5 init data; qualify descriptor spelling by installed backend; remove the assets layout assumption; and add an explicit post-iteration monitoring stage. NAK: the older findings about list numbering, TensorFlow-only wording, empty systems syntax, missing confirmation, absent resources, remote_root and batch casing requirements, link containment, and weak machine-field assertions are already resolved in the current tree, so no duplicate changes are needed. --- skills/dpgen-run/SKILL.md | 2 +- skills/dpgen-run/references/validation-and-run.md | 14 ++++++++------ skills/dpgen-run/references/workflow.md | 10 ++++++++-- 3 files changed, 17 insertions(+), 9 deletions(-) diff --git a/skills/dpgen-run/SKILL.md b/skills/dpgen-run/SKILL.md index 9bde72d27..29723930a 100644 --- a/skills/dpgen-run/SKILL.md +++ b/skills/dpgen-run/SKILL.md @@ -36,5 +36,5 @@ Use this skill for the iterative train, explore, label, and retrain workflow. - Never merge the JSON files or invent paths, queues, modules, or commands. - Preserve descriptor, backend, thresholds, schedules, ensemble, and `type_map` choices. -- Spell `se_atten_v2` exactly; `fp_style: "none"` is invalid for current `dpgen run`. +- Check `se_atten_v2` against the installed backend; `fp_style: "none"` is invalid. - Stop and ask when required scientific or execution inputs are undiscoverable. diff --git a/skills/dpgen-run/references/validation-and-run.md b/skills/dpgen-run/references/validation-and-run.md index de979b839..3ff6f3e69 100644 --- a/skills/dpgen-run/references/validation-and-run.md +++ b/skills/dpgen-run/references/validation-and-run.md @@ -19,13 +19,15 @@ DP-GEN can fill it from `init_data_sys`. Verify `deepmd_version` and the installed stack before using version-specific descriptors or features. `train_backend` accepts `tensorflow` and `pytorch`, but compatibility still -depends on the installed DeePMD-kit and descriptor. Verify the combination; -spell `se_atten_v2` exactly. Ordinary LAMMPS exploration uses the default +depends on the installed DeePMD-kit and descriptor. Verify descriptor spelling +against that backend; current releases register `se_atten_v2`. +Ordinary LAMMPS exploration uses the default `model_devi_engine`; set an alternative only when verified. Current `dpgen run` does not accept `fp_style: "none"`. -Each `init_data_sys` directory must contain `type_map.raw`, `type.raw`, and -`set.000/`; every `type_map.raw` must match `param.json.type_map` exactly. +Each `init_data_sys` entry may be a DeepMD NumPy system directory or an HDF5 +file. Each NumPy directory must contain at least `type.raw` and `set.000/`. +When `type_map.raw` is present, it must match `param.json.type_map` exactly. ## 2. `machine.json` essentials @@ -38,8 +40,8 @@ for generic local, scheduler, and SSH shapes. The launcher shell and dispatched tasks are different environments. Scheduler jobs must activate their own runtime through `source_list`; outer activation is -not inherited reliably. The training command must select the same backend as -`train_backend` (for DeePMD-kit 3.x PyTorch this may be `dp --pt`). Validate +not inherited reliably. Keep the training command as `dp`; DP-GEN appends +backend flags such as `--pt` or `--jax` from `train_backend`. Validate commands, context types, batch aliases, roots, and resources against installed DPDispatcher rather than normalizing by assumption. diff --git a/skills/dpgen-run/references/workflow.md b/skills/dpgen-run/references/workflow.md index c4def4c38..d0aab464a 100644 --- a/skills/dpgen-run/references/workflow.md +++ b/skills/dpgen-run/references/workflow.md @@ -17,6 +17,12 @@ contexts, and resources, and `dpgen run param.json machine.json` as the launcher Preparation needs Python and DP-GEN. Execution also needs DeePMD-kit, the exploration engine, the selected FP software, and any scheduler runtime. +## Monitoring stage + +After each iteration, record RMSE, model-deviation, and FP outcomes by system +and condition. Use [monitoring guidance](monitoring.md) before changing settings +or restarting a stalled iteration. + ## Preparation policy Inspect existing configuration files, training inputs, dataset metadata, and @@ -38,12 +44,12 @@ project/ |-- machine.json |-- init_data/ | `-- raw_xxx/ # DeepMD NumPy system(s) -|-- assets/ # structures and runtime support files +|-- inputs/ # optional structures and backend inputs `-- iter.*/ # created by DP-GEN ``` Keep repeated experiments in separate, clearly named directories derived from a -reviewed base configuration. Backend-specific files belong under `assets/` or +reviewed base configuration. Backend-specific files belong under `inputs/` or the path explicitly required by the selected backend. ## Reporting contract From b9ad02990544acb7d2204e8f1212476e43bc882b Mon Sep 17 00:00:00 2001 From: SchrodingersCattt Date: Fri, 11 Sep 2026 10:10:54 +0800 Subject: [PATCH 23/23] docs: align dpgen run validation with loaders --- .../references/validation-and-run.md | 22 +++++++++++-------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/skills/dpgen-run/references/validation-and-run.md b/skills/dpgen-run/references/validation-and-run.md index 3ff6f3e69..7b3afe4d7 100644 --- a/skills/dpgen-run/references/validation-and-run.md +++ b/skills/dpgen-run/references/validation-and-run.md @@ -25,9 +25,13 @@ Ordinary LAMMPS exploration uses the default `model_devi_engine`; set an alternative only when verified. Current `dpgen run` does not accept `fp_style: "none"`. -Each `init_data_sys` entry may be a DeepMD NumPy system directory or an HDF5 -file. Each NumPy directory must contain at least `type.raw` and `set.000/`. -When `type_map.raw` is present, it must match `param.json.type_map` exactly. +Each `init_data_sys` entry may be a DeepMD NumPy system directory, a parent +directory containing multiple systems, or an HDF5 file. Resolve entries with +DP-GEN's `init_data_prefix` and discovery rules, then validate each NumPy leaf: +it needs `type.raw` and one loadable `set.*` directory. `type_map.raw` is +optional; when present, its symbols must be included in `param.json.type_map`. +Subsets and different orders are valid because DeePMD remaps by element name; +without it, `type.raw` must already use the configured model order. ## 2. `machine.json` essentials @@ -57,11 +61,12 @@ Normalize `param.json` with `run_jdata_arginfo()` and convert `machine.json` wit `convert_mdata()`. Resolve all data/structure paths, compare type maps, check executables and scheduler access, and verify FP inputs and cost limits. -Before submission, inspect the generated `input.json` for every stage/task. Do -not infer the effective configuration from the root files: check model count, -seed, checkpoint, data systems, `sys_idx`, temperatures/pressures, and final -training steps. When reuse or iteration overrides exist, assert both the default -and effective values. +Before submission, inspect each stage's generated backend artifact. For training, +check task `input.json` for model count, seeds, checkpoints, data systems, and +final steps; for LAMMPS exploration, check `input.lammps` and `job.json` for +`sys_idx`, temperatures/pressures, and effective MD settings; for first-principles +labeling, inspect backend inputs such as VASP `INCAR`. When reuse or iteration +overrides exist, assert both default and effective values. Cross-check the intended iteration against `record.dpgen`, stage directories, DPDispatcher work base/submission metadata, and logs. A stage is not complete @@ -92,5 +97,4 @@ Show the exact files, validation summary, unresolved risks, and `dpgen run param.json machine.json`. Execute only after explicit confirmation. After launch, inspect `iter.*`, stage logs, failure/pending states, selected and labeled counts, and [monitoring outputs](monitoring.md). - External references: https://docs.deepmodeling.com/projects/dpgen/en/latest/run/index.html