DeepMD training problem #5950

Activity

1. 1997sankha

   added

   wontfix

   on Aug 1, 2026
2. dosubot commented on Aug 1, 2026

   

   dosubotBot

   on Aug 1, 2026 – with DosuBot

   More actions

   Hi @1997sankha, good to see the errors have come down significantly from your earlier training run. However, your force RMSE values (0.4–1.0 eV/Å) are still quite high and are fluctuating a lot at the end of training, which explains the poor DFT-vs-predicted agreement in your plots. Here are several things to look at:

   1. Learning rate is essentially zero — training has stalled.
   By step 1.5M your learning rate is ~1e-8, which means the model is barely updating. The large step-to-step fluctuations in RMSE (e.g., 0.44 → 1.23 → 0.59) at a near-zero learning rate suggest the model hasn't converged to a good minimum. Consider increasing numb_steps (4M–8M+) and/or adjusting decay_steps so the learning rate doesn't decay so fast. A common recommendation is start_lr=1e-3 with decay_steps tuned so you still have a meaningful learning rate during later training [1].

   2. Check data quality — this was the root cause previously.
   In your earlier issue on this same thread, you identified that your force data had values in the range of −222 to 222 eV/Å and near-zero coordinates. Even though you've rebuilt the dataset, make sure you've removed all outlier frames. Scan your .npy force files and remove any frame where forces exceed ~10–20 eV/Å — even a few bad frames can prevent convergence [1].

   3. Bootstrapping can introduce problems.
   You mention increasing data by bootstrapping. Duplicating data does not add new information — it only changes sampling weights. If some of the bootstrapped data contains outliers, their influence is amplified. Instead of bootstrapping, consider using auto_prob_style to control system sampling weights [2].

   4. Sampling imbalance across 56 training systems.
   With 56 training systems of potentially different sizes, the default prob_sys_size weighting can cause the model to focus on larger systems and neglect smaller ones. Check that frame counts are roughly comparable across systems, or use "auto_prob": "prob_uniform" to weight all systems equally [2].

   5. Loss prefactors look reasonable but consider tuning.
   Your start_pref_f=1000, limit_pref_f=1 is standard. But with force errors this high, try also setting start_pref_e slightly higher (e.g., 0.1–1.0) and check if equal prefactors (start_pref_f=1, limit_pref_f=1, start_pref_e=1, limit_pref_e=1) work better as a diagnostic [1].

   6. Verify units and data format.
   Double-check that your converted data (from QE CPMD) has energies in eV, forces in eV/Å, and coordinates in Å. Use dpdata to convert and verify: d = dpdata.LabeledSystem('your_qe_output', fmt='qe/pw/scf') and inspect d['forces'] and d['energies'] ranges.

   Recommended next steps:

   1. Scan all training data for force outliers (any |force component| > 15 eV/Å) and remove those frames
   2. Increase numb_steps to 4M+ or increase start_lr to 1e-3
   3. Remove bootstrapped duplicates and use auto_prob for system weighting instead
   4. Try training on a small, clean subset first to confirm the model can converge before scaling up

   To reply, just mention @dosu.

   ---

   Docs are dead. Just use Dosu.

   
3. 1997sankha commented on Aug 1, 2026

   

   1997sankha

   on Aug 1, 2026

   Author

   More actions

   Thanks @dosu. I am looking into it
4. dosubot commented on Aug 1, 2026

   

   dosubotBot

   on Aug 1, 2026 – with DosuBot

   More actions

   Sounds good! Feel free to share your findings or any updated results — happy to help you troubleshoot further.

   To reply, just mention @dosu.

   ---

   Share context across your team and agents. Try Dosu.

   
5. 1997sankha commented on Aug 1, 2026

   

   1997sankha

   on Aug 1, 2026

   Last edited by 1997sankha

   Author

   More actions

   @dosu, I want to bring to your notice that no outliers are there, every folder has 10,000 frames, out of which 8000 are for training and 2000 for validation, units are ok, after bootstraping no outkliers are there , then where should I change?
6. njzjz-bot commented on Aug 10, 2026

   

   njzjz-bot

   on Aug 10, 2026

   Contributor

   More actions

   Thanks for confirming that the units and obvious bad frames have been checked. In that case, I would not change rcut, the network sizes, or the loss prefactors first. There are two more fundamental issues to check.

   First, numb_btch: 1 means that each validation value in lcurve.out is computed from only one randomly sampled validation batch, not from all validation frames. With batch_size: "auto", a system containing more than 32 atoms normally has a batch size of one frame. This is enough to explain the large point-to-point fluctuations at the end of the curve, even though the learning rate is almost unchanged. You can use, for example, "numb_btch": 100 for a less noisy training monitor, but the reliable evaluation should be performed on the complete held-out data:

   dp freeze -o frozen_model.pb
   dp test -m frozen_model.pb --valid-data input.json -n 0 -d valid

   In DeePMD-kit 3.1.1, -n 0 tests all frames. Plot valid.e_peratom.out columns 1 and 2 for reference versus predicted energy, because rmse_e in lcurve.out is also normalized per atom. For forces, valid.f.out contains reference fx fy fz in columns 1--3 and predictions in columns 4--6.

   Second, please check when bootstrapping was performed. If data_0 through data_4 were bootstrapped independently from the same original trajectory and then some bootstrap replicas were assigned to training and another to validation, identical original frames will occur in both sets. Having 8,000/2,000 frames does not make the validation set independent. The correct order is:

   1. Deduplicate the original configurations.
   2. Split the original frame indices into training and validation sets. For an MD trajectory, use separated trajectory blocks or independent trajectories rather than randomly interleaved neighboring frames.
   3. Keep validation frames untouched. If training frames are resampled, resample only after the split; note that duplication changes weights but adds no new configurations.

   Also, DeePMD does not automatically split a listed system directory 80/20. Every frame under a path in training_data.systems is training data, and every frame under a path in validation_data.systems is validation data. The two frame sets therefore need to be physically separated into the corresponding system directories.

   If the full dp test result is still poor after making an independent split, do a small overfitting test: train on about 100 unique frames from one system and test those same frames. If the model cannot fit that tiny set, check the conversion rather than increasing numb_steps: atom ordering must match among coord.npy, force.npy, and type.raw; the QE force sign must be correct; and all 14 datasets must use exactly the same pseudopotentials, spin treatment, cutoffs, smearing, k-point settings, and energy convention. If the tiny set fits but held-out data does not, the problem is dataset coverage/generalization.

   Reaching lr = 1e-8 at the final configured step is expected because it is your stop_lr; by itself it does not show that extending training to several million steps will solve the discrepancy. Please share the complete weighted-average output from the dp test command and explain whether the split occurred before or after bootstrapping; those two pieces should identify the next change.

   Coding agent: Codex
   Codex version: codex-cli 0.144.6
   Model: gpt-5.6-sol
   Reasoning effort: xhigh
7. njzjz-bot commented on Aug 10, 2026

   

   njzjz-bot

   on Aug 10, 2026

   Contributor

   More actions

   One important correction/addition to my previous comment: after looking closely at the two plots, the data and evaluation pipeline should be checked before changing any training hyperparameters.

   The plots show more than ordinary validation noise:

   • In the force plot, the reference values span approximately -2.5 to 2.7, while the predictions are confined to approximately -0.17 to 0.16. The predicted forces have nearly collapsed to zero. The force unit should also be eV/Å, not eV as shown on the axes.
   • In the energy plot, the reference energies span approximately 0.9 eV, while the predictions vary by only about 0.008 eV and are nearly constant.
   • Both plots contain pronounced horizontal and vertical bands. In particular, apparently identical reference values are paired with many different predictions, and vice versa. This is suspicious for frame/index misalignment or an error in constructing the plots; it is not something that increasing numb_steps would fix.

   Because the dataset was enlarged by bootstrapping, please check the bootstrapping implementation first. A single frame-index array must be generated and applied unchanged to every frame-dependent array:

   idx = rng.choice(nframes, size=n_bootstrap, replace=True)
   coord = coord[idx]
   box = box[idx]
   energy = energy[idx]
   force = force[idx]
   # Apply the same idx to virial and other frame-dependent arrays if present.

   If coord.npy, energy.npy, force.npy, or box.npy were resampled or shuffled independently, the coordinates and labels are no longer paired, even though every file still has the same number of frames and contains no numerical outliers. Concatenation and train/validation splitting must likewise preserve the same frame ordering across all arrays.

   Please also check the plotting code. Generate one fresh test output and plot columns from the same file/run: use reference and prediction columns 1 and 2 from valid.e.out (or valid.e_peratom.out consistently), and columns 1--3 versus 4--6 from valid.f.out. Do not combine a reference array from one system/run with predictions from another.

   A useful integrity check for bootstrapped data is that two exactly identical coordinate/box rows must have exactly identical energy and force labels. If identical configurations have different labels, the arrays have been misaligned. Please share the bootstrapping and plotting code; those are now the most useful pieces of information. I would not continue training or tune the network until this check passes.

   Coding agent: Codex
   Codex version: codex-cli 0.144.6
   Model: gpt-5.6-sol
   Reasoning effort: xhigh
8. 1997sankha commented on Aug 15, 2026

   

   1997sankha

   on Aug 15, 2026

   Author

   More actions

   Thank you.
   Now, suppose I have one file in which 1000 frames are there. I separate it to 250 each into 4 data set . among which I use 3 as training and 1 as validation. This is for some datasets. now question is, is it right to do?
   and what is mean by
   For an MD trajectory, use separated trajectory blocks or independent trajectories rather than randomly interleaved neighboring frames. how to do?
9. 1997sankha commented on Sep 12, 2026

   

   1997sankha

   on Sep 12, 2026

   Author

   More actions

   @[njzjz-bot],
   after running with the code I get one combined force plot for training and validation but different energy plots as different configurations are there. I am uploading my plots and input.json file.
   suggest further change

   

   input2.json
10. 1997sankha commented on Sep 13, 2026

    

    1997sankha

    on Sep 13, 2026

    Author

    More actions

    pls respond @dosu
11. dosubot commented on Sep 13, 2026

    

    dosubotBot

    on Sep 13, 2026 – with DosuBot

    More actions

    Your organization has reached the limit of 200 credits per month. Your usage will reset on 2026-10-01. If you'd like to use Dosu before then, upgrade to a pro account.

    To reply, just mention @dosu.

    ---

    Docs are dead. Just use Dosu.

    
12. 1997sankha commented on Sep 13, 2026

    

    1997sankha

    on Sep 13, 2026

    Author

    More actions

    pls respond njzjz-bot
13. njzjz-bot commented on Sep 21, 2026

    

    njzjz-bot

    on Sep 21, 2026 – with ChatGPT Codex Connector

    Contributor

    More actions

    Thanks for the follow-up. The fact that you now have one combined force plot but separate energy plots for the different configurations is not, by itself, a problem.

    For energy, I would actually keep the systems/configurations separate unless they have exactly the same atom count and composition. Absolute total energies from systems with different sizes/compositions are not directly comparable on one scatter plot; they can naturally form different offsets/bands. For checking the model, use the reference-vs-predicted columns from the same dp test output for each held-out system, preferably *.e_peratom.out, and also look at the residual (E_pred - E_ref) / N_atoms and its RMSE/MAE. The important question is whether each system follows the y = x line with a small residual, not whether all configurations collapse onto one common energy cloud.

    For forces, combining all Cartesian force components is fine because they have the same units. If the new force plot still shows the prediction range strongly compressed toward zero compared with DFT, that is still a real training/data problem and I would not change rcut, descriptor size, or fitting-net size yet.

    There are two concrete changes/checks I recommend before any network tuning:

    1. Your posted input still has validation_data.numb_btch: 1. That makes each validation point in lcurve.out depend on only one sampled validation batch and can be very noisy. Increase it (for example to 100) for monitoring, but use a full held-out evaluation with dp test -n 0 for the numbers you trust.
    2. For the 1000-frame trajectory question: splitting it into four contiguous 250-frame blocks and holding out one whole block is reasonable as a first block split, provided this split is done before any bootstrapping. Do not randomly interleave neighboring MD frames between train and validation, because adjacent MD frames are strongly correlated and this gives an overly optimistic validation set. An independent trajectory is better; if you only have one trajectory, a contiguous holdout block (ideally with a small gap/buffer to the training blocks) is preferable. Bootstrap/resample only the training frames after the split; never bootstrap the validation set.

    So I would not make another model-hyperparameter change yet. Please first evaluate every held-out system with all frames and compare the per-system energy-per-atom RMSE and force RMSE. If those are still poor, the next diagnostic is the small-overfit test mentioned above: train on ~100 unique frames from one system and verify that the model can fit those same frames. That cleanly separates a conversion/label-alignment problem from a coverage/generalization problem.

    Relevant training documentation: https://github.com/deepmodeling/deepmd-kit/blob/master/doc/train/training.md

    Agent: ChatGPT
    Model: GPT-5.6 Sol