feat(dpa4): DPA4/SeZM on the NeighborGraph lower — graph-native port, multi-rank, native spin, charge-spin/bridging - #5884
Conversation
…erLayer.call_graph parity Add parametrized test case 'no_chnnl_2_no_gg1' to test_repformer_layer_call_graph_parity that exercises two previously untested branches: 1. update_chnnl_2=False: Tests the short-circuit path where g2/h2 updates are skipped and returned unchanged (lines 2397-2401 in repformers.py call_graph). This is the critical production path for the LAST layer of every DPA2 model. 2. cal_gg1=False (gg1=None): Tests the graph path when no gg1-dependent updates are needed (all of update_g1_has_drrd, update_g1_has_conv, update_g1_has_attn, update_g2_has_g1g1 are False). With g1_out_mlp=False, g1_mlp is seeded with the identity [g1] for xp.concat. Test passes all 30 cases (29 baseline + 1 new).
gen_dpa2.py section B builds a graph-eligible dpa2 (use_three_body=False; three-body is graph-ineligible), exports the graph-form .pt2 (which embeds the nested with-comm artifact automatically for message-passing models) plus an independent dense-nlist .pt2 of the same weights, cross-checks atomic energies/forces/total virial between the two at gen time (NaN-safe), and writes deeppot_dpa2_graph.expected from the graph artifact eval. Unlike the dpa1 precedent the .expected is not copied from the nlist oracle: for a message-passing model the graph path's full-to-source per-edge atomic-virial decomposition legitimately differs from the dense per-atom decomposition (only the sum agrees), and cross-path e/f noise (~1e-10) sits at the universal gtest's 1e-10 double tolerance. The dpa2_graph_ptexpt VariantDeepPotCase row mirrors dpa1_graph_ptexpt (same tolerances and capability flags).
- Add nall_real == 0 guard at the top of graph with-comm arm (lines 922-929) to throw a clear error instead of silently crashing with Dim violation - Fix stale comment (lines 1050-1051) to reflect that ghost forces fold back via both plain and with-comm graph routes
…h-comm route The with-comm GRAPH artifact derives n_local from the nlocal comm tensor IN-GRAPH (owned-node energy mask); after move_to_device_pass that derivation is a device kernel, so feeding it a CPU tensor makes the kernel read a host pointer as device memory -> CUDA illegal memory access on every multi-rank run (first live exercise of the route, T4). The other six comm tensors are consumed only by the opaque border_op whose host code dereferences their data_ptr, so they stay on CPU -- same placement the dense with-comm route uses for all eight. Verified on Tesla T4: minimal AOTI repro crashes with CPU nlocal and matches the plain graph artifact bit-for-bit with device nlocal.
… DEVICE
_trace_and_compile_graph built its synthetic NeighborGraph trace inputs on
the module-level DEVICE constant instead of the device the model's own
parameters/buffers actually live on. In real training these always match
(the trainer moves the model to DEVICE before compiling), but any caller
that intentionally holds the model on a different device (e.g. a test
pinning the model to CPU while running on a CUDA host, mirroring the .pt2
export's model.to("cpu") pattern) hits a device split between the traced
graph tensors and the model params.
DPA2.call_graph's tebd gather (dpmodel/descriptor/dpa2.py:1159) surfaced
this first: xp.take(tebd_table, atype_local, axis=0) requires tebd_table
(a model param, left un-asarray'd to avoid detaching its gradient -- the
documented dpa1 lesson) and atype_local (moved to device(graph.edge_vec))
to share a device. Under test_compiled_training_graph_smoke the model is
pinned to CPU but the trace sample was still built on the global DEVICE
(cuda:0 on a GPU box), so tebd_table stayed CPU while atype_local was CUDA.
DPA1's call_graph has the identical gather pattern and would fail the same
way, but no existing dpa1 test calls _trace_and_compile_graph directly with
a device-pinned model, so the bug was latent there since PR-B (deepmodeling#5604).
Fix: add _model_trace_device(), reading the device off the model's own
parameters/buffers (falling back to DEVICE only if the model exposes
neither), and use it to build the trace sample instead of the global
DEVICE. No change to the gather itself -- gradient flow through
type_embedding is untouched.
…back test dpa2 became graph-eligible (uses_graph_lower=True), so neighbor_list=None now routes to the carry-all graph while explicit DefaultNeighborList() forces the dense route. The smooth repformer attention diverges intentionally between these paths (sel-independent on graph, sel-padded on dense), amplified through message passing. Observed divergence at this fixture: - energy rel ~1e-4, abs ~9.5e-4 - force abs ~5e-3 - virial abs ~1.3e-2 Set atol=2e-2, rtol=1e-3 to accommodate dpa2's larger divergence compared to dpa1_smooth/se_atten_v2.
for more information, see https://pre-commit.ci
…adapter DPA2's repinit is hardwired to attn_layer=0, where the dense se_atten body leaks a phantom padding-neighbor -davg/dstd residual that the graph path deliberately omits (the physically-correct behavior, accepted as KNOWN_GRAPH_DENSE_DIVERGENT). So _call_graph_adapter is bit-exact vs _call_dense only in the trivial-statistics regime (davg=0, dstd=1); for any model with non-trivial stats it diverges. The T7 call gate routed the dense .call() through _call_graph_adapter for graph-eligible configs, which made the cross-backend consistency reference diverge from the leaky tf/pt/pd/jax dense bodies (source/tests/pt|pd/model/ test_dpa2.py::test_consistency, 71% mismatch) and crashed the universal zero_forward on empty systems. The graph-native route is reached exclusively through call_graph (pt_expt forward_atomic_graph + C++ graph .pt2), never through call, so .call() now always runs _call_dense. _call_graph_adapter is retained as the bit-exact-regime reference exercised by TestDPA2AdapterBitExact. Updates TestDPA2CallRouting.test_call_routing_graph_eligible to assert the corrected routing (call == _call_dense) and corrects the _call_graph_adapter bit-exactness docstring.
The C++ memleak matrix runs gen scripts with LD_PRELOAD=liblsan (the sanitizer-instrumented deepmd op .so needs the LSAN runtime). Evaluating the AOTInductor-compiled deeppot_dpa2_graph.pt2 BACKWARD (forces) under that runtime segfaults inside the repformer's fused backward kernel (cpp_fused_..._slice_backward_...): an AOTI-compiled-code vs LeakSanitizer allocator incompatibility, NOT a graph-code bug. The identical backward is finite and correct in eager, in the non-memleak C++ ctest, and in LAMMPS; dpa1's simpler graph .pt2 does not trip it. Leak-checking torch's own compiled kernels is meaningless (the memleak build exists to leak-check deepmd's C++ ops), so excluding the dpa2 graph artifact from the memleak build loses no real coverage. - gen_dpa2.py: skip the graph section (Section B) when LD_PRELOAD contains liblsan; the dense DPA2 .pt2 (section A) is still generated. - test_deeppot_universal.cc: add skip_if_artifact_missing to VariantDeepPotCase and set it on the dpa2_graph_ptexpt row, so SetUp GTEST_SKIPs (instead of failing) when the artifact is absent under memleak. Every other case still hard-fails on a missing artifact. LAMMPS/ipi tests already run only when !check_memleak, so no other memleak consumer of the artifact remains.
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
The flag still claimed 'nlist' was the default and described 'graph' as opt-in for eligible models, while the entrypoint has forced every graph-capable model onto the graph lower since 881e208. The help text now matches: it is consulted only for models that cannot use the graph lower, and requesting 'nlist' for a graph-capable one is overridden with a warning. Also records WHY the gate is 'can use graph' rather than a narrower 'graph is the only route': the dense lower is deprecated and every DPA model is expected to become graph-capable, so a graph-required capability would encode a distinction that stops existing.
The new override-logging test imported deepmd.pt_expt.entrypoints.main and deepmd.pt_expt.utils.serialization as modules while the file already imports names from both, which CodeQL reports as importing a module with both 'import' and 'import from'. Patch by string target instead, which needs neither module import: the mock target names the source module directly (freeze imports deserialize_to_file inside the function, so it must be patched there), and caplog takes the logger name as a literal.
…sion The previous commit made the DEFAULT require agreement among active consumers but left the DIMENSION on max() across all children, so two children wanting different widths of the one shared tensor composed happily and the wider silently won. __init__ now rejects consumers that disagree, for fparam and aparam alike, next to the intensive/extensive check. Children of dimension 0 do not consume the input and are ignored, so dpa4 + ZBL inherits dpa4's dimension AND its default rather than being constrained by the analytical term. max() in the accessors is then exact rather than a guess -- the only other value present is 0 from a non-consumer -- which the docstrings now say. aparam has no default concept on the base atomic model, so only its dimension is validated. Two test doubles needed completing: they predate the accessors the composition now validates. Completing them keeps the fail-fast honest rather than weakening it to skip children that do not implement the interface.
OutisLi
left a comment
There was a problem hiding this comment.
Reviewed the latest HEAD. The previously requested changes have been addressed.
OutisLi
left a comment
There was a problem hiding this comment.
[P2] Preserve the PT-aligned forward-method ordering in dpa4.py. The PT SeZM implementation deliberately presents this code top-down: construction, the peer forward entrypoints, their shared implementation, the core block computation, and then lower-level helpers. The dpmodel port currently orders the section as call -> _run_graph -> call_graph -> _forward_blocks, so the private _run_graph separates the two peer entrypoints and makes readers traverse roughly 260 lines of internals before reaching call_graph. Please make this a move-only organization change and use __init__ -> call -> call_graph -> _run_graph -> _forward_blocks -> remaining helpers. Do not reorganize it around DPA2 conventions; the goal is to keep the port structurally aligned with the intentionally arranged PT implementation. This should not change runtime, checkpoint, or serialization behavior.
OutisLi
left a comment
There was a problem hiding this comment.
[P2] Correct the PT DPA4 freeze and multi-rank capability matrix in dpa4.md. The important block currently says that the default dp --pt freeze output is a dense-kind .pt2 and is single-rank only. The implementation does the opposite: deepmd.pt.entrypoints.main.freeze detects DPA4/SeZM and calls freeze_sezm_to_pt2; SeZMModel.export_lower_input_kind() returns edge_vec (a compact edge-list ABI, not a padded dense nlist), and a plain model reports supports_edge_parallel()==True, so freeze embeds model/extra/forward_lower_with_comm.pt2 and sets has_comm_artifact=true. The PT export tests explicitly assert that artifact. Consequently the standard PT-frozen plain DPA4 archive supports multi-rank LAMMPS. The text also conflates the fixed PT edge_vec export route with pt_expt --lower-kind graph; these are different graph/edge ABIs, but neither should be described as the default PT dense route. Please rewrite this block to distinguish PT legacy edge_vec from the pt_expt NeighborGraph ABI and state the correct multi-rank support for both plain-energy archives.
The private _run_graph sat between the two peer forward entrypoints, so a reader met roughly 260 lines of internals before reaching call_graph. The PT SeZM implementation is deliberately arranged top-down, and the dpmodel port should read the same way beside it. Order is now __init__ -> call -> call_graph -> _run_graph -> _forward_blocks -> remaining helpers, following PT rather than DPA2 conventions. Move-only: the two blocks were swapped whole and the resulting file has an identical multiset of lines and an identical set of 54 method definitions, so no body, signature or decorator changed. No runtime, checkpoint or serialization behaviour is affected. Review 4781239183.
The important block claimed the default `dp --pt freeze` output is a dense-kind .pt2 and single-rank only. The implementation does the opposite: freeze detects DPA4/SeZM and calls freeze_sezm_to_pt2, SeZMModel.export_lower_input_kind() returns "edge_vec" (a compact edge-list ABI whose topology is built C++-side, not a padded dense nlist), and supports_edge_parallel() is False only when an inter_potential is present -- so a plain energy model embeds forward_lower_with_comm.pt2 and DOES support multi-rank LAMMPS. The block also conflated the PT edge_vec route with pt_expt's NeighborGraph `--lower-kind graph`. They are different ABIs consumed by different C++ paths, and neither is a dense route; the text now describes them separately and states multi-rank support for each. Corrected in the same pass, all invalidated by work in this branch: charge/spin conditioning and ZBL bridging are graph-eligible (not dense-only); native spin supports multi-rank since the with-comm graph spin export landed; and only the deepspin virtual-atom scheme is genuinely dense-only. Bridging remains single-rank, for the SFPG reason. Review 4781248431.
… CUDA Test CUDA failed in the merge queue (it does not run on PR checks, so every PR check was green): the pt_expt roundtrip resolved the wire type through the DPMODEL registry, giving a NumPy-backed model, while _pair_energy feeds tensors on _env.DEVICE. On CPU that is harmless; on CUDA apply_out_stat evaluates ret[kk] + out_bias[kk][atype], indexing a NumPy out_bias with a CUDA atype, and NumPy calls __array__ on it -> can't convert cuda:0 tensor to numpy. Restore through type(child) instead. The registry lookup is kept as an explicit assertion so the wire type is still covered, and a same-class restore is the stricter comparison anyway.
test_native_spin_graph_freeze asserted has_comm_artifact is True and, six lines later, that forward_lower_with_comm.pt2 is ABSENT. Those contradict each other. The metadata assertion was updated when native spin gained multi-rank support on the graph lower (804cc57); the file assertion is a leftover from the single-rank contract that preceded it. Caught only on GPU: the CPU freeze fails earlier on this workstation's inductor AVX2 codegen bug, so the stale assertion was never reached locally, and Test CUDA runs only on the merge queue. Invert it -- the nested artifact must be present -- so the two checks agree and a regression that silently stopped embedding it would fail here.
) (deepmodeling#5916) Reverts deepmodeling#5882 — "ci(cuda): overlap AOTI .pt2 compiles with GPU unit tests". This undoes the CUDA CI orchestration that PR added: - The two-lane pytest split in `test_python` (`-m aoti_compile` vs `-m "not aoti_compile"`). - The `aoti_compile` pytest marker machinery in `source/tests/conftest.py` (producer list, `pytest_collection_modifyitems` tagging, and the runtime drift guard). - The `aoti_compile` marker registration in `pyproject.toml`. - The persistent `AOTInductor` compile cache (`actions/cache`) in both CUDA jobs. - The per-lane JUnit report upload. - The phase-gating (`DP_CC_SKIP_BUILD` / `DP_CC_SKIP_CTEST`) and atomic `.so` install changes in `source/install/test_cc_local.sh`. The serial `python -m pytest source/tests` invocation and the original `test_cc_local.sh` flow are restored. ## Conflict resolution `deepmodeling#5882` was followed by unrelated commits that touched the same regions. The revert preserves those later changes and only undoes `deepmodeling#5882`: - **Dependabot bumps preserved:** `actions/setup-python@v7` (deepmodeling#5898) and `actions/upload-artifact@v7` (deepmodeling#5900) are kept. Only `deepmodeling#5882`'s additions (the `actions/cache` blocks and the JUnit upload step) are removed; the version bumps that landed in the same context stay. - **DPA4 test fixtures preserved (deepmodeling#5884):** the native-spin / bridging fixture generators `gen_dpa4_spin.py`, `gen_dpa4_zbl.py`, `gen_dpa4_spin_chgspin.py`, and `gen_dpa4_spin_zbl.py` are kept in `test_cc_local.sh` (re-indented to the restored single-lane flow). The matching entry `deepmodeling#5884` added to `_AOTI_COMPILE_MODULES` is dropped together with that set, since the whole `aoti_compile` partition feature is being removed — the `test_zbl_bridging` module still runs, just unpartitioned. - Unrelated `docs` dependency bumps in `pyproject.toml` (sphinx / myst) are untouched. Net result: `conftest.py` matches its pre-`deepmodeling#5882` state exactly; `test_cuda.yml` differs from pre-`deepmodeling#5882` only by the preserved `setup-python@v7` bumps; `test_cc_local.sh` differs only by the preserved DPA4 fixtures; `pyproject.toml` differs only by the preserved `docs` dependency bumps. --- Coding agent: opencode opencode version: 1.18.8 Model: ustc/glm-5.2 Reasoning effort: max <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Streamlined CUDA test execution with a single comprehensive test run and simplified environment settings. * Improved test setup compatibility for Paddle and TensorFlow imports. * Updated test marker configuration to support the current test suite. * **Chores** * Improved local C++ build, installation, fixture generation, and validation workflows. * C++ tests now run automatically after successful builds. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: njzjz-bot <njzjz.bot@gmail.com>
The SFPG per-node partials are completed across ranks by one reverse-accumulate + forward-broadcast border exchange of an (N,2) tensor per forward pass (issue deepmodeling#5906); also sweeps the stale pre-deepmodeling#5884 native-spin single-rank claims.
Summary
Brings the DPA4/SeZM descriptor fully onto the NeighborGraph ("graph") lower — the sel-free, edge-native route that dpa1 (#5583) and dpa2 (#5779) already use — and adds the conditioning inputs DPA4 needs on that route. Stacked on the merged dpa2 graph work (#5779); this PR is DPA4's turn on the same infrastructure.
Four layers, built and reviewed incrementally:
callbecomes a thin adapter over one graph-native math owner (_call_graph_impl); the model-level graph seam drives DPA4 throughcall_graph. Parity vs the dense route at fp64 1e-12; parity vs the pt reference at ~6e-15..pt2— per-blockborder_opghost-feature exchange on the graph lower; graph-kind.pt2embeds the nested with-comm artifact. Dense (nlist) lower stays comm-less and fails fast on multi-rank; bridging models also fail fast (a rank cannot observe a ghost owner's full outgoing-edge set).(nf, nloc, 3)conditions the descriptor and is a second autograd leaf givingforce_mag = -dE/dspin. Rides the graph route only; frozen to a graph-kind.pt2(spin at positional ABI index 10), evaluated in Python (DeepEval) and C++ (DeepSpinPTExptgraph route), run in LAMMPS (pair_style deepspin). Single-rank.DPA4NativeSpinModelin dpmodel + pt_expt.charge_spinand flips the capability gates).Validation
force_magvs finite difference: 5.5e-10 (atol 1e-6); pt weight-copied parity 1e-12.deeppot_universalDPA4 dense + graph rows 38 passed / 8 skipped (NoPbc profile); native-spinDeepSpinPTExptgraph gtest 16 passed (double/float × PBC/NoPbc).deepspinenergy/force/force_mag; 2-rank multi-rank fail-fast aborts cleanly.index_addatomics introduce 1-2 fp64 ULP run-to-run nondeterminism).Known limitations
.pt2carrieshas_comm_artifact=falseand C++ fails fast on multi-rank. Bridging is intrinsically single-rank (per-node freeze fold over the full outgoing-edge set).add_chg_spin_ebd+ native spin) — a small follow-up.sezm_native_spincheckpoints is not claimed — the deserialize alias covers the type string with dpmodel structure only.edge_vec.pt2lower schema (DPA4's original inference rail) is quarantined withDeprecationWarning, not removed — its removal is a follow-up gated on repointing the pt-SeZM freeze (and, for its spin part, on graph-spin support, now landed here).Summary by CodeRabbit
force_mag) and spin masking..pt2export/inference support on the graph route (with appropriate single-rank/MPI behavior).edge_vecdeprecation.