Skip to content

fix(jax): write the checkpoint pointer beside save_ckpt - #5726

Merged
wanghan-iapcm merged 3 commits into
deepmodeling:masterfrom
wanghan-iapcm:fix-jax-ckpt-pointer
Jul 10, 2026
Merged

fix(jax): write the checkpoint pointer beside save_ckpt#5726
wanghan-iapcm merged 3 commits into
deepmodeling:masterfrom
wanghan-iapcm:fix-jax-ckpt-pointer

Conversation

@wanghan-iapcm

@wanghan-iapcm wanghan-iapcm commented Jul 3, 2026

Copy link
Copy Markdown
Collaborator

Problem

Fixes #5678. JAX training writes checkpoint directories and the stable .jax link relative to save_ckpt (which may include a directory), but always wrote the checkpoint pointer file to the current working directory with a value that still carried the directory prefix, e.g. runs/water/model.ckpt.jax. The freeze entrypoint looks for the pointer inside the folder it is given and resolves the pointer's value relative to that folder (checkpoint_folder / pointer). So for save_ckpt = runs/water/model.ckpt, the pointer was written to ./checkpoint (not runs/water/checkpoint) and, even if relocated, its value would have double-prefixed to runs/water/runs/water/model.ckpt.jax. Passing runs/water to freeze or restart-style tooling could not find or resolve the checkpoint, even though the matching checkpoint directory and .jax link were written there.

Fix

Write the pointer into Path(save_ckpt).parent and store a value relative to that directory (the basename only). For the default bare save_ckpt (parent is .) the pointer stays in the CWD with the same value, so existing behavior is unchanged; only directory-valued save_ckpt is affected.

Test

Adds source/tests/jax/test_checkpoint_pointer.py, which drives _save_checkpoint with the checkpoint I/O mocked. The directory case asserts the pointer lands beside the checkpoint (subdir/checkpoint) with a basename value (model.ckpt.jax) and not in the CWD — this fails on master — and a bare-name control asserts the pointer stays in the CWD. The trainer's pointer writing previously had no coverage; the existing freeze test hand-wrote a correct pointer and never exercised the writer.

Summary by CodeRabbit

  • Bug Fixes

    • Fixed checkpoint “pointer” file placement to be written alongside the configured checkpoint save directory/path rather than always in the current working directory.
    • Updated pointer contents to reference the correct checkpoint name, improving downstream checkpoint resolution during freeze/restart.
  • Tests

    • Added unit tests validating pointer location and contents for both nested save paths and bare checkpoint filenames.

JAX training writes checkpoint directories and the stable .jax link relative to
save_ckpt (which may include a directory), but always wrote the "checkpoint"
pointer file to the current working directory with a value that still carried
the directory prefix (e.g. "runs/water/model.ckpt.jax"). The freeze entrypoint
looks for the pointer inside the folder it is given and resolves the value
relative to that folder, so a directory-valued save_ckpt both misplaced the
pointer and double-prefixed the resolved path, breaking freeze and restart-style
tooling.

Write the pointer into Path(save_ckpt).parent and store a value relative to that
directory (the basename only). For the default bare save_ckpt (parent == "."),
the pointer stays in the CWD with the same value, so existing behavior is
unchanged.

Adds source/tests/jax/test_checkpoint_pointer.py, which drives _save_checkpoint
with the checkpoint I/O mocked: the directory case asserts the pointer lands
beside the checkpoint with a basename value and not in the CWD (fails on master),
and a bare-name control asserts the pointer stays in the CWD. The trainer's
pointer writing previously had no test; the existing freeze test hand-wrote a
correct pointer and never exercised it.

Fix deepmodeling#5678
@coderabbitai

coderabbitai Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: d713d61a-c160-4cbd-8c29-c2e37d5ca021

📥 Commits

Reviewing files that changed from the base of the PR and between bc2095b and 767425c.

📒 Files selected for processing (1)
  • source/tests/jax/test_checkpoint_pointer.py
🚧 Files skipped from review as they are similar to previous changes (1)
  • source/tests/jax/test_checkpoint_pointer.py

📝 Walkthrough

Walkthrough

DPTrainer._save_checkpoint now writes the checkpoint pointer beside the configured save_ckpt prefix and stores its basename. Tests cover subdirectory and bare-name checkpoint paths.

Changes

Checkpoint pointer fix

Layer / File(s) Summary
Write pointer relative to save_ckpt directory
deepmd/jax/train/trainer.py
_save_checkpoint creates the save_ckpt directory when needed and writes the relative checkpoint pointer there.
Pointer placement tests
source/tests/jax/test_checkpoint_pointer.py
Tests isolate the working directory and verify pointer location and basename content for subdirectory and bare-name paths.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related issues

Possibly related PRs

Suggested reviewers: njzjz, iProzd

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main fix: writing the JAX checkpoint pointer beside save_ckpt.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@dosubot dosubot Bot added the bug label Jul 3, 2026
@wanghan-iapcm
wanghan-iapcm requested a review from njzjz July 3, 2026 17:36
@github-actions github-actions Bot added the Python label Jul 3, 2026
Comment thread source/tests/jax/test_checkpoint_pointer.py Fixed
@codecov

codecov Bot commented Jul 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.53%. Comparing base (0acd0e5) to head (767425c).
⚠️ Report is 3 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #5726      +/-   ##
==========================================
- Coverage   79.62%   79.53%   -0.09%     
==========================================
  Files        1014     1015       +1     
  Lines      115533   115740     +207     
  Branches     4276     4274       -2     
==========================================
+ Hits        91995    92056      +61     
- Misses      21994    22139     +145     
- Partials     1544     1545       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@wanghan-iapcm
wanghan-iapcm enabled auto-merge July 10, 2026 15:03
Resolve the CodeQL "Module is imported with 'import' and 'import from'"
alert on test_checkpoint_pointer.py: it imported unittest with a bare
`import unittest` (for TestCase/main) and also `from unittest import mock`.
Import the mock helper from the unittest.mock submodule instead
(`from unittest.mock import patch`), matching the prevailing convention in
the rest of the test suite, and reference `patch`/`patch.object` directly.
@wanghan-iapcm
wanghan-iapcm added this pull request to the merge queue Jul 10, 2026
Merged via the queue into deepmodeling:master with commit 7c362d7 Jul 10, 2026
57 checks passed
@wanghan-iapcm
wanghan-iapcm deleted the fix-jax-ckpt-pointer branch July 10, 2026 22:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Code scan] Write JAX checkpoint pointers beside save_ckpt

3 participants