fix(jax): write the checkpoint pointer beside save_ckpt - #5726
Conversation
JAX training writes checkpoint directories and the stable .jax link relative to save_ckpt (which may include a directory), but always wrote the "checkpoint" pointer file to the current working directory with a value that still carried the directory prefix (e.g. "runs/water/model.ckpt.jax"). The freeze entrypoint looks for the pointer inside the folder it is given and resolves the value relative to that folder, so a directory-valued save_ckpt both misplaced the pointer and double-prefixed the resolved path, breaking freeze and restart-style tooling. Write the pointer into Path(save_ckpt).parent and store a value relative to that directory (the basename only). For the default bare save_ckpt (parent == "."), the pointer stays in the CWD with the same value, so existing behavior is unchanged. Adds source/tests/jax/test_checkpoint_pointer.py, which drives _save_checkpoint with the checkpoint I/O mocked: the directory case asserts the pointer lands beside the checkpoint with a basename value and not in the CWD (fails on master), and a bare-name control asserts the pointer stays in the CWD. The trainer's pointer writing previously had no test; the existing freeze test hand-wrote a correct pointer and never exercised it. Fix deepmodeling#5678
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthrough
ChangesCheckpoint pointer fix
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related issues
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #5726 +/- ##
==========================================
- Coverage 79.62% 79.53% -0.09%
==========================================
Files 1014 1015 +1
Lines 115533 115740 +207
Branches 4276 4274 -2
==========================================
+ Hits 91995 92056 +61
- Misses 21994 22139 +145
- Partials 1544 1545 +1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Resolve the CodeQL "Module is imported with 'import' and 'import from'" alert on test_checkpoint_pointer.py: it imported unittest with a bare `import unittest` (for TestCase/main) and also `from unittest import mock`. Import the mock helper from the unittest.mock submodule instead (`from unittest.mock import patch`), matching the prevailing convention in the rest of the test suite, and reference `patch`/`patch.object` directly.
Problem
Fixes #5678. JAX training writes checkpoint directories and the stable
.jaxlink relative tosave_ckpt(which may include a directory), but always wrote thecheckpointpointer file to the current working directory with a value that still carried the directory prefix, e.g.runs/water/model.ckpt.jax. The freeze entrypoint looks for the pointer inside the folder it is given and resolves the pointer's value relative to that folder (checkpoint_folder / pointer). So forsave_ckpt = runs/water/model.ckpt, the pointer was written to./checkpoint(notruns/water/checkpoint) and, even if relocated, its value would have double-prefixed toruns/water/runs/water/model.ckpt.jax. Passingruns/waterto freeze or restart-style tooling could not find or resolve the checkpoint, even though the matching checkpoint directory and.jaxlink were written there.Fix
Write the pointer into
Path(save_ckpt).parentand store a value relative to that directory (the basename only). For the default baresave_ckpt(parent is.) the pointer stays in the CWD with the same value, so existing behavior is unchanged; only directory-valuedsave_ckptis affected.Test
Adds
source/tests/jax/test_checkpoint_pointer.py, which drives_save_checkpointwith the checkpoint I/O mocked. The directory case asserts the pointer lands beside the checkpoint (subdir/checkpoint) with a basename value (model.ckpt.jax) and not in the CWD — this fails on master — and a bare-name control asserts the pointer stays in the CWD. The trainer's pointer writing previously had no coverage; the existing freeze test hand-wrote a correct pointer and never exercised the writer.Summary by CodeRabbit
Bug Fixes
Tests