Remove the experimental PRM trainer

[qgallouedec]

Removes trl.experimental.prm (PRMTrainer, PRMConfig), added in #2127.

Why

Low adoption. 9 genuine trainings in the last 2 months. Only 2 models publicly released in 2026 carry both the prm and trl tags: https://huggingface.co/models?other=prm,trl&sort=created

No development. Landed 2024-12-13 (#2127). Of the 39 commits that touched it since, none add a feature: they are repo-wide sweeps (disable_dropout generalization, doc-builder style, PEFT preparation, Python 3.9 drop, copyright years) and maintenance fixes. It was moved out of the stable API 2025-11-21 (#4483) and the deprecated alias dropped 2026-02-10 (#5052); nothing followed.

Notes

trl.experimental has no stability guarantee, so no deprecation cycle. The code stays in git history for anyone wanting to revive it. The paper entry goes from paper_index.md along with it.

This removes a capability, it does not consolidate one. Unlike #7136, where BCO users go to KTOTrainer, nothing in the stable API covers process supervision. The stepwise supervision dataset type leaves with the trainer: the entry in dataset_formats.md and its conversion recipes, the two dataset scripts (examples/datasets/math_shepherd.py, examples/datasets/prm800k.py), and the standard_stepwise_supervision config in the zen generator. PRM was its only consumer in the tree, so nothing dangles, but a documented format leaves the library's vocabulary. The published trl-internal-testing/zen keeps the config, only the generator stops emitting it.

One downstream consumer. axolotl imports trl.experimental.prm.PRMTrainer and subclasses it in src/axolotl/core/trainers/trl.py, and ships a stepwise_supervised prompt strategy against the same format. They pin trl==1.9.0, so nothing breaks today; they lose PRM whenever they next bump.

---

Note

Medium Risk
Removes a public experimental API and dataset-format documentation without a deprecation cycle; downstream imports and PRM training workflows will break on upgrade.

Overview
Removes the experimental process reward model (PRM) stack from TRL: PRMTrainer / PRMConfig under trl.experimental.prm, its unit tests, and the math_shepherd / prm800k dataset prep scripts.

Documentation is trimmed to match: the PRM trainer page and nav entries, the stepwise supervision dataset type (overview table, trainer mapping, conversion recipes), and the GSM8K process-vs-outcome paper blurb tied to PRMTrainer in paper_index.md. The zen test dataset generator no longer publishes a standard_stepwise_supervision config.

PRMTrainer is dropped from the _BaseTrainer trainer list. This is a breaking change for from trl.experimental.prm import ...; no deprecation path is applied because experimental APIs are unstable.

^{Reviewed by Cursor Bugbot for commit 110fd61. Bugbot is set up for automated code reviews on this repo. Configure here.}

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-09T14:38:57.425737Z	52fc777	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[albertvillanova]

Not opposed, and the reasoning is the same as #7136.

PRM is in the demoted group (#4483 took it out of stable), so it lands on exactly the open question in #7182: whether that group gets notice before removal. Worth settling there and applying the outcome to both PRs rather than deciding it twice here.

One thing wider than BCO: this also drops the stepwise supervision dataset type from dataset_formats.md, the two conversion scripts and the zen config. PRM was its only consumer so nothing dangles, but that is a documented format leaving the library's vocabulary rather than just a trainer, and it belongs in the body. (The published trl-internal-testing/zen keeps the config, only the generator stops emitting it.)

[albertvillanova]

One data point on the downstream question: axolotl imports this one directly.

# src/axolotl/core/trainers/trl.py
from trl.experimental.cpo import CPOTrainer
from trl.experimental.orpo import ORPOTrainer
from trl.experimental.prm import PRMTrainer

AxolotlPRMTrainer subclasses PRMTrainer, and they carry a stepwise_supervised prompt strategy matching the dataset format this PR also removes. They pin trl==1.9.0, so nothing breaks today, they lose PRM whenever they next bump.

Not an objection, and it does not change where the usage numbers point. Worth a line in the body though, since downstream use is one of the inputs we weigh.

[albertvillanova]

Approving, with two things I would like in the body first, since they are the axes that do not point the same way as the numbers:

1. Nothing in the stable API covers this. BCO had a destination in KTOTrainer; PRM has none, and the stepwise supervision dataset type leaves with it, so this removes a capability rather than consolidating one. That is a fair call to make, it just should be stated rather than left implicit.

2. axolotl imports trl.experimental.prm.PRMTrainer and subclasses it, and ships a stepwise_supervised prompt strategy against the same format. They pin trl==1.9.0, so nothing breaks now, they lose it whenever they next bump. That also refines what I said earlier in this thread: nothing dangles inside our tree, but the format does have a consumer outside it.

Everything else supports removal, which is why this is an approval rather than a block: usage is at the bottom among modules old enough to have been discovered, the external reports are few and not from running it, the code is small, no owner has turned up, and it has been available since 2024-12.

[qgallouedec]

Both added to the body. Agreed on the first: no stable destination and the format goes with it, so this is a capability leaving. On axolotl, i'll report on slack