New reference benchmark development checklist

[ShriyaRishab]

New benchmark task force leads should complete this checklist before the benchmark is finalized.

Initial reference code: roadmap

• Review guidelines on how to create a good mlperf training reference
• Finalize dataset
• Finalize model architecture
• Finalize reference framework
• Finalize platform/hardware that the reference would be implemented in, add it to the approved list
• Finalize reference precision and generate initial loss curves to understand training behavior. At this stage, decide the rough benchmarking region and figure out whether the benchmark should start training from randomly initialized weights or from a previously trained checkpoint. If training from random weights shows lots of instability in losses, it might be better to train for a a few hundred steps and generate a checkpoint so the benchmarking region is more stable and smooth.
• Finalize batch sizes for RCPs and get good hyperparameters at these batch sizes (need at least 3 batch sizes). We typically choose one small batch size, one very large batch size and one in the middle to cover a decent range. Can ask task force members and Training WG members for batch size range suggestions so as to choose options that cover what submitters are targeting.
• Finalize evaluation metric and dataset. Understand how much the initial dataset can be reduced, ideally we want the smallest possible dataset (both training and evaluation) that allows for a reasonable benchmark.
• Finalize which hyperparameters are unconstrained for submitters to modify

Finalize code

• Upload processed dataset to MLC bucket by reaching out to support@mlcommons.org who will give you write access and download instructions
• Create PR with initial codebase to https://github.com/mlcommons/training. This should have instructions on how to run the code and the README needs to follow the benchmark readme template. Note that you need to follow these Contribution guidelines to ensure that your github handle can contribute to MLC repositories.
• Add checkpoint + resume capability to the reference codebase because this feature tends to be very important when trying to run the reference on a small set of accelerators or when collecting RCPs which require expensive evaluation.

Generate RCPs

• Choose target accuracy
• Choose eval frequency
• Choose number of submission runs
• Generate RCPs
• Add new RCPs to the logging repo
• Commit RCP log files to the reference implementation folder in training/

Add new benchmark details

• Update logging repo to include compliance checks for the new benchmark (example PR359)
• Update training rules by adding the new benchmark in all relevant tables. Tables list - benchmarks, division, hyperparameters, quality, results, rcp rules. Add any benchmark specific rules to the appendix
• Update the compatibility table with the new benchmark
• Create a new benchmark presentation deck and present to the Training WG so everyone is aware of this benchmark and its technical details
• Create an initial blog write-up with technical details. Once its ready, MLC will bring in a technical writer who can polish it and public the blog before the first round when this benchmark is introduced. Sample blogs: flux.1, llama31_405b.

[CarlosGomes98]

Here's some thoughts I had coming fresh from the Flux development:

1. I would really like to see some explicit time assigned for reviewers to read / play with the code, and for that review to be addressed. I realize its hard to find folks with bandwidth to dedicate the time for this, but I think it would be a great improvement to the process. This would help avoiding the worst case, where bugs are found during or after the RCP generation, which might mean a lot of compute having to be re-invested.
2. Another explicit step could be, after the code is correct but maybe not yet finalized, understand how much the initial dataset can be reduced (I think this would fit here Finalize evaluation metric and dataset). I would think ideally we want the smallest possible dataset that allows for a reasonable benchmark
3. The order of finding target accuracy and eval frequency -> finding best hparams for RCPs is important. The hparams that lead to the "best convergence" are probably not going to be the same as those which reach the target accuracy the fastest. I think we probably want to choose target accuracy and eval frequency first, and then do the hparam search (as the benchmarks go through further submission rounds, other hparams tend do be found, so we end up optimizing for the target accuracy rather than convergence in general anyway)

Not sure how many of the above are actionable, but just some thoughts :)

[ShriyaRishab]

Thanks for your comments, Carlos. That's very helpful. I addressed (2) and (3) above!

For (1) - this is super important but we need to discuss it further with the WG to set those timelines. Lets discuss it in today's training WG meeting.

[pbaumstarck]

Hi, @ShriyaRishab , seems like this was addressed previously, and now we're in the process of commissioning new benchmarks. Do we need any additional follow-up on this issue, Carlos?

[ShriyaRishab]

@ShriyaRishab make this an issue template so new reference PRs have to work through the checklist to be approved and merged.