Improve preset creation harness and CLI output

[peterschmidt85]

What

Improvements to dstack preset create, driven by three end-to-end runs on real fleets.

Objective is now pinned. baseline: true makes the first trial a reference point rather than an optimization attempt, so an improvement percentage means something. input_tokens and output_tokens pin the benchmark workload alongside concurrency, so trials and the final service are comparable.

context_length meant two different things, a requirement in the config and a measurement in the preset. The config field is now min_context_length; context_length keeps the measured value. Each trial records the largest context it actually handles, found by sending real requests.

The final benchmark runs inside the service replica, directly against the engine, so it is comparable with the trial benchmarks instead of measuring the proxy and a different client.

Benchmarks may not reuse the previous one's prompts. Benchmark tools generate prompts from a fixed seed, so consecutive runs against a live engine were hitting its prefix cache and inflating later trials.

Final service attempts are recorded in verifications.jsonl and mirrored into the session directory, so dstack preset reads the phase instead of inferring it from a spent trial budget. That inference was wrong for every run that stopped early.

dstack preset output reworked: ps-style filtering with -a/-n, separate WORKLOAD and BEST columns, and one sparkline glyph per trial showing the shape of the search.

Evidence

Validated on three runs, not on tests alone.

The prompt-reuse fix is measurable. Trial and final-service benchmarks of the same configuration disagreed by 8.6% before it, and by 2.5% and 0.4% in the two runs after.

The status fix matters in practice: two of the runs stopped at 6 of 10 trials and one at 7 of 10, so the old count >= max_trials inference never fired and they showed trialing throughout verification.

The last run also covered ground the harness had never been exercised on: 4 GPUs, a new backend, concurrency 64, a 397B model and a 238GB download.

Notes

min_context_length is a breaking rename for existing preset configs, which fail with extra fields not permitted. Presets are marked experimental in the docs, so no alias was added.

340 CLI tests pass.

🤖 Generated with Claude Code