diff --git a/mkdocs/docs/concepts/presets.md b/mkdocs/docs/concepts/presets.md
index 1ea18073f..a57f5e537 100644
--- a/mkdocs/docs/concepts/presets.md
+++ b/mkdocs/docs/concepts/presets.md
@@ -33,6 +33,11 @@ base: Qwen/Qwen2.5-7B-Instruct
# The number of benchmarked trials
max_trials: 3
+# The benchmark workload (defaults to 8, 1024, 1024)
+concurrency: 16
+input_tokens: 2048
+output_tokens: 512
+
env:
- HF_TOKEN
```
@@ -66,9 +71,11 @@ $ dstack preset create -f preset.dstack.yml --resume a1b2c3d4
+When resuming, `model`, `prompt`, `min_context_length`, `max_trials`, `concurrency`, `baseline`, and `fleets` are read from the original session, not from the configuration file. Editing them and resuming has no effect. To change any of them, create a new preset.
+
To stop a creation and its runs, use `dstack preset stop`.
-!!! info "Claude configuration"
+??? info "Claude configuration"
By default, preset creation uses the existing `claude` login. To use an Anthropic API key instead, set:
```shell
@@ -84,6 +91,11 @@ To stop a creation and its runs, use `dstack preset stop`.
Supported effort levels are `low`, `medium`, `high`, `xhigh`, and `max`.
+!!! info "Recommended setup"
+ - Profile settings such as `spot_policy`, `max_price`, and `backends` are ignored during preset creation. Configure them on the fleet instead, and pass a specific fleet so the agent uses the offers you intend.
+ - The agent chooses the hardware that gets the best performance. If you have a preference on hardware (e.g. price or specific GPUs), use a fleet that is pre-configured correspondingly.
+ - If `min_context_length` is unset, the agent may choose a short context to get a better benchmark result. Set it explicitly.
+
## Configuration options
### Model
@@ -106,15 +118,21 @@ To stop a creation and its runs, use `dstack preset stop`.
### Trials
-`max_trials` is required and sets how many benchmarked trials the agent runs before promoting the best one. Set `concurrency` to control the benchmark concurrency.
+`max_trials` is required and sets how many benchmarked trials the agent runs before promoting the best one.
+
+`concurrency`, `input_tokens`, and `output_tokens` define the benchmark workload, and default to 8, 1024, and 1024. The same values are used for every trial and for the final service benchmark, so results are comparable.
+
+### Baseline
+
+Set `baseline: true` to make the first trial a baseline: the agent serves the model the way the chosen serving framework recommends, without tuning it for performance. Later trials are optimization attempts.
### Context length
-Set `context_length` to require a minimum supported context length.
+Set `min_context_length` to require a minimum supported context length.
### Fleets
-Set `fleets` to restrict creation and reuse to specific [fleets](fleets.md). Placement properties such as `backends`, `max_price`, and `spot_policy` constrain both creation and reuse too.
+Set `fleets` to restrict creation and reuse to specific [fleets](fleets.md).
### Prompt
@@ -147,7 +165,7 @@ $ dstack preset apply -f preset.dstack.yml --id 532f3f4b
Spot policy on-demand
Max price off
Model Qwen/Qwen3.5-27B (base)
- Preset 532f3f4b (ctx=8K con=8 387 tok/s TTFT 582ms)
+ Preset 532f3f4b (ctx=8K io=512/512 con=8 tps=387 ttft=582)
# BACKEND RESOURCES INSTANCE TYPE PRICE
1 runpod (EU-RO-1) cpu=12 mem=54GB disk=100GB gpu=RTXPRO4500:32GB:1 NVIDIA RTX PRO 4500 Blackwell $0.74
@@ -167,20 +185,30 @@ Use `dstack preset` to list presets:
```shell
$ dstack preset list
- BASE ID GPU BENCHMARK STATUS SUBMITTED NAME
- Qwen/Qwen2.5-0.5B
- bc592b38 clauding (0/3) 23 sec ago qwen05
- Qwen/Qwen3-32B
- f91d6b60 RTX5090:32GB:1 con=8 576 tok/s TTFT 368ms verified (10/10) 2 days ago qwen3-32b
- Qwen/Qwen3.5-27B
- 3c4d5e6f verifying (3/3) 2 min ago qwen35-27b-2
- 532f3f4b RTXPRO4500:32GB:1.. con=8 387 tok/s TTFT 582ms verified (4/4) yesterday qwen35-27b
- d1c2e12b RTX5090:32GB:1 con=8 266 tok/s TTFT 2.15s verified (7/10) yesterday
+ ID BASE GPU WORKLOAD BEST STATUS SUBMITTED
+ bc592b38 Qwen/Qwen3.6-27B RTXPRO4500:32GB:1 ctx=32K io=1024/1024 con=8 tps=422 ttft=501 ▁▄█ trialing (3/10) 2 min ago
```
-Presets are grouped by base model. In-progress creations appear too, with a live status like `clauding` or `verifying`. Pass `-w` to watch in realtime. Pass `-v` to include validation resources and all benchmark metrics, or `--json` for complete preset objects. Filter with `--base` or `--repo`.
+By default, `dstack preset` shows creations that are still running, or the most recent one if none are. Pass `-a` to show every preset, or `-n` to show the last N:
+
+
+
+```shell
+$ dstack preset list -a
+ ID BASE GPU WORKLOAD BEST STATUS SUBMITTED
+ bc592b38 Qwen/Qwen3.6-27B RTXPRO4500:32GB:1 ctx=32K io=1024/1024 con=8 tps=422 ttft=501 ▁▄█ trialing (3/10) 2 min ago
+ f91d6b60 Qwen/Qwen3-32B RTX5090:32GB:1 ctx=32K io=1024/512 con=8 tps=576 ttft=368 ▁▁▅▅▄▅▇█ verified (8/8) 2 days ago
+ 532f3f4b Qwen/Qwen3.5-27B RTXPRO4500:32GB:1 ctx=8K io=512/512 con=8 tps=387 ttft=582 ▁▇██ verified (4/4) yesterday
+ d1c2e12b Qwen/Qwen3.5-27B RTX5090:32GB:1 ctx=32K io=1024/128 con=8 tps=266 ttft=2154 █▆█▁███ verified (7/10) yesterday
+```
+
+
+
+The `BEST` column is the best trial so far, followed by one glyph per trial: height is throughput, `·` is a trial that produced no benchmark. The shape shows whether a run converged or wandered.
+
+Pass `-w` to watch in realtime. Pass `-v` to include validation resources, the model repo, and the preset name, or `--json` for complete preset objects. Filter with `--base` or `--repo`.
### Delete presets
diff --git a/src/dstack/_internal/cli/commands/preset.py b/src/dstack/_internal/cli/commands/preset.py
index 184df2592..3dc5334cc 100644
--- a/src/dstack/_internal/cli/commands/preset.py
+++ b/src/dstack/_internal/cli/commands/preset.py
@@ -235,12 +235,26 @@ def _list(self, args: argparse.Namespace) -> None:
verbose = args.verbose
if not getattr(args, "watch", False):
presets, sessions = self._list_presets_and_sessions(base=base, repo=repo)
- print_presets(presets, sessions=sessions, verbose=verbose)
+ print_presets(
+ presets,
+ sessions=sessions,
+ verbose=verbose,
+ all_presets=args.all_presets,
+ limit=args.limit,
+ )
return
with Live(console=console, refresh_per_second=LIVE_TABLE_REFRESH_RATE_PER_SEC) as live:
while True:
presets, sessions = self._list_presets_and_sessions(base=base, repo=repo)
- live.update(get_presets_table(presets, sessions=sessions, verbose=verbose))
+ live.update(
+ get_presets_table(
+ presets,
+ sessions=sessions,
+ verbose=verbose,
+ all_presets=args.all_presets,
+ limit=args.limit,
+ )
+ )
time.sleep(LIVE_TABLE_PROVISION_INTERVAL_SECS)
def _list_presets_and_sessions(
@@ -408,6 +422,21 @@ def _add_list_args(parser: argparse.ArgumentParser) -> None:
action="store_true",
help="Output in JSON format",
)
+ parser.add_argument(
+ "-a",
+ "--all",
+ action="store_true",
+ dest="all_presets",
+ help="Show all presets. By default, it only shows unfinished creations or the last one.",
+ )
+ parser.add_argument(
+ "-n",
+ "--last",
+ metavar="COUNT",
+ type=int,
+ dest="limit",
+ help="Show only the last N presets. Implies --all",
+ )
model_filter = parser.add_mutually_exclusive_group()
model_filter.add_argument(
"--base",
diff --git a/src/dstack/_internal/cli/models/configurations.py b/src/dstack/_internal/cli/models/configurations.py
index b0e08942b..1372900bd 100644
--- a/src/dstack/_internal/cli/models/configurations.py
+++ b/src/dstack/_internal/cli/models/configurations.py
@@ -12,6 +12,9 @@
from dstack._internal.utils.json_schema import add_extra_schema_types
DEFAULT_CONCURRENCY = 8
+DEFAULT_INPUT_TOKENS = 1024
+DEFAULT_OUTPUT_TOKENS = 1024
+DEFAULT_BASELINE = False
class PresetModelRepo(CoreModel):
@@ -134,7 +137,7 @@ class PresetConfiguration(
)
),
] = None
- context_length: Annotated[
+ min_context_length: Annotated[
Optional[PositiveInt], Field(description="The minimum required context length")
] = None
max_trials: Annotated[
@@ -155,6 +158,34 @@ class PresetConfiguration(
)
),
] = None
+ input_tokens: Annotated[
+ Optional[PositiveInt],
+ Field(
+ description=(
+ "The number of input tokens per request used for benchmarks during"
+ f" preset creation. Defaults to `{DEFAULT_INPUT_TOKENS}`"
+ )
+ ),
+ ] = None
+ output_tokens: Annotated[
+ Optional[PositiveInt],
+ Field(
+ description=(
+ "The number of output tokens per request used for benchmarks during"
+ f" preset creation. Defaults to `{DEFAULT_OUTPUT_TOKENS}`"
+ )
+ ),
+ ] = None
+ baseline: Annotated[
+ Optional[bool],
+ Field(
+ description=(
+ "Whether the first trial must be a baseline that serves the model with the"
+ " serving framework's recommended defaults instead of an optimization attempt."
+ " Defaults to `false`"
+ )
+ ),
+ ] = None
gateway: Annotated[
Optional[Union[bool, EntityReference, str]],
Field(
@@ -173,6 +204,18 @@ class PresetConfiguration(
def effective_concurrency(self) -> int:
return self.concurrency if self.concurrency is not None else DEFAULT_CONCURRENCY
+ @property
+ def effective_input_tokens(self) -> int:
+ return self.input_tokens if self.input_tokens is not None else DEFAULT_INPUT_TOKENS
+
+ @property
+ def effective_output_tokens(self) -> int:
+ return self.output_tokens if self.output_tokens is not None else DEFAULT_OUTPUT_TOKENS
+
+ @property
+ def effective_baseline(self) -> bool:
+ return self.baseline if self.baseline is not None else DEFAULT_BASELINE
+
@root_validator(pre=True)
def apply_model_shorthand(cls, values: Any) -> Any:
if not isinstance(values, dict):
@@ -211,9 +254,12 @@ class PresetConstraints(CoreModel):
run_name_prefix: str
model: PresetModelSpec
- context_length: Optional[PositiveInt] = None
+ min_context_length: Optional[PositiveInt] = None
max_trials: PositiveInt
concurrency: PositiveInt
+ input_tokens: PositiveInt
+ output_tokens: PositiveInt
+ baseline: bool = False
fleets: list[str] = Field(min_items=1)
env: list[str] = []
diff --git a/src/dstack/_internal/cli/services/presets/agent.py b/src/dstack/_internal/cli/services/presets/agent.py
index 201c2ed35..6dd1d8bbb 100644
--- a/src/dstack/_internal/cli/services/presets/agent.py
+++ b/src/dstack/_internal/cli/services/presets/agent.py
@@ -423,6 +423,14 @@ async def _session_tailers(
offset_key="trials",
echo=agent_session.echo,
),
+ _RecordMirror(
+ source=workspace.verifications_path,
+ target=agent_session.verifications_path,
+ redacted_values=redacted_values,
+ offset_store=offset_store,
+ offset_key="verifications",
+ echo=agent_session.echo,
+ ),
]
tailer_tasks = [
asyncio.create_task(tailer.run()) for tailer in [progress_tailer, *record_mirrors]
diff --git a/src/dstack/_internal/cli/services/presets/apply.py b/src/dstack/_internal/cli/services/presets/apply.py
index 40495a357..e5b784916 100644
--- a/src/dstack/_internal/cli/services/presets/apply.py
+++ b/src/dstack/_internal/cli/services/presets/apply.py
@@ -8,6 +8,7 @@
from dstack._internal.cli.services.configurators.run import ServiceConfigurator
from dstack._internal.cli.services.presets.output import (
format_preset_benchmark,
+ format_preset_objective,
)
from dstack._internal.cli.services.presets.store import PresetStore
from dstack._internal.core.errors import CLIError
@@ -58,11 +59,11 @@ def _validate_preset_matches(preset: Preset, *, configuration: PresetConfigurati
service_model = preset.service.model
if service_model is None or service_model.name.lower() != model_name.lower():
raise CLIError(f"Preset {preset.id} does not serve {model_name}")
- if configuration.context_length is not None:
- if preset.context_length < configuration.context_length:
+ if configuration.min_context_length is not None:
+ if preset.context_length < configuration.min_context_length:
raise CLIError(
f"Preset {preset.id} does not support context length"
- f" {configuration.context_length}"
+ f" {configuration.min_context_length}"
)
if configuration.model.allows_variant_selection:
if preset.base.lower() != model_name.lower():
@@ -94,5 +95,7 @@ def _format_requested_model(configuration: PresetConfiguration) -> str:
def _format_selected_preset(preset: Preset) -> str:
- details = format_preset_benchmark(preset, verbose=True)
- return f"{escape(preset.id)} ([secondary]{details}[/])"
+ # The formatter dims its own keys; wrapping it again would flatten that.
+ # One line, so the objective and the result are joined rather than columned.
+ details = f"{format_preset_objective(preset)} {format_preset_benchmark(preset, verbose=True)}"
+ return f"{escape(preset.id)} ({details})"
diff --git a/src/dstack/_internal/cli/services/presets/create.py b/src/dstack/_internal/cli/services/presets/create.py
index cf14cfebf..6b6d98d5f 100644
--- a/src/dstack/_internal/cli/services/presets/create.py
+++ b/src/dstack/_internal/cli/services/presets/create.py
@@ -554,7 +554,10 @@ async def _create_preset(
workspace=setup.workspace,
token=token,
)
- prompt = get_preset_agent_system_prompt(user_prompt=setup.user_prompt)
+ prompt = get_preset_agent_system_prompt(
+ user_prompt=setup.user_prompt,
+ baseline=configuration.effective_baseline,
+ )
if setup.write_constraints:
if setup.user_prompt:
agent_session.write_user_prompt(setup.user_prompt)
@@ -849,9 +852,12 @@ def _build_constraints(
{
"run_name_prefix": build_name,
"model": json.loads(configuration.model.json(exclude_none=True)),
- "context_length": configuration.context_length,
+ "min_context_length": configuration.min_context_length,
"max_trials": configuration.max_trials,
"concurrency": configuration.effective_concurrency,
+ "input_tokens": configuration.effective_input_tokens,
+ "output_tokens": configuration.effective_output_tokens,
+ "baseline": configuration.effective_baseline,
"fleets": list(allowed_fleets),
"env": list(configuration.env),
}
diff --git a/src/dstack/_internal/cli/services/presets/output.py b/src/dstack/_internal/cli/services/presets/output.py
index 4a9da5252..e2552687b 100644
--- a/src/dstack/_internal/cli/services/presets/output.py
+++ b/src/dstack/_internal/cli/services/presets/output.py
@@ -12,7 +12,7 @@
_STATUS_DISPLAY = {
"ready": ("verified", "grey"),
- "running": ("clauding", "bold sea_green3"),
+ "running": ("trialing", "bold sea_green3"),
"verifying": ("verifying", "bold deep_sky_blue1"),
"interrupted": ("interrupted", "bold gold1"),
"failed": ("failed", "indian_red1"),
@@ -24,15 +24,45 @@ def _format_status(status: str) -> str:
return f"[{style}]{text}[/]" if style else text
-def _trials_exhausted(session: dict[str, Any]) -> bool:
+def _verifying(session: dict[str, Any]) -> bool:
+ """Whether the agent has moved on to the final service. Read from the session's
+ verification records rather than inferred from a spent trial budget, which
+ misses every run that stopped early. An attempt that failed still counts: the
+ agent is picking the next trial to verify, not trialing again."""
+ return isinstance(session.get("verification"), dict)
+
+
+_SPARK_BLOCKS = "▁▂▃▄▅▆▇█"
+
+
+def _format_trial_spark(session: Optional[dict[str, Any]]) -> str:
+ """One glyph per trial, scaled within the run: the shape of the search.
+ `·` marks a trial that produced no benchmark."""
+ if not isinstance(session, dict):
+ return ""
trials = session.get("trials")
- max_trials = session.get("max_trials")
- return (
- isinstance(trials, dict)
- and isinstance(max_trials, int)
- and isinstance(trials.get("count"), int)
- and trials["count"] >= max_trials
- )
+ series = trials.get("series") if isinstance(trials, dict) else None
+ if not isinstance(series, list) or not series:
+ return ""
+ values = [v for v in series if isinstance(v, (int, float))]
+ if not values:
+ return "·" * len(series)
+ low, high = min(values), max(values)
+ span = high - low
+ out = []
+ for value in series:
+ if not isinstance(value, (int, float)):
+ out.append("[indian_red1]·[/]")
+ continue
+ glyph = (
+ _SPARK_BLOCKS[-1]
+ if span <= 0
+ else _SPARK_BLOCKS[round((value - low) / span * (len(_SPARK_BLOCKS) - 1))]
+ )
+ # The best trial is the answer the run found; everything else is context.
+ style = "bold sea_green3" if value >= high else "secondary"
+ out.append(f"[{style}]{glyph}[/]")
+ return "".join(out)
def _format_trial_progress(session: Optional[dict[str, Any]]) -> str:
@@ -54,8 +84,14 @@ def print_presets(
presets: list[Preset],
sessions: Optional[list[dict[str, Any]]] = None,
verbose: bool = False,
+ all_presets: bool = False,
+ limit: Optional[int] = None,
) -> None:
- console.print(get_presets_table(presets, sessions=sessions, verbose=verbose))
+ console.print(
+ get_presets_table(
+ presets, sessions=sessions, verbose=verbose, all_presets=all_presets, limit=limit
+ )
+ )
console.print()
@@ -63,15 +99,22 @@ def get_presets_table(
presets: list[Preset],
sessions: Optional[list[dict[str, Any]]] = None,
verbose: bool = False,
+ all_presets: bool = False,
+ limit: Optional[int] = None,
) -> Table:
table = Table(box=None)
- table.add_column("BASE", no_wrap=True)
table.add_column("ID", no_wrap=True)
+ table.add_column("BASE", no_wrap=True, style="secondary")
table.add_column("RESOURCES" if verbose else "GPU", style="secondary")
- table.add_column("BENCHMARK", min_width=len("con=1"), overflow="fold")
+ # WORKLOAD is the test that was defined; BEST is the best trial under it.
+ table.add_column("WORKLOAD", no_wrap=True)
+ table.add_column("BEST", min_width=len("tps=1"), overflow="fold")
+ # The search shape, one glyph per trial. Unlabelled: it reads on sight.
+ table.add_column("", no_wrap=True)
table.add_column("STATUS", no_wrap=True)
table.add_column("SUBMITTED", no_wrap=True, style="secondary")
- table.add_column("NAME", no_wrap=True, style="secondary")
+ if verbose:
+ table.add_column("NAME", no_wrap=True, style="secondary")
presets_by_base: dict[str, list[Preset]] = defaultdict(list)
repo_to_base: dict[str, str] = {}
for preset in presets:
@@ -87,23 +130,32 @@ def get_presets_table(
model = str(session.get("model") or "unknown")
sessions_by_model[repo_to_base.get(model, model)].append(session)
- for base in sorted({*presets_by_base, *sessions_by_model}, key=str.lower):
- add_row_from_dict(table, {"BASE": base})
- # Newest first within a group, as in `dstack ps`.
- for preset in sorted(
- presets_by_base.get(base, []), key=lambda p: p.created_at, reverse=True
- ):
- _add_preset(table, preset, verbose=verbose, creation=creations_by_id.get(preset.id))
- for session in sorted(
- sessions_by_model.get(base, []),
- key=lambda s: str(s.get("created_at") or ""),
- reverse=True,
- ):
- _add_session(table, session)
+ # One flat list, newest first, as in `dstack ps`. The base is a column, so
+ # runs of different models still sort together by when they were submitted.
+ # Same contract as `dstack ps`: only active by default, or the single most
+ # recent row when nothing is active. `-a` and `-n` show everything.
+ rows: list[tuple[str, Any, bool]] = []
+ for preset_list in presets_by_base.values():
+ rows += [(preset.created_at.isoformat(), preset, True) for preset in preset_list]
+ for session_list in sessions_by_model.values():
+ rows += [
+ (str(session.get("created_at") or ""), session, False) for session in session_list
+ ]
+ rows.sort(key=lambda r: r[0], reverse=True)
+ only_active = not all_presets and limit is None
+ if only_active:
+ active = [r for r in rows if not r[2] and str(r[1].get("status")) == "running"]
+ rows = active or rows[:1]
+
+ for _, item, is_preset in rows[:limit] if limit is not None else rows:
+ if is_preset:
+ _add_preset(table, item, verbose=verbose, creation=creations_by_id.get(item.id))
+ else:
+ _add_session(table, item, verbose=verbose)
return table
-def _add_session(table: Table, session: dict[str, Any]) -> None:
+def _add_session(table: Table, session: dict[str, Any], *, verbose: bool = False) -> None:
created = ""
created_at = session.get("created_at")
if isinstance(created_at, str):
@@ -114,29 +166,50 @@ def _add_session(table: Table, session: dict[str, Any]) -> None:
benchmark = ""
gpu = ""
status_key = str(session.get("status", ""))
- if status_key == "running" and _trials_exhausted(session):
- # The trial budget is spent, so the agent is deploying and verifying
- # the final service.
+ if status_key == "running" and _verifying(session):
status_key = "verifying"
status = _format_status(status_key) + _format_trial_progress(session)
trials = session.get("trials")
- if isinstance(trials, dict):
- best = trials.get("best")
- if isinstance(best, dict):
- parts = ["best trial:"]
- if best.get("concurrency"):
- parts.append(f"con={best['concurrency']}")
- parts.append(f"{_format_number(best['tok_s'])} tok/s")
+ best = trials.get("best") if isinstance(trials, dict) else None
+ if isinstance(best, dict):
+ gpu = best.get("gpu") or ""
+ constraints = session.get("constraints") or {}
+ parts = []
+ objective = []
+ if constraints.get("input_tokens") and constraints.get("output_tokens"):
+ objective.append(
+ f"[secondary]io=[/]{constraints['input_tokens']}/{constraints['output_tokens']}"
+ )
+ concurrency = (best or {}).get("concurrency") or constraints.get("concurrency")
+ if concurrency:
+ objective.append(f"[secondary]con=[/]{concurrency}")
+ # Stays empty until a trial has produced a benchmark: a run that has measured
+ # nothing yet has no best, and `n/a` is noise in a column of numbers.
+ if isinstance(best, dict):
+ tps = _format_number(best["tok_s"])
+ if objective:
+ parts.append(f"[secondary]tps=[/]{tps}")
+ ttft_ms = best.get("ttft_ms")
+ if isinstance(ttft_ms, (int, float)):
+ parts.append(f"[secondary]ttft=[/]{_format_number(ttft_ms)}")
+ context_length = best.get("context_length")
+ if isinstance(context_length, int):
+ parts.append(f"[secondary]ctx=[/]{_format_token_count(context_length)}")
benchmark = " ".join(parts)
- gpu = best.get("gpu") or ""
+ else:
+ benchmark = f"[secondary]con=[/]{best.get('concurrency')} [secondary]tps=[/]{tps}"
+ benchmark = benchmark.strip()
add_row_from_dict(
table,
{
"ID": str(session.get("id", "")),
+ "BASE": str(session.get("model") or ""),
"NAME": str(session.get("name") or ""),
"GPU": gpu,
"RESOURCES": gpu,
- "BENCHMARK": benchmark,
+ "": _format_trial_spark(session),
+ "WORKLOAD": " ".join(objective),
+ "BEST": benchmark,
"STATUS": status,
"SUBMITTED": created,
},
@@ -156,8 +229,11 @@ def _add_preset(
"ID": preset.id,
"NAME": preset.name or "",
column: _format_resources(groups[0].resources, verbose=verbose),
+ "BASE": preset.base,
"STATUS": _format_status("ready") + _format_trial_progress(creation),
- "BENCHMARK": format_preset_benchmark(preset, verbose=verbose),
+ "": _format_trial_spark(creation),
+ "WORKLOAD": format_preset_objective(preset),
+ "BEST": format_preset_benchmark(preset, verbose=verbose),
"SUBMITTED": pretty_date(preset.created_at),
}
if verbose and preset.model != preset.base:
@@ -175,18 +251,28 @@ def _add_preset(
)
+def format_preset_objective(preset: Preset) -> str:
+ """What the benchmark ran. The context is not part of it: it is a capability of
+ the configuration, measured separately, and belongs next to the results."""
+ workload = preset.validations[0].benchmark.workload
+ return (
+ f"[secondary]io=[/]{workload.input_tokens}/{workload.output_tokens} "
+ f"[secondary]con=[/]{workload.concurrency}"
+ )
+
+
def format_preset_benchmark(preset: Preset, *, verbose: bool = False) -> str:
benchmark = preset.validations[0].benchmark
- workload = benchmark.workload
metrics = benchmark.metrics
output_tokens_per_second = metrics.total_output_tokens / metrics.duration_seconds
+ # The workload and context define the number, so they are always shown next
+ # to it: two presets are comparable only when all three match.
parts = [
- f"con={workload.concurrency}",
- f"{_format_number(output_tokens_per_second)} tok/s",
- f"TTFT {_format_latency(metrics.ttft_ms.p50)}",
+ f"[secondary]tps=[/]{_format_number(output_tokens_per_second)}",
+ # Always milliseconds: mixed units make two rows hard to compare.
+ f"[secondary]ttft=[/]{_format_number(metrics.ttft_ms.p50)}",
+ f"[secondary]ctx=[/]{_format_token_count(preset.context_length)}",
]
- if verbose:
- parts.insert(0, f"ctx={_format_token_count(preset.context_length)}")
return " ".join(parts)
diff --git a/src/dstack/_internal/cli/services/presets/prompt.py b/src/dstack/_internal/cli/services/presets/prompt.py
index 0d0052c32..e565fc2fd 100644
--- a/src/dstack/_internal/cli/services/presets/prompt.py
+++ b/src/dstack/_internal/cli/services/presets/prompt.py
@@ -11,26 +11,40 @@
# the document; this module only applies the rule.
_DIRECTIVE_PATTERN = re.compile(r"", re.DOTALL)
+# `` is a note for maintainers and is dropped before the agent sees
+# the document. Any other comment is left alone, so that a plain `` or a
+# malformed directive stays visible instead of disappearing silently.
+_NOTE_PATTERN = re.compile(r"\n?", re.DOTALL)
+
# TODO: reintroduce a `# Resume` section in system_prompt.md once session resume
# (seeded from `runs.jsonl` and `trials.jsonl`) is designed.
-def get_preset_agent_system_prompt(user_prompt: Optional[str] = None) -> str:
+def get_preset_agent_system_prompt(
+ user_prompt: Optional[str] = None,
+ baseline: bool = False,
+) -> str:
text = _SYSTEM_PROMPT_PATH.read_text(encoding="utf-8").strip()
- variables = {"prompt": user_prompt.strip() if user_prompt else None}
- applied = 0
+ variables = {
+ "prompt": user_prompt.strip() if user_prompt else None,
+ # Rendered for its presence only; the directive body must not interpolate it.
+ "baseline": "on" if baseline else None,
+ }
+ applied: set[str] = set()
def substitute(match: re.Match) -> str:
- nonlocal applied
name, content = match.group(1), match.group(2)
if name not in variables:
raise CLIError(f"Unknown variable {name!r} in the agent system prompt")
value = variables[name]
if not value:
return ""
- applied += 1
+ applied.add(name)
return content.replace("{" + name + "}", value)
- rendered = _DIRECTIVE_PATTERN.sub(substitute, text)
- if variables["prompt"] and not applied:
+ rendered = _NOTE_PATTERN.sub("", _DIRECTIVE_PATTERN.sub(substitute, text))
+ if variables["prompt"] and "prompt" not in applied:
raise CLIError("The agent system prompt has no place for the user prompt")
+ for name, value in variables.items():
+ if value and name not in applied:
+ raise CLIError(f"The agent system prompt has no place for {name!r}")
return re.sub(r"\n{3,}", "\n\n", rendered)
diff --git a/src/dstack/_internal/cli/services/presets/resources/system_prompt.md b/src/dstack/_internal/cli/services/presets/resources/system_prompt.md
index 238709970..816e3af3e 100644
--- a/src/dstack/_internal/cli/services/presets/resources/system_prompt.md
+++ b/src/dstack/_internal/cli/services/presets/resources/system_prompt.md
@@ -20,11 +20,17 @@ Field semantics:
different precision or quantization, or another trusted compatible repo.
The client-facing model name of the final service is `model.name` when
set, otherwise `model.repo` or `model.base`.
-- `context_length`: the minimum context length the selected repo/path and
+- `min_context_length`: the minimum context length the selected repo/path and
the final service must support. `null` means no minimum is required.
- `max_trials`: the maximum number of trials in this session.
- `concurrency`: the number of simultaneous requests for every benchmark in
this session. It is fixed so that benchmark results are comparable.
+
+- `input_tokens`, `output_tokens`: the request shape for every benchmark in
+ this session. They are fixed for the same reason.
+- `baseline`: whether the first trial must be a baseline rather than an
+ optimization attempt; see `# Trials`.
- `fleets`: use these existing `dstack` fleets only. Do not create, delete,
apply, or edit fleets.
- `env`: the environment variable names available to runs; the values are
@@ -72,6 +78,8 @@ Files you are expected to maintain in the workspace root:
- `progress.jsonl`: progress messages, written through the `progress` helper;
see `# Progress`.
- `trials.jsonl`: the append-only record of completed trials; see `# Trials`.
+- `verifications.jsonl`: the append-only record of final service attempts;
+ see `# Final Service`.
- `final_report.json`: the final report; see `# Final Report`.
You may create any other working files (run YAML files, benchmark output,
@@ -135,9 +143,13 @@ so-called trials, where each trial is formed around a substantive idea on
how to get better performance than the previous trials. Do not consider
P/D disaggregation setups yet.
-
-
+
+
Trial ideas must not rely only on what you already know. Research how to
get the best performance for the chosen model, serving framework, and
hardware in trustworthy sources. Start with these:
@@ -174,10 +186,10 @@ commands, and log it together with the corresponding benchmark results (see
be skipped in one case only: you failed to make the configuration run at
all — a failed trial.
-Each `trials.jsonl` record is one JSON line with exactly three fields:
+Each `trials.jsonl` record is one JSON line with exactly four fields:
```
-{"task": {...}, "resources": {...}, "benchmark": {...}}
+{"task": {...}, "resources": {...}, "context_length": ..., "benchmark": {...}}
```
- `task`: the compiled `dstack` task configuration described above, as JSON.
@@ -193,6 +205,8 @@ Each `trials.jsonl` record is one JSON line with exactly three fields:
`dstack run get --json`, converting MiB values to GB and the
`gpus` list into one `gpu` object with the GPU `name`, per-GPU `memory`,
and `count`.
+- `context_length`: the largest context the trial's configuration handles,
+ found as described in `## Benchmark`; `null` only for a failed trial.
- `benchmark`: the trial benchmark (see `## Benchmark` for the structure);
`null` only for a failed trial.
@@ -216,14 +230,19 @@ no trials remain. In that case, log the failure to `final_report.json` (see
## Benchmark
During trials, run benchmarks via SSH inside the task, directly against the
-serving engine: use `concurrency` from `constraints.json` and measure all
-trials the same way so that their results are comparable with each other.
-In trial benchmarks too, all measured requests must succeed.
+serving engine: use `concurrency`, `input_tokens`, and `output_tokens` from
+`constraints.json` and measure all trials the same way so that their results
+are comparable with each other.
+Before any benchmark, make sure the engine has none of the previous benchmark's
+prompts in its cache, or ensure the benchmark uses a different seed. Otherwise
+the benchmark will depend on what has been cached by the previous benchmark.
+
Before any benchmark — a trial one or the final one — warm the engine up by
verifying that the model works as expected: send real requests and check
-the responses, including reasoning output when the model supports it. These
-verification requests are never part of the measured metrics.
+the responses, including reasoning output when the model supports it. These verification requests are never part of the measured metrics.
+
+All verification and benchmark requests must succeed.
Record every benchmark using the following structure and field names —
trial benchmarks in `trials.jsonl`, the final benchmark as
@@ -250,8 +269,14 @@ invocation. For the final benchmark, run it with streaming responses, set
`workload.concurrency` to `concurrency` from `constraints.json`, produce
every field of the structure, and calculate all metrics from the
`num_requests` measured requests only — exclude setup, health-check, and
-warmup requests; all measured requests must succeed. Never invent missing
-values.
+warmup requests. Never invent missing values.
+
+After each benchmark, find the largest context the configuration handles by
+sending real requests, and record it: for a trial, as the `context_length`
+field of its `trials.jsonl` record; for the final benchmark, as
+`final_report.json.context_length`. Stopping at the required minimum is not
+enough.
+
# Task Usage
@@ -302,17 +327,34 @@ the trial. Set the service `model` name to the client-facing model name from
serve requests. If the service never passes the probe, treat that as a real
failure of the configuration, not something to work around by removing `model`.
+Record every attempt in `verifications.jsonl`, append-only: one line
+immediately after submitting the service, one when the attempt ends (values
+are illustrative):
+
+```json
+{"trial": 3, "run_name": "qwen-preset-2", "status": "verifying"}
+{"trial": 3, "run_name": "qwen-preset-2", "status": "failed", "reason": "..."}
+{"trial": 2, "run_name": "qwen-preset-3", "status": "verifying"}
+{"trial": 2, "run_name": "qwen-preset-3", "status": "verified"}
+```
+
+`trial` is the 1-based line number of that trial in `trials.jsonl`. Keep
+`reason` to one sentence.
+
Before the final benchmark, verify the model through the service: send real
requests using the client-facing model name and check that the model works
as it should, including reasoning output when the model supports it. This
-verification also warms the service up. Only then run the final benchmark
-(see `## Benchmark`). When verifying or benchmarking the service, use its
+verification also warms the service up. When verifying the service, use its
`service.url` reported by `dstack run get --json`, along with
`DSTACK_TOKEN` as the bearer token. If `service.url` is a relative path,
prepend `DSTACK_SERVER_URL` to build the absolute URL.
-During the service verification, test the context length the service
-actually supports and report it as `final_report.json.context_length`.
+Only then run the final benchmark (see `## Benchmark`). Unlike the
+verification above, do not run it through `service.url`: run it via SSH inside
+the service replica, directly against the serving engine, the same way as the
+trial benchmarks so that the results are comparable with each other. Attach to
+the service with `dstack attach `, which enables `ssh `
+into the replica.
If the service or its benchmark cannot be completed, stop that service,
pick the next-best trial, and repeat, until a service is verified or there
@@ -348,7 +390,8 @@ On success, include exactly:
- `service_yaml`: the full YAML of the verified final service
- `base`: the base model repo, determined by the rules below
- `model`: the exact repo/path loaded by the final service command
-- `context_length`: the context length verified for the final service
+- `context_length`: the largest context verified for the final service, as
+ described in `## Benchmark`
- `benchmark`: the final service benchmark described in `## Benchmark`
Set `final_report.json.base` as follows:
diff --git a/src/dstack/_internal/cli/services/presets/session.py b/src/dstack/_internal/cli/services/presets/session.py
index 72ec86e0f..e1b63a0ab 100644
--- a/src/dstack/_internal/cli/services/presets/session.py
+++ b/src/dstack/_internal/cli/services/presets/session.py
@@ -30,6 +30,7 @@
_PROGRESS_FILENAME = "progress.jsonl"
_RUNS_FILENAME = "runs.jsonl"
_TRIALS_FILENAME = "trials.jsonl"
+_VERIFICATIONS_FILENAME = "verifications.jsonl"
_CONSTRAINTS_FILENAME = "constraints.json"
_FINAL_REPORT_FILENAME = "final_report.json"
_SESSION_FILENAME = "session.json"
@@ -68,6 +69,10 @@ def runs_path(self) -> Path:
def trials_path(self) -> Path:
return self.path / _TRIALS_FILENAME
+ @property
+ def verifications_path(self) -> Path:
+ return self.path / _VERIFICATIONS_FILENAME
+
def write_prompt(self, prompt: str) -> None:
_write_private_text(self.path / "prompt.md", prompt + "\n")
@@ -427,10 +432,42 @@ def list_agent_sessions() -> list[dict[str, Any]]:
entry["name"] = claimed_session_name(manifest)
entry["status"] = status
entry["trials"] = _summarize_session_trials(path / _TRIALS_FILENAME)
+ entry["verification"] = _read_last_session_verification(path / _VERIFICATIONS_FILENAME)
+ entry["constraints"] = _read_session_constraints(path)
entries.append(entry)
return entries
+def _read_session_constraints(path: Path) -> dict[str, Any]:
+ """The objective the session was given. It lives in the agent workspace, which
+ exists for as long as the session is unfinished — exactly the sessions shown."""
+ try:
+ data = json.loads(
+ (path / "workspace" / "w" / _CONSTRAINTS_FILENAME).read_text(encoding="utf-8")
+ )
+ except (OSError, ValueError):
+ return {}
+ return data if isinstance(data, dict) else {}
+
+
+def _read_last_session_verification(path: Path) -> Optional[dict[str, Any]]:
+ """The final service attempt in flight or last finished, from the session's
+ mirrored verification records. The last line wins: the agent appends one when
+ an attempt starts and another when it ends."""
+ try:
+ lines = path.read_text(encoding="utf-8").splitlines()
+ except OSError:
+ return None
+ for line in reversed(lines):
+ try:
+ record = json.loads(line)
+ except json.JSONDecodeError:
+ continue
+ if isinstance(record, dict) and isinstance(record.get("status"), str):
+ return record
+ return None
+
+
def _summarize_session_trials(path: Path) -> Optional[dict[str, Any]]:
"""Best-so-far summary from a session's mirrored trial records."""
try:
@@ -439,6 +476,8 @@ def _summarize_session_trials(path: Path) -> Optional[dict[str, Any]]:
lines = []
count = 0
best: Optional[dict[str, Any]] = None
+ # One entry per trial in order, `None` for a trial that produced no benchmark.
+ series: list[Optional[float]] = []
for line in lines:
try:
record = json.loads(line)
@@ -451,16 +490,20 @@ def _summarize_session_trials(path: Path) -> Optional[dict[str, Any]]:
count += 1
benchmark = record.get("benchmark")
if not isinstance(benchmark, dict):
+ series.append(None)
continue
metrics = benchmark.get("metrics") or {}
workload = benchmark.get("workload") or {}
duration = metrics.get("duration_seconds")
tokens = metrics.get("total_output_tokens")
if not isinstance(duration, (int, float)) or duration <= 0:
+ series.append(None)
continue
if not isinstance(tokens, (int, float)):
+ series.append(None)
continue
tok_s = tokens / duration
+ series.append(tok_s)
if best is None or tok_s > best["tok_s"]:
resources = record.get("resources") or {}
gpu = resources.get("gpu") if isinstance(resources, dict) else None
@@ -471,12 +514,16 @@ def _summarize_session_trials(path: Path) -> Optional[dict[str, Any]]:
gpu_text += f":{gpu['memory']}"
if gpu.get("count"):
gpu_text += f":{gpu['count']}"
+ ttft = (metrics.get("ttft_ms") or {}).get("p50")
+ context_length = record.get("context_length")
best = {
"tok_s": tok_s,
+ "ttft_ms": ttft if isinstance(ttft, (int, float)) else None,
+ "context_length": context_length if isinstance(context_length, int) else None,
"concurrency": workload.get("concurrency"),
"gpu": gpu_text,
}
- return {"count": count, "best": best}
+ return {"count": count, "best": best, "series": series}
def print_preset_progress(message: str, *, agent_session: PresetAgentSession) -> None:
diff --git a/src/dstack/_internal/cli/services/presets/verify.py b/src/dstack/_internal/cli/services/presets/verify.py
index 983b397a4..7746a5eda 100644
--- a/src/dstack/_internal/cli/services/presets/verify.py
+++ b/src/dstack/_internal/cli/services/presets/verify.py
@@ -92,8 +92,8 @@ def build_verified_preset(
elif report.model != preset_configuration.model.exact_repo:
raise CLIError("Claude changed an exact model request")
if (
- preset_configuration.context_length is not None
- and report.context_length < preset_configuration.context_length
+ preset_configuration.min_context_length is not None
+ and report.context_length < preset_configuration.min_context_length
):
raise CLIError("Claude final service does not meet the requested context length")
diff --git a/src/dstack/_internal/cli/services/presets/workspace.py b/src/dstack/_internal/cli/services/presets/workspace.py
index 504a9baf0..54b24abf5 100644
--- a/src/dstack/_internal/cli/services/presets/workspace.py
+++ b/src/dstack/_internal/cli/services/presets/workspace.py
@@ -18,6 +18,7 @@
_PROGRESS_FILENAME,
_RUNS_FILENAME,
_TRIALS_FILENAME,
+ _VERIFICATIONS_FILENAME,
PresetAgentSession,
)
from dstack._internal.compat import IS_WINDOWS
@@ -54,6 +55,10 @@ def runs_path(self) -> Path:
def trials_path(self) -> Path:
return self.path / _TRIALS_FILENAME
+ @property
+ def verifications_path(self) -> Path:
+ return self.path / _VERIFICATIONS_FILENAME
+
@property
def constraints_path(self) -> Path:
return self.path / _CONSTRAINTS_FILENAME
@@ -170,6 +175,7 @@ def _prepare_workspace(workspace: PresetAgentWorkspace) -> None:
workspace.progress_path,
workspace.runs_path,
workspace.trials_path,
+ workspace.verifications_path,
]:
path.touch()
workspace.bin_path.mkdir()
diff --git a/src/tests/_internal/cli/commands/test_preset.py b/src/tests/_internal/cli/commands/test_preset.py
index 5f8c17c65..e2161a7dd 100644
--- a/src/tests/_internal/cli/commands/test_preset.py
+++ b/src/tests/_internal/cli/commands/test_preset.py
@@ -109,18 +109,21 @@ def test_lists_presets_without_api_client(self, tmp_path):
assert "8f3a12c4" in output
# The repo row is shown only in verbose mode.
assert "repo=community/Qwen3.5-27B-GPTQ-Int4" not in output
- # The context column is shown only in verbose mode.
+ # The context column is shown only in verbose mode, but the benchmark
+ # always carries the context and workload that define its number.
assert "CONTEXT" not in output
- assert "BENCHMARK" in output
- assert "32K" not in output
+ assert "WORKLOAD" in output
+ assert "BEST" in output
+ assert "ctx=32K" in "".join(output.split())
+ assert "io=1024/128" in "".join(output.split())
assert "42.1" in output
assert "con=1" in "".join(output.split())
- assert "tok/s" in output
- assert "TTFT" in output
- assert "108ms" in output
+ assert "tps=" in "".join(output.split())
+ assert "ttft=" in "".join(output.split())
+ assert "ttft=[/]108" in "".join(output.split()) or "ttft=108" in "".join(output.split())
assert "A6000:48GB:1" not in output
- def test_verbose_list_adds_repo_and_context(self, tmp_path):
+ def test_verbose_list_adds_repo(self, tmp_path):
preset = get_preset()
PresetStore(tmp_path / ".dstack" / "presets").save(preset)
@@ -130,7 +133,7 @@ def test_verbose_list_adds_repo_and_context(self, tmp_path):
).split()
)
- # Verbose adds only the repo and the ctx= benchmark prefix.
+ # Verbose adds only the repo row.
assert "repo=community/Qwen3.5-27B-GPTQ-Int4" in joined_verbose
assert "ctx=32K" in joined_verbose
assert "con=1" in joined_verbose
diff --git a/src/tests/_internal/cli/services/presets/test_agent.py b/src/tests/_internal/cli/services/presets/test_agent.py
index 9e96c4412..7cb7976cf 100644
--- a/src/tests/_internal/cli/services/presets/test_agent.py
+++ b/src/tests/_internal/cli/services/presets/test_agent.py
@@ -29,6 +29,7 @@
)
from dstack._internal.cli.services.presets.session import (
PresetAgentSession,
+ _read_last_session_verification,
_summarize_session_trials,
create_preset_agent_session,
load_resumable_agent_session,
@@ -872,7 +873,49 @@ def test_counts_records_even_when_trials_share_a_task(self, tmp_path):
# 4 records = 4 trials: one long-lived task commonly hosts several
# trials, so shared task names must not collapse the count.
assert summary["count"] == 4
- assert summary["best"] == {"tok_s": 2300.0, "concurrency": 8, "gpu": "A40:48GB:1"}
+ assert summary["best"] == {
+ "tok_s": 2300.0,
+ "ttft_ms": None,
+ "context_length": None,
+ "concurrency": 8,
+ "gpu": "A40:48GB:1",
+ }
+
+
+class TestReadLastSessionVerification:
+ def test_last_record_wins_and_a_missing_file_is_not_verifying(self, tmp_path):
+ path = tmp_path / "verifications.jsonl"
+
+ assert _read_last_session_verification(path) is None
+
+ path.write_text(
+ "\n".join(
+ json.dumps(entry)
+ for entry in [
+ {"trial": 3, "run_name": "p-2", "status": "verifying"},
+ {"trial": 3, "run_name": "p-2", "status": "failed", "reason": "probe"},
+ {"trial": 2, "run_name": "p-3", "status": "verifying"},
+ ]
+ )
+ + "\n"
+ )
+
+ assert _read_last_session_verification(path) == {
+ "trial": 2,
+ "run_name": "p-3",
+ "status": "verifying",
+ }
+
+ def test_skips_partial_trailing_lines(self, tmp_path):
+ # The mirror appends as the agent writes, so the file can be read
+ # mid-line.
+ path = tmp_path / "verifications.jsonl"
+ path.write_text(
+ json.dumps({"trial": 1, "run_name": "p-2", "status": "verifying"})
+ + '\n{"trial": 1, "run_na'
+ )
+
+ assert _read_last_session_verification(path)["status"] == "verifying"
class TestFileLineReader:
diff --git a/src/tests/_internal/cli/services/presets/test_apply.py b/src/tests/_internal/cli/services/presets/test_apply.py
index 3eae3cf69..8105c776e 100644
--- a/src/tests/_internal/cli/services/presets/test_apply.py
+++ b/src/tests/_internal/cli/services/presets/test_apply.py
@@ -22,7 +22,7 @@ def test_accepts_matching_base_model_and_context(self):
configuration = PresetConfiguration(
name="qwen",
model={"base": "Qwen/Qwen3.5-27B"},
- context_length=8192,
+ min_context_length=8192,
)
_validate_preset_matches(preset, configuration=configuration)
@@ -32,7 +32,7 @@ def test_rejects_insufficient_context(self):
configuration = PresetConfiguration(
name="qwen",
model={"base": "Qwen/Qwen3.5-27B"},
- context_length=8192,
+ min_context_length=8192,
)
with pytest.raises(CLIError, match="context length"):
@@ -124,7 +124,7 @@ def test_applies_the_referenced_preset(self, monkeypatch):
configurator_args=service_args,
plan_properties={
"Model": "Qwen/Qwen3.5-27B ([secondary]base[/])",
- "Preset": "8f3a12c4 ([secondary]ctx=32K con=1 42.1 tok/s TTFT 108ms[/])",
+ "Preset": "8f3a12c4 ([secondary]io=[/]1024/128 [secondary]con=[/]1 [secondary]tps=[/]42.1 [secondary]ttft=[/]108 [secondary]ctx=[/]32K)",
},
)
diff --git a/src/tests/_internal/cli/services/presets/test_create.py b/src/tests/_internal/cli/services/presets/test_create.py
index e87e589f5..7dc289bdf 100644
--- a/src/tests/_internal/cli/services/presets/test_create.py
+++ b/src/tests/_internal/cli/services/presets/test_create.py
@@ -88,7 +88,7 @@ def creation_context(tmp_path, monkeypatch):
configuration = PresetConfiguration(
name="qwen-build",
model={"base": "Qwen/Qwen3.5-27B"},
- context_length=8192,
+ min_context_length=8192,
max_trials=1,
fleets=["gpu-fleet"],
env={"LICENSE": "license-secret", "TOKENIZERS_PARALLELISM": "false"},
@@ -96,7 +96,7 @@ def creation_context(tmp_path, monkeypatch):
source_configuration = PresetConfiguration(
name="qwen-build",
model={"base": "Qwen/Qwen3.5-27B"},
- context_length=8192,
+ min_context_length=8192,
max_trials=1,
fleets=["gpu-fleet"],
env=["LICENSE", "TOKENIZERS_PARALLELISM=false"],
@@ -446,9 +446,12 @@ def test_renders_all_fields_with_explicit_nulls_and_defaults(self):
assert json.loads(text) == {
"run_name_prefix": "qwen-abc123",
"model": {"base": "Qwen/Qwen3-32B"},
- "context_length": None,
+ "min_context_length": None,
"max_trials": 3,
"concurrency": 8,
+ "input_tokens": 1024,
+ "output_tokens": 1024,
+ "baseline": False,
"fleets": ["gpu-fleet"],
"env": ["HF_TOKEN"],
}
@@ -457,7 +460,7 @@ def test_renders_configured_values(self):
configuration = PresetConfiguration(
name="qwen",
model={"repo": "Qwen/Qwen3-32B-AWQ", "name": "qwen3"},
- context_length=32768,
+ min_context_length=32768,
max_trials=10,
concurrency=16,
)
@@ -471,7 +474,7 @@ def test_renders_configured_values(self):
)
assert data["model"] == {"repo": "Qwen/Qwen3-32B-AWQ", "name": "qwen3"}
- assert data["context_length"] == 32768
+ assert data["min_context_length"] == 32768
assert data["max_trials"] == 10
assert data["concurrency"] == 16
assert data["fleets"] == ["a", "b"]
diff --git a/src/tests/_internal/cli/services/presets/test_output.py b/src/tests/_internal/cli/services/presets/test_output.py
index 7e76dbe06..b966ecb23 100644
--- a/src/tests/_internal/cli/services/presets/test_output.py
+++ b/src/tests/_internal/cli/services/presets/test_output.py
@@ -33,8 +33,12 @@ def test_formats_second_scale_ttft_without_scientific_notation(self):
output = output_module.format_preset_benchmark(preset, verbose=True)
- assert output.startswith("ctx=32K ")
- assert "TTFT 8.15s" in output
+ assert output.startswith("[secondary]tps=[/]42.1 ")
+ assert output_module.format_preset_objective(preset).startswith(
+ "[secondary]io=[/]1024/128 "
+ )
+ assert "[secondary]ctx=[/]32K" in output
+ assert "[secondary]ttft=[/]8151" in output
assert "e+03" not in output
@@ -60,7 +64,7 @@ def test_prints_submitted_column(self, monkeypatch):
def _session_row(session: dict) -> dict:
table = Table(box=None)
- for column in ("BASE", "ID", "GPU", "BENCHMARK", "STATUS", "SUBMITTED"):
+ for column in ("BASE", "ID", "GPU", "WORKLOAD", "BEST", "", "STATUS", "SUBMITTED"):
table.add_column(column)
_add_session(table, session)
return {
@@ -82,8 +86,8 @@ def test_shows_progress_after_status_and_best_benchmark(self):
}
)
- assert row["STATUS"] == "[bold sea_green3]clauding[/] [secondary](2/3)[/]"
- assert row["BENCHMARK"] == "best trial: con=8 2339 tok/s"
+ assert row["STATUS"] == "[bold sea_green3]trialing[/] [secondary](2/3)[/]"
+ assert row["BEST"].startswith("[secondary]tps=[/]2339")
assert row["GPU"] == "A40:48GB:1"
def test_shows_zero_progress_without_benchmark(self):
@@ -91,8 +95,7 @@ def test_shows_zero_progress_without_benchmark(self):
{"id": "ab12cd34", "status": "running", "max_trials": 3, "trials": {"count": 0}}
)
- assert row["STATUS"] == "[bold sea_green3]clauding[/] [secondary](0/3)[/]"
- assert row["BENCHMARK"] == ""
+ assert row["STATUS"] == "[bold sea_green3]trialing[/] [secondary](0/3)[/]"
def test_omits_progress_without_trials_data(self):
row = _session_row({"id": "ab12cd34", "status": "interrupted"})
@@ -105,8 +108,8 @@ def test_counts_without_max_trials(self):
assert row["STATUS"] == "[bold gold1]interrupted[/] [secondary](2)[/]"
-class TestGroupOrdering:
- def test_sorts_presets_and_sessions_newest_first(self, monkeypatch):
+class TestOrdering:
+ def test_sorts_all_rows_newest_first_without_grouping(self, monkeypatch):
buffer = StringIO()
monkeypatch.setattr(output_module, "console", plain_console(buffer, width=200))
old = get_preset()
@@ -129,8 +132,20 @@ def test_sorts_presets_and_sessions_newest_first(self, monkeypatch):
output_module.print_presets([old, new], sessions=sessions)
text = buffer.getvalue()
+ # Same contract as `dstack ps`: nothing is active here, so exactly one
+ # row is shown, the most recent.
+ assert "bbbbbbbb" in text
+ assert "aaaaaaaa" not in text
+ assert "11aa22bb" not in text and old.id not in text
+
+ buffer.truncate(0)
+ buffer.seek(0)
+ output_module.print_presets([old, new], sessions=sessions, all_presets=True)
+ text = buffer.getvalue()
+
+ # With -a: one flat list, newest first, presets and sessions interleaved.
+ assert text.index("bbbbbbbb") < text.index("aaaaaaaa") < text.index("11aa22bb")
assert text.index("11aa22bb") < text.index(old.id)
- assert text.index(old.id) < text.index("bbbbbbbb") < text.index("aaaaaaaa")
class TestDoneProgress:
@@ -151,21 +166,42 @@ def test_completed_creation_decorates_preset_row_without_extra_session_row(self,
output_module.print_presets([preset], sessions=sessions)
text = buffer.getvalue()
- assert "verified (3/4)" in text
+ assert "(3/4)" in text
+ assert "verified" in text
assert text.count(preset.id) == 1
class TestVerifyingStatus:
- def test_running_session_with_exhausted_trials_shows_verifying(self):
+ def test_running_session_with_a_verification_record_shows_verifying(self):
row = _session_row(
- {"id": "ab12cd34", "status": "running", "max_trials": 2, "trials": {"count": 2}}
+ {
+ "id": "ab12cd34",
+ "status": "running",
+ "max_trials": 10,
+ "trials": {"count": 6},
+ "verification": {"trial": 3, "run_name": "p-2", "status": "verifying"},
+ }
)
- assert row["STATUS"] == "[bold deep_sky_blue1]verifying[/] [secondary](2/2)[/]"
+ # The trial budget is not spent, which the old inference required.
+ assert row["STATUS"] == "[bold deep_sky_blue1]verifying[/] [secondary](6/10)[/]"
- def test_running_session_with_remaining_trials_stays_clauding(self):
+ def test_a_failed_attempt_still_counts_as_verifying(self):
row = _session_row(
- {"id": "ab12cd34", "status": "running", "max_trials": 2, "trials": {"count": 1}}
+ {
+ "id": "ab12cd34",
+ "status": "running",
+ "max_trials": 2,
+ "trials": {"count": 2},
+ "verification": {"trial": 3, "status": "failed", "reason": "probe never passed"},
+ }
+ )
+
+ assert row["STATUS"].startswith("[bold deep_sky_blue1]verifying[/]")
+
+ def test_a_spent_trial_budget_alone_stays_trialing(self):
+ row = _session_row(
+ {"id": "ab12cd34", "status": "running", "max_trials": 2, "trials": {"count": 2}}
)
- assert row["STATUS"].startswith("[bold sea_green3]clauding[/]")
+ assert row["STATUS"].startswith("[bold sea_green3]trialing[/]")
diff --git a/src/tests/_internal/cli/services/presets/test_prompt.py b/src/tests/_internal/cli/services/presets/test_prompt.py
index 111576340..fc1d8cf2c 100644
--- a/src/tests/_internal/cli/services/presets/test_prompt.py
+++ b/src/tests/_internal/cli/services/presets/test_prompt.py
@@ -13,7 +13,8 @@ def test_stays_byte_identical_without_user_prompt(self):
assert text == get_preset_agent_system_prompt(None) == get_preset_agent_system_prompt("")
assert "## Additional instructions" not in text
- assert "\n"
+ "Plain and stay.\n"
+ )
+ monkeypatch.setattr(prompt_module, "_SYSTEM_PROMPT_PATH", noted)
+
+ text = get_preset_agent_system_prompt()
+
+ assert "TODO" not in text
+ assert text == "Kept.\nPlain and stay."
def test_rejects_unknown_directive_variables(self, tmp_path, monkeypatch):
broken = tmp_path / "system_prompt.md"
diff --git a/src/tests/_internal/cli/services/presets/test_verify.py b/src/tests/_internal/cli/services/presets/test_verify.py
index 793bcda71..00a274c77 100644
--- a/src/tests/_internal/cli/services/presets/test_verify.py
+++ b/src/tests/_internal/cli/services/presets/test_verify.py
@@ -49,7 +49,7 @@ def test_builds_portable_self_contained_preset(self):
preset_configuration=PresetConfiguration(
name="qwen-build",
model={"base": "Qwen/Qwen3.5-27B"},
- context_length=8192,
+ min_context_length=8192,
gateway="benchmark-gateway",
env=["LICENSE", "TOKENIZERS_PARALLELISM=false"],
),