Skip to content

prep-caly-model-devi receives no step config and mixes prep/run settings #407

Description

@wanghan-iapcm

The prep-caly-model-devi step in _prep_run_caly receives no step config at all, and mixes the two phases' settings: it builds its PythonOPTemplate from run_template_config but runs on prep_executor.

At e45c147ed261d7c43b4b5bc9ada6e7d271e32c51:

# prep_caly_model_devi
prep_caly_model_devi = Step(
"prep-caly-model-devi",
template=PythonOPTemplate(
prep_caly_model_devi_op,
python_packages=upload_python_packages,
**run_template_config,
),
parameters={
"task_name": "prep-calypso-model-deviation",
"config": prep_run_caly_steps.inputs.parameters["explore_config"],
},
artifacts={
"traj_results": caly_evo_step.outputs.artifacts["traj_results"],
},
key="%s--prep-caly-model-devi"
% (prep_run_caly_steps.inputs.parameters["block_id"],),
executor=prep_executor,
)
prep_run_caly_steps.add(prep_caly_model_devi)

The Step(...) ends at executor=prep_executor, — there is no **prep_config and no **run_config. So parallelism, continue_on_failed, continue_on_num_success and continue_on_success_ratio cannot be set for this step by any means, whether the user configures prep_explore_config or run_explore_config.

Separately, line 254 takes **run_template_config (image, timeout, retry settings, envs) while line 265 takes executor=prep_executor. Every other step in the file keeps those two consistent: prep-caly-input is prep_template_config + prep_executor + **prep_config, and run-caly-model-devi is run_template_config + run_executor (+ **run_config once #380 lands). This step is the only one that crosses them, and nothing in the file explains why.

Origin

Introduced in this exact shape by 07df321 ("Calypso speedup by refactorizing model deviation", #217, 2024-04-30) — the same commit that introduced the **prep_config miswiring on run-caly-model-devi that #380 fixes. A config spread was never present here, so nothing was lost later; it was simply never added.

Suggested fix

Decide which phase this step belongs to and make the template, executor and step config agree. If it is a prep step, use prep_template_config + prep_executor + **prep_config. If it is a run step, use run_template_config + run_executor + a run config.

Either way, note that this step is not sliced, so spreading a config that still contains continue_on_num_success / continue_on_success_ratio will raise UnboundLocalError inside dflow — see the companion issue on the prep_config pop/spread mismatch. The no_slice_run_config idiom at caly_evo_step.py L150-L152 is the existing pattern for this.

Found while reviewing #380.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions