Fix geometry once per workflow from migrating branch

[baptistefraisse]

Closes Issue #38

Move geometry modification and compilation from per-job execution to once per workflow. Previously, geometry modification and compilation were performed for every job. This change ensures that geometry preparation is performed once per workflow (i.e. per design point), which is more efficient.

Changes

• Introduce a WorkflowSharedContext to share data across all jobs of a workflow
• Add EpicStack.prepare_workflow_geometry() to:
  copy the geometry from EPIC_INSTALL into a workflow-local directory
  apply XML modifications
  build the geometry once
• Update DAGExecutor.execute() to:
  prepare geometry once at workflow start (per design point)
  store the prepared geometry directory in the shared workflow context
• Simplify EpicStack.prepare_for_execution() to reuse the pre-built geometry instead of rebuilding it per job
• Update make_driver_script() to use the workflow-level geometry directory
• Fix a circular import in DAGExecutor by moving the EpicStack import inside execute()

Testing

The following tests were run successfully:

pytest tests/test_utilities/test_epic_utils/test_epic_stack.py -q
pytest tests/test_workflows/test_dag_executor.py -q
pytest tests/test_workflows -q

Running the full test suite on migrating-configuration-models currently fails due to unrelated issues (e.g. missing ClassVar import in epic_env_config.py and AxParameterConstraint in optimizers.ax). To validate the changes, tests were executed on top of the fixes introduced in PR #39, which resolve the ClassVar issue. The geometry workflow changes were verified in that context.

Additional

XML modifications are now applied to the workflow-local copy of the geometry rather than the original EPIC_INSTALL directory
No changes were made to EpicDesignConfig or configuration APIs

[karthik18495]

Looks good.

[ruse-traveler]

Thanks for the development, Baptiste! Overall, I think it looks great! I do have a couple suggestions. The details are spelled out in the comments, but the thrust is that:

1. I think we can deal the circular import issue without having to import EpicStack in DAGExectuor.execute, which helps keep the stack interface generic; and
2. We'll want to make sure compilation happens inside the eic-shell!

After these are resolved, I think this is ready to ship!

[ruse-traveler]

I actually think it could be useful to make this a function of ExperimentStack! Naturally, the details of how are going to vary, but every experiment will need to modify their geometry! The default behavior could be to do nothing, and then each experiment would be responsible for overriding with their desired behavior.

# in workflows/experimental_stack.py
ExperimentStack(ABC):
    <... other stuff ...>

    def prepare_workflow_geometry(
        self,
        workflow_dir: str,
        design_point: Dict[str, Any],
        problem_config: Any,
        workflow_id: str,
    ) -> str:
    """
    Prepare geometry for workflow/design point.
    Returns the prepare geometry directory.
    """
    pass

    <... other stuff ...>

This would also resolve the circular import issue without having to pull ePIC-specific code into the DAGExecutor! I'll sketch out some more details below.

[ruse-traveler]

I actually think the documentation here is pretty useful, especially the example, so I would suggest keeping it! We can always do a dedicated pass to clean/tighten up the documentation down the road.

[ruse-traveler]

Like I mentioned above, if we make prepare_workflow_geometry a member of ExperimentalStack, we can avoid having to import EpicStack! The way to go would be use the StackRegistry to retrieve the relevant class, e.g.

Suggested change

	from aid2e.utilities.epic_utils.epic_stack import EpicStack
	stack = EpicStack()
	stack = StackRegistry.get_experimental_stack(self.workflow.stack_type)

The only thing we would need to do is make sure that the DAGExecutor knows which stack to use. For this PR, maybe we could add an optional field to WorkflowDefinition to note which type? Later we might handle this by introducing a specialized EpicWorkflow or similar...

[ruse-traveler]

Hmmm... Should this have been removed?

[karthik18495]

Hi @baptistefraisse Do you want look more into this?