diff --git a/lib/ramble/docs/conf.py b/lib/ramble/docs/conf.py index d0eff610c9..1a7e1a2d36 100644 --- a/lib/ramble/docs/conf.py +++ b/lib/ramble/docs/conf.py @@ -145,11 +145,13 @@ def setup(sphinx): # Set default graphviz options graphviz_dot_args = [ - "-Grankdir=LR", "-Gbgcolor=transparent", + "-Gfontname=Helvetica,Arial,sans-serif", "-Nshape=box", - "-Nfontname=monaco", - "-Nfontsize=10", + "-Nfontname=Helvetica,Arial,sans-serif", + "-Nfontsize=11", + "-Efontname=Helvetica,Arial,sans-serif", + "-Efontsize=10", ] # Get nice vector graphics diff --git a/lib/ramble/docs/dev_guides/advanced_topics.rst b/lib/ramble/docs/dev_guides/advanced_topics.rst index 01b122bb38..a238c65190 100644 --- a/lib/ramble/docs/dev_guides/advanced_topics.rst +++ b/lib/ramble/docs/dev_guides/advanced_topics.rst @@ -28,13 +28,34 @@ types supported in Ramble. Experiment Pipelines and Phases ------------------------------- -Ramble has a concept of ``pipeline``, which represent full actions that can -be taken on a workspace. Some of the common ``pipelines`` that are used are the -``setup`` (driven with ``ramble workspace setup``) and ``analyze`` (driven with -``ramble workspace analyze``) pipelines. +Architecture Overview +===================== -There are several more pipelines that Ramble uses to perform complex actions -on a workspace, which can be seen in :mod:`ramble.pipeline`. +Ramble organizes workspaces into a modular **Pipeline** and **Phase Graph** architecture. + +Pipelines +--------- + +Ramble has a concept of a ``pipeline`` which represents full actions that can +be taken on a workspace. Each ``pipeline`` has a corresponding workspace +command and targets a specific stage of the experiment lifecycle. The most +:ref:`common pipelines ` are: + + * :ref:`ramble workspace setup ` + → :py:class:`~ramble.pipeline.SetupPipeline` + * :ref:`ramble on ` + → :py:class:`~ramble.pipeline.ExecutePipeline` + * :ref:`ramble workspace analyze ` + → :py:class:`~ramble.pipeline.AnalyzePipeline` + * :ref:`ramble workspace archive ` + → :py:class:`~ramble.pipeline.ArchivePipeline` + +There are more pipelines that Ramble can use to perform complex actions +on a workspace, which can be seen in the :ref:`Additional Pipelines +` section below. + +Phase Graphs +------------ Pipelines are built out of phases. In Ramble, a ``phase`` represents a specific step along the path of completing the action defined by the ``pipeline``. @@ -42,9 +63,35 @@ Examples of phases include ``get_inputs`` (for downloading input files needed by a workload) and ``software_install`` (for performing software installation using a package manager). -Phases can be defined in a variety of locations. Some base classes define phases for specific pipelines. Additionally, -instances of application, modifier, or package manager definitions can define -phases as well. Each phase is defined in two parts. The first part is to define +For each pipeline, Ramble builds a directed acyclic graph +(:py:class:`ramble.graphs.PhaseGraph`) containing all phases registered for that pipeline. + + * **Constraint Declaration**: Phase dependency constraints are declared within object class + definitions using ``run_before=['phase_name']`` or ``run_after=['phase_name']``. + * **Order Resolution**: When a pipeline initializes, :py:class:`ramble.graphs.PhaseGraph` + resolves phase dependencies using a topological sort. If cyclical dependencies exist, + Ramble raises a :py:class:`~ramble.error.RambleError`. + +Phase Registration +================== + +Phases can be defined in a variety of locations. Some base classes define phases for specific +pipelines. Additionally, Applications, Modifiers, and Package Managers can all define and +register their own phases to build more complex pipelines for specific use cases. + + * :doc:`Applications <../application_list>` define core command generation, input downloading, + license inclusion, and analysis hooks. + * :doc:`Package Managers <../package_managers>` (e.g., Spack, Pip) inject software environment + creation and installation phases. + * :doc:`Modifiers <../modifier_list>` (e.g., containers, profilers) inject pre-exec and post-exec + hooks. + * :doc:`Workflow Managers <../workflow_managers>` (e.g., Slurm, PBS, Google Batch) define batch + submission logic and job execution wrappers. + * :doc:`Systems <../systems>` and :doc:`Platforms <../platforms>` contribute hardware-specific + variables, CPU architecture parameters, and compiler configurations expanded during pipeline + setup. + +Each phase is defined in two parts. The first part is to define a class method on a object definition. Phase names need to begin with an underscore, and they have the following signature: @@ -70,8 +117,400 @@ phase. Phase registration can also define a ``run_after`` list of phases to exec before the newly registered phase. A phase can also be registered into multiple pipelines, by calling the ``register_phase`` directive multiple times. -Applications, Modifiers, and Package Managers can all defined and register -their own phases to build more complex pipelines for specific usecases. +The diagram below illustrates a :py:class:`~ramble.pipeline.SetupPipeline` example of how +classes register specific phases via ``register_phase()`` into the pipeline's +:py:class:`~ramble.graphs.PhaseGraph` as phases: + +.. graphviz:: + + digraph setup_phase_registration { + newrank=true; + fontsize=12; + rankdir=TB; + nodesep=0.8; + ranksep=0.4; + compound=true; + + node [shape=box, style="filled,rounded", fontsize=11, margin="0.2,0.12"]; + edge [fontsize=10, color="#0056b3", penwidth=1.5]; + + // Classes + subgraph cluster_classes { + label=<Classes>; + labelloc=t; + labeljust=c; + style="filled,dashed"; + fillcolor="#f8f9fa"; + color="#cbd5e1"; + fontsize=12; + margin=20; + + mod [ + label="Modifier\n(e.g., Profilers)", + fillcolor="#eef6ff", + color="#0056b3", + width=2.5 + ]; + pm [ + label="Package Manager\n(e.g., Spack)", + fillcolor="#eef6ff", + color="#0056b3", + width=2.5 + ]; + app [ + label="Application Definition\n(e.g., Gromacs)", + fillcolor="#eef6ff", + color="#0056b3", + width=2.5 + ]; + + mod -> pm -> app [style=invis]; + } + + // PhaseGraph Execution Order + subgraph cluster_phases { + label=<SetupPipeline PhaseGraph>; + labelloc=t; + labeljust=c; + style="filled,dashed"; + fillcolor="#f8f9fa"; + color="#cbd5e1"; + fontsize=12; + margin=20; + + p1 [ + label="Phase 1: bootstrap_utilities", + fillcolor="#e6f4ea", + color="#28a745", + width=2.5 + ]; + p2 [ + label="Phase 2: software_create_env", + fillcolor="#e6f4ea", + color="#28a745", + width=2.5 + ]; + p3 [ + label="Phase 3: get_inputs", + fillcolor="#e6f4ea", + color="#28a745", + width=2.5 + ]; + p4 [ + label="Phase 4: license_includes", + fillcolor="#e6f4ea", + color="#28a745", + width=2.5 + ]; + p5 [ + label="Phase 5: make_experiments", + fillcolor="#e6f4ea", + color="#28a745", + width=2.5 + ]; + + p1 -> p2 -> p3 -> p4 -> p5 [color="#28a745", penwidth=2.0, weight=10]; + } + + // Horizontal rank alignment between Classes and Phases + { rank=same; mod; p1; } + { rank=same; pm; p2; } + { rank=same; app; p3; } + + // Classes --> Phases + mod -> p1 [ + label=<register_phase('bootstrap_utilities')>, + color="#0056b3" + ]; + pm -> p2 [ + label=<register_phase('software_create_env')>, + color="#0056b3" + ]; + app -> p3 [ + label=<register_phase('get_inputs')>, + color="#0056b3" + ]; + app:se -> p4:w [ + label=<register_phase('license_includes')>, + color="#0056b3" + ]; + app:s -> p5:w [ + label=<register_phase('make_experiments')>, + color="#0056b3" + ]; + } + +.. _standard-pipelines: + +Standard Pipelines +================== + +Setup Pipeline +-------------- + +* **CLI Command**: :ref:`ramble workspace setup ` +* **Class**: :py:class:`ramble.pipeline.SetupPipeline` + +The setup pipeline prepares everything required to execute experiments: + +.. graphviz:: + + digraph setup_pipeline { + fontsize=12; + rankdir=TB; + nodesep=0.3; + ranksep=0.4; + + node [ + shape=box, + style="filled,rounded", + fontsize=11, + margin="0.25,0.12", + fillcolor="#d4edda", + color="#28a745", + width=3.5 + ]; + edge [fontsize=10, color="#28a745", penwidth=1.3]; + + s1 [label="bootstrap_utilities\n(Fetch tool dependencies)"]; + s2 [label="software_create_env / software_install\n(Concretize & install via Package Manager)"]; + s3 [label="get_inputs\n(Download workload datasets)"]; + s4 [label="license_includes\n(Inject license variables)"]; + s5 [label="make_experiments\n(Render execute_experiment scripts)"]; + + s1 -> s2 -> s3 -> s4 -> s5; + } + +Execute Pipeline +---------------- + +* **CLI Command**: :ref:`ramble on ` +* **Class**: :py:class:`ramble.pipeline.ExecutePipeline` + +Unlike setup or analysis, base application definitions do not register built-in default execution +phases because experiment execution centers on evaluating the ``{batch_submit}`` executor command. +However, custom execution phases registered via ``register_phase(..., pipeline="execute")`` +(e.g., by modifiers or specialized applications) are processed first by the +:py:class:`~ramble.graphs.PhaseGraph`: + +.. graphviz:: + + digraph execute_pipeline { + fontsize=12; + rankdir=TB; + nodesep=0.3; + ranksep=0.4; + + node [ + shape=box, + style="filled,rounded", + fontsize=11, + margin="0.25,0.12", + fillcolor="#cce5ff", + color="#004085", + width=3.5 + ]; + edge [fontsize=10, color="#004085", penwidth=1.3]; + + e1 [label="Custom Registered Phases\n(Optional register_phase(..., pipeline='execute'))"]; + e2 [label="Expand Executor Command\n(Evaluate {batch_submit})"]; + e3 [label="Submit / Launch Experiments\n(Invoke sbatch, mpirun, or local shell)"]; + + e1 -> e2 -> e3; + } + +* **Batch Scheduler Integration**: :doc:`Workflow Managers <../workflow_managers>` (e.g., Slurm, PBS, + Google Batch) expand the ``{batch_submit}`` executor command to submit batch scripts to the + workload manager or execute scripts locally. + +Analyze Pipeline +---------------- + +* **CLI Command**: :ref:`ramble workspace analyze ` +* **Class**: :py:class:`ramble.pipeline.AnalyzePipeline` + +The analyze pipeline evaluates completed experiments, extracts metrics, and writes results using +registered phases: + +.. graphviz:: + + digraph analyze_pipeline { + fontsize=12; + rankdir=TB; + nodesep=0.3; + ranksep=0.4; + + node [ + shape=box, + style="filled,rounded", + fontsize=11, + margin="0.25,0.12", + fillcolor="#ffe8cc", + color="#d9480f", + width=3.5 + ]; + edge [fontsize=10, color="#d9480f", penwidth=1.3]; + + a1 [label="prepare_analysis\n(Pre-processing hook for output logs)"]; + a2 [label="analyze_experiments\n(Extract FOMs & evaluate success_criteria)"]; + a3 [label="write_status & append_results_to_workspace\n(Persist ramble_status.json & workspace results)"]; + a4 [label="write_results_cache\n(Dump results text/YAML/JSON & upload if requested)"]; + + a1 -> a2 -> a3 -> a4; + } + +Archive Pipeline +---------------- + +* **CLI Command**: :ref:`ramble workspace archive ` +* **Class**: :py:class:`ramble.pipeline.ArchivePipeline` + +The archive pipeline preserves experiment artifacts using its registered phase followed by +optional archive creation: + +.. graphviz:: + + digraph archive_pipeline { + fontsize=12; + rankdir=TB; + nodesep=0.3; + ranksep=0.4; + + node [ + shape=box, + style="filled,rounded", + fontsize=11, + margin="0.25,0.12", + fillcolor="#eebefa", + color="#862e9c", + width=3.5 + ]; + edge [fontsize=10, color="#862e9c", penwidth=1.3]; + + ar1 [label="archive_experiments\n(Registered Phase: collect logs, templates & FOM files)"]; + ar2 [label="create_tarball / upload_archive\n(Generate archive.latest.tar.gz & upload)"]; + + ar1 -> ar2; + } + +Standard Pipeline Phases +------------------------- + +The following table summarizes the standard built-in phases registered across Ramble's primary +pipelines: + +.. list-table:: + :widths: 22 15 20 43 + :header-rows: 1 + + * - Phase Name + - Pipeline + - Owner / Source + - Description & Purpose + * - ``bootstrap_utilities`` + - Setup + - Application Base + - Downloads or builds external tool dependencies required for script rendering and experiment + processing. + * - ``software_create_env`` + - Setup + - Package Manager + - Creates software environment definitions (e.g., Spack environments) and concretizes + package specs. + * - ``software_install`` + - Setup + - Package Manager + - Installs required software packages, compilers, and dependencies into the target software + stack. + * - ``get_inputs`` + - Setup + - Application Base + - Downloads, verifies checksums, and extracts workload input datasets into workspace + directories. + * - ``license_includes`` + - Setup + - Application Base + - Resolves license environment variables and paths for commercial or proprietary software. + * - ``make_experiments`` + - Setup + - Application Base + - Expands variables, renders templates, creates ``execute_experiment`` scripts, and writes the + ``all_experiments`` submission script. + * - ``prepare_analysis`` + - Analyze + - Application Base + - Application-specific pre-processing hook executed prior to FOM extraction. + * - ``analyze_experiments`` + - Analyze + - Application Base + - Parses log files using regular expressions defined by Figures of Merit (FOMs) and evaluates + ``success_criteria``. + * - ``calculate_statistics`` + - Analyze + - Pipeline Engine + - Computes summary statistics (mean, stddev, min, max) for repeated experiment runs. + * - ``archive_experiments`` + - Archive + - Application Base + - Copies logs, rendered templates, FOM output files, and inventory metadata into the workspace + archive directory. + +.. _additional-pipelines: + +Additional Pipelines +==================== + +Ramble provides several additional pipelines for specialized workflow operations: + +* **Mirror Pipeline** (:py:class:`~ramble.pipeline.MirrorPipeline`): + + * **CLI Command**: :ref:`ramble workspace mirror ` + * Downloads software tarballs and workload input files into a local workspace mirror for offline + execution (see :doc:`../mirror_config`). + +* **PushDeployment Pipeline** (:py:class:`~ramble.pipeline.PushDeploymentPipeline`): + + * **CLI Command**: :ref:`ramble deployment push ` + * Packages workspace configurations, software definitions, and templates into a deployment + bundle suitable for distribution (see :doc:`../workspace`). + +* **PushToCache Pipeline** (:py:class:`~ramble.pipeline.PushToCachePipeline`): + + * **CLI Command**: :ref:`ramble workspace push-to-cache ` + * Pushes compiled software environments to a Spack build cache (see :doc:`../package_managers`). + +* **Bootstrap Pipeline** (:py:class:`~ramble.pipeline.BootstrapPipeline`): + + * **CLI Command**: :ref:`ramble workspace bootstrap ` + * Bootstraps external utilities required by the workspace (see :doc:`../utilities`). + +* **Logs Pipeline** (:py:class:`~ramble.pipeline.LogsPipeline`): + + * **CLI Command**: :ref:`ramble workspace experiment-logs ` + * Inspects log files and archive patterns across workspace experiments (see :doc:`../workspace`). + +Pipeline Reference Links +======================== + +* **Pipeline Engine**: + + * Base Class: :py:class:`ramble.pipeline.Pipeline` + * Setup: :py:class:`ramble.pipeline.SetupPipeline` + * Execute: :py:class:`ramble.pipeline.ExecutePipeline` + * Analyze: :py:class:`ramble.pipeline.AnalyzePipeline` + * Archive: :py:class:`ramble.pipeline.ArchivePipeline` + * Mirror: :py:class:`ramble.pipeline.MirrorPipeline` + * PushDeployment: :py:class:`ramble.pipeline.PushDeploymentPipeline` + * PushToCache: :py:class:`ramble.pipeline.PushToCachePipeline` + +* **Phase Graph Engine**: + + * Graph: :py:class:`ramble.graphs.PhaseGraph` + * Node: :py:class:`ramble.util.graph.GraphNode` + +* **Phase Directives**: + + * Directive: :py:func:`ramble.language.shared_language.register_phase` .. _ramble-builtins: diff --git a/lib/ramble/docs/index.rst b/lib/ramble/docs/index.rst index c7b3758274..31f6b1a1f6 100644 --- a/lib/ramble/docs/index.rst +++ b/lib/ramble/docs/index.rst @@ -40,6 +40,7 @@ If you're new to Ramble and want to start using it, see :doc:`getting_started`. success_criteria results mirror_config + pipelines .. toctree:: :maxdepth: 2 diff --git a/lib/ramble/docs/pipelines.rst b/lib/ramble/docs/pipelines.rst new file mode 100644 index 0000000000..26d71d719e --- /dev/null +++ b/lib/ramble/docs/pipelines.rst @@ -0,0 +1,104 @@ +.. Copyright 2022-2026 The Ramble Authors + + Licensed under the Apache License, Version 2.0 or the MIT license + , at your + option. This file may not be copied, modified, or distributed + except according to those terms. + +.. _ramble-pipelines: + +========================= +Pipelines and Phases +========================= + +Unlike frameworks with a single fixed execution pipeline, Ramble defines high-level workflows +(**Pipelines**) that execute an ordered graph of steps (**Phases**). + +------------------------- +Common Workspace Workflow +------------------------- + +An example of a Ramble workspace workflow might look like the following: + +.. graphviz:: + + digraph workspace_workflow { + fontsize=12; + rankdir=TB; + nodesep=0.4; + ranksep=0.4; + + node [ + shape=box, + style="filled,rounded", + fontsize=11, + margin="0.3,0.12", + width=3.8 + ]; + edge [fontsize=10, color="#333333", penwidth=1.5]; + + step1 [ + label=<1. Workspace Creation
(ramble workspace create)>, + fillcolor="#f8f9fa", + color="#495057" + ]; + step2 [ + label=<2. YAML & Template Configuration
(ramble workspace edit)>, + fillcolor="#e9ecef", + color="#495057" + ]; + step3 [ + label=<3. Workspace Concretization
(ramble workspace concretize)>, + fillcolor="#e2e3e5", + color="#495057" + ]; + step4 [ + label=<4. Setup Pipeline
(ramble workspace setup)>, + fillcolor="#d4edda", + color="#28a745" + ]; + step5 [ + label=<5. Execute Pipeline
(ramble on)>, + fillcolor="#cce5ff", + color="#004085" + ]; + step6 [ + label=<6. Analyze Pipeline
(ramble workspace analyze)>, + fillcolor="#ffe8cc", + color="#d9480f" + ]; + step7 [ + label=<7. Workspace Reporting
(ramble results)>, + fillcolor="#d1ecf1", + color="#0c5460" + ]; + step8 [ + label=<8. Archive Pipeline
(ramble workspace archive)>, + fillcolor="#eebefa", + color="#862e9c" + ]; + + step1 -> step2 -> step3 -> step4 -> step5 -> step6 -> step7 -> step8; + } + +1. **Creation**: Initialize a workspace using :ref:`ramble workspace create + `. +2. **YAML and Template Configuration**: Edit the primary configuration file + ``$workspace/configs/ramble.yaml`` or custom execution templates like ``execute_experiment.tpl`` + using :ref:`ramble workspace edit ` (see :doc:`workspace_config`). +3. **Concretization**: Resolve software specs and experiment matrix combinations using :ref:`ramble + workspace concretize `. +4. **Setup Pipeline**: Run :ref:`ramble workspace setup ` to build + software environments, download datasets, and render execution scripts. +5. **Execution Pipeline**: Launch experiments via :ref:`ramble on `. :doc:`Workflow + Managers ` (e.g., Slurm, PBS, Google Batch) expand ``{batch_submit}`` to + submit jobs to batch schedulers or execute scripts directly on compute nodes. +6. **Analysis Pipeline**: Extract Figures of Merit (FOMs) and evaluate success criteria using + :ref:`ramble workspace analyze `. +7. **Reporting**: View summary statistics and result tables using :ref:`ramble results + ` (see :doc:`results`). +8. **Archiving Pipeline**: Package logs, rendered templates, and inventory files into an archive + via :ref:`ramble workspace archive `. + +For a more detailed explanation of pipelines and phases, refer to the :ref:`advanced pipelines and phases ` documentation. \ No newline at end of file