Skip to content

#594 Added command line options to skip checkouts or stop after checkout #595

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Joerg Henrichs (hiker) wants to merge 4 commits into
base: main
Choose a base branch
from
Open

#594 Added command line options to skip checkouts or stop after checkout #595

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
5c52702
#594 Added command line options to skip checkouts or stop after check…
hiker Aug 13, 2026
9175e1a
#594 Added test for new option --checkout-only.
hiker Aug 13, 2026
2d9f410
#594 Added documentation.
hiker Aug 14, 2026
96d27c2
#594 Updated contributors.
hiker Aug 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions CONTRIBUTORS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,11 @@ To indicate your agreement, add your details to the the following table.
If you are not submitting contributions on behalf of an organisation please use
"n/a" for your affiliation.

| GitHub Username | Real Name | Affiliation |
|-----------------|-----------------|-------------|
| MatthewHambley | Matthew Hambley | Met Office |
| yaswant | Yaswant Pradhan | Met Office |
| GitHub Username | Real Name | Affiliation |
|-----------------|-----------------|----------------------------------|
| MatthewHambley | Matthew Hambley | Met Office |
| yaswant | Yaswant Pradhan | Met Office |
| hiker | Joerg Henrichs | Bureau of Meteorology, Australia |

---

Expand Down
11 changes: 10 additions & 1 deletion Documentation/source/fab_base/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,8 @@ description of the all options:
.. parsed-literal::

usage: fab_base.py [-h] [--suite SUITE] [--available-compilers] [--fc FC] [--cc CC] [--ld LD] [--fflags FFLAGS] [--cflags CFLAGS] [--ldflags LDFLAGS] [--nprocs NPROCS]
[--mpi] [--no-mpi] [--openmp] [--no-openmp] [--openacc] [--host HOST] [--site SITE] [--platform PLATFORM]
[--mpi] [--no-mpi] [--openmp] [--no-openmp] [--openacc] [--host HOST] [--checkout-only] [--skip-checkout] [--site SITE] [--platform PLATFORM]
[--fab-workspace FAB_WORKSPACE] [--profile PROFILE]

A Fab-based build system. Note that if --suite is specified, this will change the default for compiler and linker

Expand Down Expand Up @@ -69,9 +70,17 @@ description of the all options:
--openacc, -openacc Enable OpenACC (default: True)
--host HOST, -host HOST
Determine the OpenACC or OpenMP: either 'cpu' or 'gpu'. (default: cpu)
--checkout-only Only do the checkout steps, not any actual build steps.This can be useful if checkout and compilation steps need to run on different nodes. (default:
False)
--skip-checkout Do not do any checkouts. This flag can be used if a checkout was already done, to just do the compilation. This is useful if checkout and compilation
needs to be done on different nodes. (default: False)
--site SITE, -s SITE Name of the site to use. (default: $SITE or 'default')
--platform PLATFORM, -p PLATFORM
Name of the platform of the site to use. (default: $PLATFORM or 'default')
--fab-workspace FAB_WORKSPACE
Fab workspace, in which the build directory will be created. (default: None)
--profile PROFILE, -pro PROFILE
Sets the compiler profile, choose from '['full-debug', 'fast-debug', 'production', 'unit-tests']'. (default: full-debug)


Some command line option have an environment variable as default
Expand Down
41 changes: 41 additions & 0 deletions Documentation/source/fab_base/usage_patterns.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -225,3 +225,44 @@ For example:

linker = tr.get_tool(Category.LINKER, "linker-gfortran")
linker.add_post_lib_flags(["-static-libasan"], "memory-debug")

Running checkout and building independently
-------------------------------------------
On many platforms, only a few dedicated nodes might have internet access,
while the majority of compute nodes cannot access the internet at all.
In order to support these platforms, it is important that the checkout
of an application (e.g. using git) can be done without building, and
similarly that building can be executed without a checkout (meaning the
checkout must have ran before).

The FabBase class provides two command line options to support this:

1. ``--checkout-only``
If this command line option is specified, Fab will exit (successfully)
after ``grab_files_step``. If the user should be running additional
tasks that require internet access, these must therefore be part of
``grab_files_step``. A user code might need to check for this flag
(using ``fab_application.args.checkout_only``).

2. ``--skip-checkout``
This command line parameter is intended to avoid running any checkouts
(git, svn, ...). The Fab base class itself does not trigger any
checkouts, and so this flag is not actually used internally. It is the
responsibility of the application to implement this behaviour.
Example code for this:

.. code-block:: python

for repo_info in repo_infos:
if self.args.skip_checkout:
logger.info(f"Skipping extraction of '{repo}' from "
f"'{repo_info.source}' ")
continue

logger.info(f"Extracting '{repo}' from '{repo_info.source}' "
f" to 'science/{repo}', "
f"revisions {repo_info.ref}")
git_checkout(self.config,
repo_info.source,
dst_label=f'science/{repo}',
revision=repo_info.ref)
16 changes: 16 additions & 0 deletions source/fab/fab_base/fab_base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -472,6 +472,18 @@ class which can provide its own instance (to easily allow for a
'--host', '-host', default="cpu", type=str,
help="Determine the OpenACC or OpenMP: either 'cpu' or 'gpu'.")

parser.add_argument(
'--checkout-only', action="store_true", default=False,
help=("Only do the checkout steps, not any actual build steps."
"This can be useful if checkout and compilation steps "
"need to run on different nodes."))
parser.add_argument(
'--skip-checkout', action="store_true", default=False,
help=("Do not do any checkouts. This flag can be used if a "
"checkout was already done, to just do the compilation. "
"This is useful if checkout and compilation needs to be "
"done on different nodes."))
Comment on lines +475 to +485

@allynt allynt Aug 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These 2 options are mutually exclusive. Is it worth explicitly marking them as such? Something along the lines of:

checkout_group = parser.add_mutually_exclusive_group()
checkout_group.add_argument('--checkout-only', action="store_true", default=False)
checkout_group.add_argument('--skip-checkout', action="store_true", default=False)

Otherwise, you can add an explicit check in handle_command_line_options. Something along the lines of:

if self.args.checkout_only and self.args.skip_checkout:
  raise RuntimeError(f"--checkout-only and --skip-checkout are mutually exclusive arguments")


parser.add_argument("--site", "-s", type=str,
default="$SITE or 'default'",
help="Name of the site to use.")
Expand Down Expand Up @@ -791,6 +803,10 @@ def build(self) -> None:
# need to use it anywhere.
with self._config as _:
self.grab_files_step()
if self.args.checkout_only:
self.logger.info("Aborting after checkout due to "
"'--checkout-only' flag.")
return
self.find_source_files_step()
# This is a Fab function, which the user won't need to be
# able to overwrite.
Expand Down
36 changes: 36 additions & 0 deletions tests/unit_tests/fab_base/test_fab_base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
"""
import argparse
import inspect
import logging
import os
from pathlib import Path
import sys
Expand Down Expand Up @@ -375,6 +376,41 @@ def test_site_specific_inside_dir(monkeypatch) -> None:
assert "site_specific" == sys.path[0]


def test_checkout_only(monkeypatch, caplog) -> None:
'''
Tests that FabBase does not run any build steps if
the --checkout-only flag is provided.
'''

monkeypatch.setattr(sys, "argv", ["fab_base.py", "--checkout-only"])

fab_base = FabBase(name="test")

# We need to patch a lot of Fab functions (to avoid dependencies
# on the runtime environment):
mocks = {}
for function_name in ["grab_folder", "find_source_files",
"preprocess_c", "preprocess_fortran",
"compile_fortran", "compile_c", "analyse"]:
patcher = mock.patch(f"fab.fab_base.fab_base.{function_name}")
mocks[function_name] = (patcher, patcher.start())

with caplog.at_level(logging.INFO):
fab_base.build()
assert ("Aborting after checkout due to '--checkout-only' flag."
in caplog.text)

mocks["grab_folder"][0].stop()
mocks["grab_folder"][1].assert_called_once_with(
fab_base.config, src=".")
# Check that no other function (except grab_folder) is being called.
for function_name, func_patcher in mocks.items():
if function_name == "grab_folder":
continue
func_patcher[0].stop()
func_patcher[1].assert_not_called()


def test_build_binary(monkeypatch) -> None:
'''
Tests an actual trivial build. We patch all fab functions called
Expand Down