tomobank tests, documentation updates

.github/workflows/httomo_docs.yml

@@ -39,6 +39,8 @@ jobs:
         - name: Generate full yaml pipelines using pipeline directives
           run: |
             python ./docs/source/scripts/execute_pipelines_build.py -o ./docs/source/pipelines_full/
+            python ./docs/source/scripts/modify_pipeline.py -i ./docs/source/pipelines_full/tomopy_tomobank.yaml
+            python ./docs/source/scripts/modify_pipeline.py -i ./docs/source/pipelines_full/FBP3d_tomobar_tomobank.yaml
 
         - name: Create ZIP archive for pipelines
           run: |

docs/source/howto/how_to_run/real_data_example.rst

@@ -0,0 +1,80 @@
+.. _real-data-example:
+
+Real data processing
+====================
+
+This section presents an example of processing real experimental data using HTTomo from the `TomoBank`_ data archive.
+
+.. list-table::
+
+
+    * - .. figure:: ../../_static/real_data/sino_tomo088.jpg
+
+           Dark/Flat field corrected sinogram of the `Lorentz data set`_.
+
+      - .. figure:: ../../_static/real_data/recon_tomo088.jpg
+
+           Reconstructed slice using FBP method
+
+
+Before starting, we assume that HTTomo has been successfully installed. If you have not installed HTTomo yet, please follow the
+:ref:`installation_main`.
+
+We also recommend running the :ref:`run_tests` to verify that all required dependencies are installed and that the framework is 
+functioning correctly.
+
+For this example, we will use raw data from the `TomoBank`_ data archive. Please download the `Lorentz data set`_. The dataset is 
+hosted using the Globus file management system, which requires authentication. You can sign in using your GitHub credentials.
+
+.. _TomoBank: https://tomobank.readthedocs.io/en/latest/
+
+.. _Lorentz data set: https://tomobank.readthedocs.io/en/latest/source/data/docs.data.lorentz.html
+
+Once the dataset has been downloaded, you should have the file :code:`tomo_00088.h5` on your disk. You can then proceed with running a simple HTTomo pipeline.
+
+TomoPy (CPU) pipeline
++++++++++++++++++++++
+
+This pipeline uses the CPU implementation of the TomoPy library. TomoPy must be installed before running the pipeline. 
+See :ref:`backends_list`.
+
+Running this pipeline requires TomoPy package to be installed, see :ref:`backends_list`. Copy the following pipeline into a YAML file,
+for example: :code:`tomopy_tomo_00088.yaml`. 
+
+.. dropdown:: Standard 180 degrees pipeline using TomoPy (CPU) for tomo_00088.h5 dataset
+
+    .. literalinclude:: ../../pipelines_full/tomopy_tomobank.yaml
+        :language: yaml
+
+
+Then run HTTomo according to :ref:`howto_run_outside_diamond` documentation. In particular, provide the path to the input dataset, the pipeline YAML file, and the output directory.
+
+.. code-block:: console
+
+    $ python -m httomo run path/to/tomo_00088.h5 tomopy_tomo_00088.yaml /path/to/output_folder
+
+GPU pipeline
+++++++++++++
+
+If a CUDA-enabled GPU is available, the same dataset can be processed using GPU-accelerated libraries. 
+This can significantly reduce the processing time for suitable pipelines. Run the pipeline bellow in a similar way as explained above. 
+
+.. dropdown:: GPU-enabled processing for tomo_00088.h5 dataset
+
+    .. literalinclude:: ../../pipelines_full/FBP3d_tomobar_tomobank.yaml
+        :language: yaml
+
+Output results
+++++++++++++++
+
+In the output folder you will find:
+
+1. Copied YAML file with the executed pipeline.
+2. Debug log and the user log, see more :ref:`info_logger`.
+3. The result of the reconstruction saved as HDF5 file. This file can be open, for instance with `Dawn`_ software.
+4. Saved tiff files of the reconstructed image. You can use `ImageJ`_ or `ImageJ.JS`_ to visualise.
+
+
+.. _Dawn: https://dawnsci.org/
+.. _ImageJ: https://imagej.net/ij/
+.. _ImageJ.JS: https://ij.imjoy.io/

docs/source/howto/how_to_run/run_in_depth.rst

@@ -13,7 +13,7 @@ As mentioned earlier, the preliminary step to accessing installed HTTomo softwar
 depends on if you are using a Diamond machine or not:
 
 - not on a Diamond machine: activate the conda environment that HTTomo was
-  installed into (please refer to :doc:`installation` for instructions on how to
+  installed into (please refer to :ref:`installation_main` for instructions on how to
   install HTTomo)
 
 - on a Diamond machine: run the command :code:`module load httomo`

docs/source/howto/run_httomo.rst

@@ -39,4 +39,5 @@ split into these two subsections where relevant.
    how_to_run/at_diamond
    how_to_run/outside_diamond
    how_to_run/run_in_depth
+   how_to_run/real_data_example
 

docs/source/scripts/modify_pipeline.py

@@ -0,0 +1,123 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+# ---------------------------------------------------------------------------
+# Copyright 2022 Diamond Light Source Ltd.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ---------------------------------------------------------------------------
+# Created By  : Tomography Team <scientificsoftware@diamond.ac.uk>
+# Created Date: 11/August/2026
+# version ='0.1'
+# ---------------------------------------------------------------------------
+"""This script modifies a given pipeline by changing parameters in it."""
+
+import argparse
+import yaml
+from typing import Union
+
+
+def get_args():
+    parser = argparse.ArgumentParser(
+        description="Script that modifies parameters in a YAML pipeline for HTTomo "
+    )
+    parser.add_argument(
+        "-i",
+        "--input",
+        type=str,
+        default="./",
+        help="Full path to a specific pipeline that needs modification.",
+    )
+    return parser.parse_args()
+
+
+def change_value_parameters_method_pipeline(
+    yaml_path: str,
+    method: list,
+    key: list,
+    value: list,
+    save_result: Union[None, bool] = None,
+):
+    # changes methods parameters in the given pipeline and re-save the pipeline
+    with open(yaml_path, "r") as f:
+        conf = list(yaml.load_all(f, Loader=yaml.FullLoader))
+    opened_yaml = conf[0]
+    methods_no = len(opened_yaml)
+    methods_no_correct = len(method)
+    for i in range(methods_no):
+        method_content = opened_yaml[i]
+        method_name = method_content["method"]
+        for j in range(methods_no_correct):
+            if method[j] == method_name:
+                # change something in parameters here
+                opened_yaml[i]["parameters"][key[j]] = value[j]
+                if save_result is not None:
+                    # add save_result to the list of keys
+                    opened_yaml[i]["save_result"] = save_result
+
+    with open(yaml_path, "w") as file_descriptor:
+        yaml.dump(
+            opened_yaml, file_descriptor, default_flow_style=False, sort_keys=False
+        )
+
+
+if __name__ == "__main__":
+    args = get_args()
+    path_to_pipeline = args.input
+
+    change_value_parameters_method_pipeline(
+        path_to_pipeline,
+        method=[
+            "standard_tomo",
+            "standard_tomo",
+            "standard_tomo",
+            "standard_tomo",
+            "standard_tomo",
+            "standard_tomo",
+            "find_center_vo",
+        ],
+        key=[
+            "preview",
+            "data_path",
+            "image_key_path",
+            "rotation_angles",
+            "darks",
+            "flats",
+            "ind",
+        ],
+        value=[
+            {"detector_y": {"start": 500, "stop": 510}},
+            "/exchange/data",
+            None,
+            {
+                "user_defined": {
+                    "start_angle": 0,
+                    "stop_angle": 179.876,
+                    "angles_total": 1500,
+                }
+            },
+            {
+                "file": "input_data",
+                "image_key_path": None,
+                "data_path": "/exchange/data_dark",
+            },
+            {
+                "file": "input_data",
+                "image_key_path": None,
+                "data_path": "/exchange/data_white",
+            },
+            "mid",
+        ],
+    )
+
+    message_str = f"Pipeline {path_to_pipeline} has been modified."
+    print(message_str)

tests/conftest.py

@@ -6,6 +6,7 @@
 from typing import Any, Callable, Dict, List, TypeAlias, Union, Tuple
 import numpy as np
 from PIL import Image
+import h5py
 
 import pytest
 import yaml
@@ -234,6 +235,16 @@ def angles_averaging():
     return "docs/source/pipelines_full/angles_averaging.yaml"
 
 
+@pytest.fixture
+def tomopy_tomobank():
+    return "docs/source/pipelines_full/tomopy_tomobank.yaml"
+
+
+@pytest.fixture
+def FBP3d_tomobar_tomobank():
+    return "docs/source/pipelines_full/FBP3d_tomobar_tomobank.yaml"
+
+
 @pytest.fixture
 def FISTA3d_tomobar():
     return "docs/source/pipelines_full/FISTA3d_tomobar.yaml"
@@ -317,6 +328,12 @@ def i12_119647():
     return "tests/test_data/raw_data/i12/119647.nxs"
 
 
+@pytest.fixture
+def tomobank_00088():
+    # TomoBank data
+    return "tests/test_data/raw_data/tomobank/tomo_00088.h5"
+
+
 ############## --Ground Truth references-- #################
 
 
@@ -386,6 +403,18 @@ def FISTA3d_tomobar_k11_38731_npz():
     return np.load("tests/test_data/raw_data/diad/FISTA3d_tomobar_k11_38731.npz")
 
 
+@pytest.fixture
+def tomobank00088_tomopy_npz():
+    # 10 slices numpy array
+    return np.load("tests/test_data/raw_data/tomobank/tomobank00088_tomopy.npz")
+
+
+@pytest.fixture
+def tomobank00088_FBP3d_tomobar():
+    # 10 slices numpy array
+    return np.load("tests/test_data/raw_data/tomobank/tomobank00088_FBP3d_tomobar.npz")
+
+
 @pytest.fixture
 def pipeline_sweep_FBP3d_tomobar_i13_177906_tiffs():
     # 8 tiff files of 32bit
@@ -539,6 +568,44 @@ def compare_tif(files_list_to_compare: list, file_path_to_references: list):
         assert res_norm < 1e-3
 
 
+def calculate_gt_residual(
+    path_to_data: str,
+    h5_file_name: str,
+    h5_files: list,
+    data_gt: np.ndarray,
+    axis_slice: int,
+) -> float:
+
+    slices, sizeX, sizeY = np.shape(data_gt)
+    step = axis_slice // (slices + 2)
+    data_result = np.zeros(
+        (slices, sizeX, sizeY),
+        dtype=np.float32,
+    )
+    for file_to_open in h5_files:
+        if h5_file_name in file_to_open:
+            with h5py.File(file_to_open, "r") as h5f:
+                dataset = h5f[path_to_data]
+
+                if not isinstance(dataset, h5py.Dataset):
+                    raise TypeError(f"{path_to_data!r} is not an HDF5 dataset.")
+
+                index_prog = step
+
+                for i in range(slices):
+                    data_result[i, :, :] = dataset[:, index_prog, :]
+                    index_prog += step
+
+            break
+    else:
+        raise FileNotFoundError(
+            f"File name containing '{h5_file_name}' cannot be found."
+        )
+    residual_im = data_gt - data_result
+    res_norm = np.linalg.norm(residual_im.flatten()).astype("float32")
+    return res_norm
+
+
 def change_value_parameters_method_pipeline(
     yaml_path: str,
     method: list,