dalesteam · fjansson · Aug 6, 2026
diff --git a/book/_toc.yml b/book/_toc.yml
@@ -22,6 +22,10 @@ parts:
       # - file: running/ibm.ipynb
       # - file: running/openbcs.ipynb
       - file: running/gpu.md
+      - file: running/quirks.md
+  - caption: Hints for new DALES developers
+    chapters:
+      - file: developers/developers.md
   - caption:  Recent publications using DALES
     chapters:
       - file: running/papers.ipynb

diff --git a/book/developers/developers.md b/book/developers/developers.md
@@ -0,0 +1,59 @@
+# For new developers
+
+* git and GitHub are used for DALES development. See this [Software Carpentry course](http://swcarpentry.github.io/git-novice/) for git basics. Especially, learn about
+  - git clone (download DALES from github to your computer)
+  - branches (different "versions" of the DALES code, for example for developing a specific function)
+  - forking (make a copy of the DALES repository to your own GitHub account)
+
+* Avoid distributing DALES source code outside git, e.g. by email. It's hard to get the
+  code back into git. If you want to send a version to someone else, make a new
+  branch (in your own fork) for it, and send the link.
+
+* When changing the official version: we don't want surprise changes
+  to the default behavior (exceptions: fixing bugs). New features
+  should in general be introduced with namelist flags to turn them on,
+  and the default should be the old behavior.
+
+* Prefer netCDF output over ASCII (netCDF is standardized, more robust, better
+  tools available, easier for others to process).
+
+* DALES should work with multiple Fortran compilers. gfortran and intel are most
+  used at the moment. Automatic tests with GitHub actions, which verify that the code
+  compiles with multiple compilers are in use.
+
+## Coding patterns
+
+* DALES can be run in parallel using MPI. The domain is split in `nprocx * nprocy` tiles in the horizontal direction.
+Each tile contains `imax * jmax * kmax` grid points. The whole domain contains `itot * jtot * kmax` grid points.
+* index ranges for the 3D fields are `(2:i1, 2:j1, 1:kmax)` for historical reasons.
+  `i1 = imax+1` and `j1 = jmax+1`. 
+* Ghost cells are added at the edges of the 3D fields of each process, and contain values copied from neighboring tiles. `ih` and `jh` are the number of ghost cells in the x and y directions. Generally 1 <= ih,jh <= 3, depending on the advection schemes used.
+A 3D field including ghost cells is indexed like this: `allocate(thl0 (2-ih:i1+ih,2-jh:j1+jh,k1))`. 
+The "proper" grid points are in the range (2:i1, 2:j1, 1:kmax), and ih or jh ghost cells are added on each side in the horizontal directions.
+* warm starts should be bitwise exact - after a restart the model should continue as if the restart never happened. But see the note of reproducibility with MPI above.
+* field naming: `thl0` is the thl value at the current (sub)time step,
+  `thlm` is the value at the beginning of the full step, and `thlp` is the tendency
+  calculated for the current time step. 
+
+* the top level file calls procedures for different physical processes and statistics, each procedure keeps track
+  of when it should run, `tnext_...` 
+
+## Reproducibility
+
+* DALES runs that are even slightly different will diverge and become
+  more and more different over time.
+
+* Parallel runs, especially with more than one node are not bitwise
+  reproducible, because MPI does not guarantee bitwise reproducibility
+  (probably the reduce operations which can happen in different
+  order).
+
+
+## Programming quirks
+
+* the startup order is fragile, be careful if re-ordering
+* Fortran doesn't allow circular module references. This can
+  create confusing errors, where compilation stops working olny when
+  one does a clean build.
+* time is counted both with floating point number of seconds, and integer number of milliseconds.
+
diff --git a/book/running/compilation.md b/book/running/compilation.md
@@ -1,6 +1,8 @@
-# Downloading the code
+# Compiling DALES
 
-The next step after installed these libraries and packages is to get the DALES code. The example below uses the `Git` software for this purpose. After going to the destination path on your system, download the code by using the command below
+## Downloading the code
+
+The next step after installing these libraries and packages is to get the DALES code. The example below uses the `Git` software for this purpose. After going to the destination path on your system, download the code by using the command below
 ``` shell
 <username> <currentpath> % git clone https://github.com/dalesteam/dales.git
 ```
@@ -15,6 +17,8 @@ which should give as output
  * main
 ```
 
+Note: until DALES 5.0 has been released, this manual applies only to the `dev` branch.
+
 Other available branches of the main repository can be viewed with the `-a` option
 ``` shell
 <username> dales % git branch -a
@@ -27,9 +31,9 @@ remotes/origin/4.1_aero_m7
 ...
 ```
 
-Checking out a specific branch (e.g., the `ruisdael` branch) is done via
+Checking out a specific branch (e.g., the `dev` branch) is done via
 ``` shell
-git checkout -b <newname> remotes/origin/ruisdael
+git checkout dev
 ```
 
 ### Initialize submodules
@@ -69,20 +73,21 @@ This option will checkout the code in your current directory, _without_ first ma
 ## Generic compilation
 The next step after obtaining the code and installing required dependencies, is to build the code. As a starting point, we assume you are still in the main `dales` directory after checking out the main branch via git. It is advised the build the code in a different folder to better maintain overview. We therefore first move one directory up, make a new directory and move into that directory.
 ``` shell
-cd ../
+
 mkdir build
 cd build
 ```
 
 We then invoke `cmake` on the directory where the code is placed. This will configure the build of the code.
 ``` shell
-cmake ../dales
+cmake ..
 ```
 Finally, the code can be build with the following command:
 ``` shell
-make
+make -j 4
 ```
-After successfull compilation, the executable `dales` is located in the subdirectory `bin/`. Compilation of the code may be sped up using the `-j <nprocs>` specifier, with `nprocs` being the amount of parallel processes.
+Compilation of the code may be sped up using the `-j <nprocs>` specifier, with `nprocs` being the amount of parallel processes (4 to 8 is usually a good choice). After successfull compilation, the executable `dales` is located in the subdirectory `bin/`.
+
 
 ## Compilation options
 It is possible to specify optional features at the compilation stage of the model. These optional features can be activated by adding them as specifyers to the cmake command. CMake options are specified as `-D<option>=<value>`. For example:
@@ -98,6 +103,7 @@ will produce a debug build. The debug build is much slower than the release buil
 | `-DENABLE_FP32_FIELDS` | Use single precision floating-point numbers for prognostic fields (momentum, temperature, etc.) | True/False | False |
 | `-DENABLE_FP32_POIS` | Use single precision floating-point numbers for the Poisson solver | True/False | False |
 | `-DENABLE_ACC` | Build with GPU support through OpenACC | True/False | False |
+| `-DCMAKE_BUILD_TYPE=Debug` | Release build (fast) or Debug build (slow, with many error checks) | Release,Debug | Releas |
 
 ```{caution}
 To use HYPRE or FFTW, the library needs to both be enabled at compilation and selected at runtime by setting the &SOLVER section of the namoptions input file. See [Alternative Poisson solvers (Wiki)](https://github.com/dalesteam/dales/wiki/Alternative-Poisson-solvers). By default, FFTW is used if the library was found at compilation.
@@ -106,8 +112,35 @@ To use HYPRE or FFTW, the library needs to both be enabled at compilation and se
 ```{hint}
 `-DPOIS_PRECISION=32` and `-DUSE_HYPRE=True` can be used together, but HYPRE is 64-bit only, so only use this, if it is really what you want.
 ```
+(sec:compilation:systems)=
+## Compilation on specific systems
+
+(sec:compilation:debian)=
+### Debian and Ubuntu
+
+Also for Ubuntu in WSL (Windows).
+
+```
+# install requirements with the system package manager (needed once)
+sudo apt install git cmake gfortran libnetcdf-dev libnetcdff-dev libopenmpi-dev libhypre-dev libfftw3-dev
+
+# extra tools for working with netCDF data
+# netcdf-bin is included for the ncdump tool
+sudo apt install cdo ncview netcdf-bin
+
+git clone https://github.com/dalesteam/dales.git
+cd dales
+
+git checkout dev
+git submodule init
+git submodule update
+
+mkdir build
+cd build
+cmake ..
+make -j 4
+```
 
-## Compilation on specific clusters
 
 ### Delftblue (TUDelft HPC)
 **Tested 20-11-2023**
@@ -119,39 +152,83 @@ module load netcdf-fortran/4.6.0
 module load fftw/3.3.10
 
 git clone https://github.com/dalesteam/dales
+cd dales
 
 # for dev branch:
 git checkout dev
 git submodule init
 git submodule update
 
-
-cd dales
 mkdir build
 cd build
-
-export SYST=gnu-fast
-cmake ..  -DUSE_FFTW=True
+cmake ..
 
 make -j 8
 ```
 
+
+
 ### Snellius (Dutch National system)
-**Tested with module set 2023**
+**Tested with module set 2025**
 ``` shell
-module load 2023
-module load foss/2023a
-module load netCDF-Fortran/4.6.1-gompi-2023a
-module load CMake/3.26.3-GCCcore-12.3.0
-# module load Hypre/2.29.0-foss-2023a
+module load 2025
+module load foss/2025a
+module load netCDF-Fortran/4.6.2-gompi-2025a
+module load CMake/3.31.3-GCCcore-14.2.0
+#module load Hypre/2.33.0-foss-2025a
+module load FFTW/3.3.10-GCC-14.2.0
 
 mkdir build
 cd build
-export SYST=gnu-fast
 cmake ..
 
-# or, for single precision (options for version >= 5.0 or dev):
-# cmake ../dales -DENABLE_FP32_FIELDS=ON -DENABLE_FP32_POIS=ON
+# or, for single precision:
+# cmake .. -DENABLE_FP32_FIELDS=ON -DENABLE_FP32_POIS=ON
 
 make -j 8
 ```
+
+### TU Delft VR-lab
+```
+ssh login03
+module load dales.rc4
+module load cmake fftw netcdf-fortran
+
+git clone https://github.com/dalesteam/dales
+cd dales
+git checkout dev
+git submodule init
+git submodule update
+
+mkdir build
+cd build
+cmake ..
+make -j 8
+```
+
+### Fugaku (2026)
+
+Fugaku uses `spack` to load modules. Extra steps needed are to pass the fortran and C++ compiler to cmake,
+and some flags for linking the hdf5 library.
+
+
+```
+. /vol0004/apps/oss/spack/share/spack/setup-env.sh
+spack load netcdf-fortran/gixlbro
+spack load fftw%fj@4.12.1
+spack load cmake@3.31.8/s7xjmta
+
+git clone https://github.com/dalesteam/dales
+cd dales
+git checkout dev
+git submodule init
+git submodule update
+
+mkdir build
+cd build
+
+FC=mpifrtpx CC=mpifccpx cmake ../dales -DENABLE_FP32_FIELDS=True  -DENABLE_FP32_POIS=True   -DCMAKE_Fortran_FLAGS="-L/vol0004/apps/oss/spack-v1.0.1/opt/spack/linux-a64fx/hdf5-1.14.6-mngsrojzjw54lchhyi4d3ujre3r55yef/lib -lhdf5 -lhdf5_hl"
+
+make -j 4 2>&1 | tee compilation-log.txt
+
+```
diff --git a/book/running/gpu.md b/book/running/gpu.md
@@ -1,6 +1,6 @@
 (sec:gpu)=
 # Using the GPU
-DALES now has the option to use graphics processing units (GPU's) to accelerate computations. 
+DALES now has the option to use graphics processing units (GPU's) to accelerate computations.
 
 ## Prerequisites
 
@@ -74,4 +74,77 @@ If you have compiled DALES succesfully with OpenACC enabled, running it is not v
 mpirun -np N <path-to-DALES> <path-to-namoptions>
 ```
 
-where N should match the number of GPU's you want to use. 
+where N should match the number of GPU's you want to use.
+
+
+
+# Compiling on specific GPU systems
+
+https://doc.dhpc.tudelft.nl/delftblue/ (requires login)
+
+## DelftBlue GPU (2026)
+
+### Compile netcdf-fortran
+
+```
+module load nvhpc/25.7
+module load netcdf-c/4.9.2
+module load cmake/3.30.5
+module load hdf5/1.14.3
+
+git clone https://github.com/unidata/netcdf-fortran
+cd netcdf-fortran
+
+cmake .. -DCMAKE_INSTALL_PREFIX=$HOME
+make -j 4
+make install
+```
+
+### Compile DALES for GPU (dev branch, single precision)
+```
+module load nvhpc/25.7
+module load netcdf-c/4.9.2
+module load hdf5/1.14.3
+module load cmake/3.30.5
+
+git clone https://github.com/dalesteam/dales
+git checkout dev
+git submodule init
+git submodule update
+
+cd dales
+mkdir build-gpu-sp
+cd build-gpu-sp
+
+cmake .. -DENABLE_FP32_POIS=True -DENABLE_FP32_FIELDS=True -DENABLE_ACC=True -DNetCDF_Fortran_ROOT=$HOME
+make -j 4
+```
+
+### Job script
+to be submitted in the directory where the simulation will run, with the input files already in place.
+
+```
+#!/bin/bash
+#SBATCH --job-name="dales"
+#SBATCH --partition=gpu-a100
+#SBATCH --time=1:00:00
+#SBATCH --ntasks=1
+#SBATCH --cpus-per-task=8
+#SBATCH --gpus-per-task=1
+#SBATCH --mem-per-cpu=8000M
+#SBATCH --account=research-ceg-grs
+
+# you may have to adapt the account line above,
+# according to your account and group status
+
+module load nvhpc/25.7
+module load netcdf-c/4.9.2
+#module load cmake/3.30.5
+module load hdf5/1.14.3
+
+DALES=$HOME/dales/build-gpu-sp/bin/dales
+
+srun --mpi=pmix -n 1 mpiexec -n 1 $DALES namoptions.001
+
+```
+Note: srun and mpiexec on the last line seem redundant, mpiexec seem required, uncertain about srun.