From 7dad78055b47340161c32de4e0588dcfa3a4debb Mon Sep 17 00:00:00 2001 From: Fredrik Jansson Date: Thu, 6 Aug 2026 14:00:16 +0200 Subject: [PATCH] Edit and expand setup and compilation instructions --- book/_toc.yml | 4 ++ book/developers/developers.md | 59 ++++++++++++++++ book/running/compilation.md | 123 +++++++++++++++++++++++++++------- book/running/gpu.md | 77 ++++++++++++++++++++- book/running/quirks.md | 69 +++++++++++++++++++ book/running/settingup.md | 3 + book/running/tracers.001.nc | Bin 7152 -> 7152 bytes 7 files changed, 310 insertions(+), 25 deletions(-) create mode 100644 book/developers/developers.md create mode 100644 book/running/quirks.md diff --git a/book/_toc.yml b/book/_toc.yml index 6e83772..d735ea4 100644 --- a/book/_toc.yml +++ b/book/_toc.yml @@ -22,6 +22,10 @@ parts: # - file: running/ibm.ipynb # - file: running/openbcs.ipynb - file: running/gpu.md + - file: running/quirks.md + - caption: Hints for new DALES developers + chapters: + - file: developers/developers.md - caption: Recent publications using DALES chapters: - file: running/papers.ipynb diff --git a/book/developers/developers.md b/book/developers/developers.md new file mode 100644 index 0000000..fdc0ffc --- /dev/null +++ b/book/developers/developers.md @@ -0,0 +1,59 @@ +# For new developers + +* git and GitHub are used for DALES development. See this [Software Carpentry course](http://swcarpentry.github.io/git-novice/) for git basics. Especially, learn about + - git clone (download DALES from github to your computer) + - branches (different "versions" of the DALES code, for example for developing a specific function) + - forking (make a copy of the DALES repository to your own GitHub account) + +* Avoid distributing DALES source code outside git, e.g. by email. It's hard to get the + code back into git. If you want to send a version to someone else, make a new + branch (in your own fork) for it, and send the link. + +* When changing the official version: we don't want surprise changes + to the default behavior (exceptions: fixing bugs). New features + should in general be introduced with namelist flags to turn them on, + and the default should be the old behavior. + +* Prefer netCDF output over ASCII (netCDF is standardized, more robust, better + tools available, easier for others to process). + +* DALES should work with multiple Fortran compilers. gfortran and intel are most + used at the moment. Automatic tests with GitHub actions, which verify that the code + compiles with multiple compilers are in use. + +## Coding patterns + +* DALES can be run in parallel using MPI. The domain is split in `nprocx * nprocy` tiles in the horizontal direction. +Each tile contains `imax * jmax * kmax` grid points. The whole domain contains `itot * jtot * kmax` grid points. +* index ranges for the 3D fields are `(2:i1, 2:j1, 1:kmax)` for historical reasons. + `i1 = imax+1` and `j1 = jmax+1`. +* Ghost cells are added at the edges of the 3D fields of each process, and contain values copied from neighboring tiles. `ih` and `jh` are the number of ghost cells in the x and y directions. Generally 1 <= ih,jh <= 3, depending on the advection schemes used. +A 3D field including ghost cells is indexed like this: `allocate(thl0 (2-ih:i1+ih,2-jh:j1+jh,k1))`. +The "proper" grid points are in the range (2:i1, 2:j1, 1:kmax), and ih or jh ghost cells are added on each side in the horizontal directions. +* warm starts should be bitwise exact - after a restart the model should continue as if the restart never happened. But see the note of reproducibility with MPI above. +* field naming: `thl0` is the thl value at the current (sub)time step, + `thlm` is the value at the beginning of the full step, and `thlp` is the tendency + calculated for the current time step. + +* the top level file calls procedures for different physical processes and statistics, each procedure keeps track + of when it should run, `tnext_...` + +## Reproducibility + +* DALES runs that are even slightly different will diverge and become + more and more different over time. + +* Parallel runs, especially with more than one node are not bitwise + reproducible, because MPI does not guarantee bitwise reproducibility + (probably the reduce operations which can happen in different + order). + + +## Programming quirks + +* the startup order is fragile, be careful if re-ordering +* Fortran doesn't allow circular module references. This can + create confusing errors, where compilation stops working olny when + one does a clean build. +* time is counted both with floating point number of seconds, and integer number of milliseconds. + diff --git a/book/running/compilation.md b/book/running/compilation.md index decd122..0fa9f1f 100644 --- a/book/running/compilation.md +++ b/book/running/compilation.md @@ -1,6 +1,8 @@ -# Downloading the code +# Compiling DALES -The next step after installed these libraries and packages is to get the DALES code. The example below uses the `Git` software for this purpose. After going to the destination path on your system, download the code by using the command below +## Downloading the code + +The next step after installing these libraries and packages is to get the DALES code. The example below uses the `Git` software for this purpose. After going to the destination path on your system, download the code by using the command below ``` shell % git clone https://github.com/dalesteam/dales.git ``` @@ -15,6 +17,8 @@ which should give as output * main ``` +Note: until DALES 5.0 has been released, this manual applies only to the `dev` branch. + Other available branches of the main repository can be viewed with the `-a` option ``` shell dales % git branch -a @@ -27,9 +31,9 @@ remotes/origin/4.1_aero_m7 ... ``` -Checking out a specific branch (e.g., the `ruisdael` branch) is done via +Checking out a specific branch (e.g., the `dev` branch) is done via ``` shell -git checkout -b remotes/origin/ruisdael +git checkout dev ``` ### Initialize submodules @@ -69,20 +73,21 @@ This option will checkout the code in your current directory, _without_ first ma ## Generic compilation The next step after obtaining the code and installing required dependencies, is to build the code. As a starting point, we assume you are still in the main `dales` directory after checking out the main branch via git. It is advised the build the code in a different folder to better maintain overview. We therefore first move one directory up, make a new directory and move into that directory. ``` shell -cd ../ + mkdir build cd build ``` We then invoke `cmake` on the directory where the code is placed. This will configure the build of the code. ``` shell -cmake ../dales +cmake .. ``` Finally, the code can be build with the following command: ``` shell -make +make -j 4 ``` -After successfull compilation, the executable `dales` is located in the subdirectory `bin/`. Compilation of the code may be sped up using the `-j ` specifier, with `nprocs` being the amount of parallel processes. +Compilation of the code may be sped up using the `-j ` specifier, with `nprocs` being the amount of parallel processes (4 to 8 is usually a good choice). After successfull compilation, the executable `dales` is located in the subdirectory `bin/`. + ## Compilation options It is possible to specify optional features at the compilation stage of the model. These optional features can be activated by adding them as specifyers to the cmake command. CMake options are specified as `-D