Skip to content

Commit 62d5273

Browse files
zmoonzmoon
committed
Draft global run
1 parent 7ec7095 commit 62d5273

4 files changed

Lines changed: 260 additions & 20 deletions

File tree

‎01-compile.md‎

Lines changed: 33 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,17 @@ title: Build
33
subtitle: Building MPAS on NSF NCAR Derecho
44
---
55

6+
First we need to clone the model repository.
7+
We will use Kelly's fork, which adds additional vertical levels (based on ERA5)
8+
and isobaric diagnostics.
9+
10+
```{code} bash
11+
cd $HOME
12+
git clone https://github.com/knubez/MPAS-Model.git MPAS-Model_v8.3
13+
cd $_
14+
git switch isolevels-v8
15+
```
16+
617
It's good practice to compile the model in an interactive job,
718
to avoid stressing the login nodes, which are a shared resource,
819
_and_ to ensure that the CPU resources detected at build time are the same as those used at run time[^derecho-nodes].
@@ -16,6 +27,18 @@ _and_ to ensure that the CPU resources detected at build time are the same as th
1627
qsub -I -l walltime=3600 -l select=1:ncpus=4:mem=80gb -A UTAM0025 -q develop
1728
```
1829

30+
```{tip}
31+
:open: false
32+
33+
NCAR also [provides](https://ncar-hpc-docs.readthedocs.io/en/latest/pbs/)
34+
`qcmd` and `qinteractive` shortcut commands that you can use,
35+
but the above `qsub` should work on other PBS-based systems as well.
36+
37+
The [`develop` queue](https://ncar-hpc-docs.readthedocs.io/en/latest/pbs/charging/#derecho-queues)
38+
has a 6-hour walltime limit and is intended for testing and development.
39+
Above we have requested 3600 seconds (1 hour).
40+
```
41+
1942
After our session starts, we need to load some modules.
2043

2144
::::{tab-set}
@@ -56,6 +79,9 @@ module load intel/2025.1.0
5679
module load cray-mpich/8.1.29
5780
module load parallel-netcdf/1.14.0
5881
82+
# Hack to get pnetcdf working again
83+
export LD_LIBRARY_PATH="/glade/u/apps/derecho/24.12/spack/opt/spack/parallel-netcdf/1.14.0/cray-mpich/8.1.29/oneapi/2025.1.0/vrc7/lib:$LD_LIBRARY_PATH"
84+
5985
# Tell mpifort to use ifx
6086
export MPICH_FC=ifx
6187
```
@@ -64,16 +90,16 @@ export MPICH_FC=ifx
6490

6591
::::
6692

67-
```{tip}
68-
NCAR also [provides](https://ncar-hpc-docs.readthedocs.io/en/latest/pbs/)
69-
`qcmd` and `qinteractive` shortcut commands that you can use,
70-
but the above `qsub` should work on other PBS-based systems as well.
93+
Select one of the above module sets and save it to a file:
94+
`~/mpas-modules-intel.sh`.
95+
Then, load the modules.
7196

72-
The [`develop` queue](https://ncar-hpc-docs.readthedocs.io/en/latest/pbs/charging/#derecho-queues)
73-
has a 6-hour walltime limit and is intended for testing and development.
74-
Above we have requested 3600 seconds (1 hour).
97+
```bash
98+
source ~/mpas-modules-intel.sh
7599
```
76100

101+
Run `module list` to verify that the correct modules are loaded.
102+
77103
First we build the model initialization program.
78104
This is what we use to generate initial conditions and such.
79105

‎02-run.md‎

Lines changed: 214 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -7,33 +7,111 @@ subtitle: Running MPAS on NSF NCAR Derecho
77

88
We will use the quasi-uniform 240-km mesh (10242 cells),
99
as in the [official virtual tutorial](https://www2.mmm.ucar.edu/projects/mpas/tutorial/Virtual2025/)
10-
(section 1.3).
10+
(Section 1.3).
11+
12+
## Run directory
1113

1214
First we build out our run directory.
1315

1416
```{code} bash
1517
cd $SCRATCH
1618
mkdir mpas-global-240km
1719
cd $_
20+
1821
ln -s /glade/campaign/mmm/wmr/mpas_tutorial/meshes/x1.10242.grid.nc .
19-
ln -s /glade/campaign/mmm/wmr/mpas_tutorial/meshes/x1.10242.graph.info.part.16 .
20-
cp /glade/campaign/mmm/wmr/mpas_tutorial/job_scripts/init_real.pbs .
22+
23+
ln -s /glade/campaign/mmm/wmr/mpas_tutorial/meshes/x1.10242.graph.info.part.32 .
24+
ln -s ~/MPAS-Model_v8.3/init_atmosphere_model .
25+
cp ~/MPAS-Model_v8.3/namelist.init_atmosphere .
26+
cp ~/MPAS-Model_v8.3/streams.init_atmosphere .
27+
28+
ln -s /glade/campaign/mmm/wmr/mpas_tutorial/meshes/x1.10242.graph.info.part.128 .
29+
ln -s ~/MPAS-Model_v8.3/atmosphere_model .
30+
cp ~/MPAS-Model_v8.3/namelist.atmosphere .
31+
cp ~/MPAS-Model_v8.3/streams.atmosphere .
32+
cp ~/MPAS-Model_v8.3/stream_list.atmosphere.*
33+
cp ~/MPAS-Model_v8.3/src/core_atmosphere/physics/physics_wrf/files/* .
2134
```
2235

23-
TODO: above job script is for GNU, gotta update
36+
## Static file
37+
38+
Now we edit the `init_atmosphere` config files to set up for generating the static file.
39+
40+
Update `namelist.init_atmosphere` with these settings:
41+
42+
| parameter | value |
43+
| ----------------------------------------------- | ---------------------------------------- |
44+
| `nhyd_model.config_init_case` | `7` [^init-case7] |
45+
| `data_sources.geog_data_path` | `'/glade/work/wrfhelp/WPS_GEOG/'` [^gdp] |
46+
| `preproc_stages.config_static_interp` | `true` |
47+
| `preproc_stages.config_native_gwd_static` | `true` |
48+
| `preproc_stages.config_native_gwd_gsl_static` | `false` |
49+
| `preproc_stages.config_vertical_grid` | `false` |
50+
| `preproc_stages.config_met_interp` | `false` |
51+
| `preproc_stages.config_input_sst` | `false` |
52+
| `preproc_stages.config_frac_seaice` | `false` |
53+
| `decomposition.config_block_decomp_file_prefix` | `'x1.10242.graph.info.part.'` |
54+
55+
[^init-case7]: Case 7 is the "real-data initialization" case.
56+
57+
[^gdp]: Another geog data path option is `/glade/campaign/mmm/wmr/mpas_tutorial/mpas_static/`.
58+
59+
Then, in `streams.init_atmosphere`, set the input file name template to `x1.10242.grid.nc`,
60+
and the output file name template to `x1.10242.static.nc`.
61+
62+
Create a job script to run the model initialization program.
63+
64+
```{code} bash
65+
:filename: init_real.pbs
66+
67+
#!/usr/bin/env bash
68+
# Based on /glade/campaign/mmm/wmr/mpas_tutorial/job_scripts/init_real.pbs
69+
70+
#
71+
# For more information on submitting jobs to Derecho, see this documentation:
72+
# https://ncar-hpc-docs.readthedocs.io/en/latest/pbs/job-scripts/
73+
#
74+
75+
#--- Give our job a reasonable name
76+
#PBS -N init_real
77+
78+
#--- Run in the main queue
79+
#PBS -q main
80+
#PBS -l job_priority=regular
81+
82+
#--- Set the project under which the job will run
83+
#PBS -A UTAM0025
84+
85+
#--- Specify wallclock limit and resources
86+
#PBS -l walltime=00:30:00
87+
#PBS -l select=1:ncpus=128:mpiprocs=32:mem=235gb
88+
89+
source ~/mpas-modules-intel.sh
90+
91+
mpiexec ./init_atmosphere_model
92+
```
93+
94+
Submit the job.
2495

2596
```bash
26-
qsub init_real.pbs -A UTAM0025
97+
qsub init_real.pbs
2798
```
2899

29100
:::{tip}
101+
:open: false
30102

31-
Get info about running jobs.
103+
Get info about your running jobs:
32104

33105
```bash
34106
qstat -u $USER
35107
```
36108

109+
Watch it:
110+
111+
```bash
112+
watch -n1 qstat -u $USER
113+
```
114+
37115
See more:
38116

39117
```bash
@@ -42,16 +120,142 @@ qstat -u $USER -f -w
42120

43121
:::
44122

123+
## Initial conditions
124+
125+
**20 September 2017, 00 UTC**, the day when Hurricane Maria made landfall in Puerto Rico.
126+
127+
First we need [WPS intermediate files](https://www2.mmm.ucar.edu/wrf/OnLineTutorial/Basics/IM_files/).
128+
129+
The [era5_to_int](https://github.com/NCAR/era5_to_int) tool
130+
can be used to create intermediate files from the NSF NCAR RDA ERA5 netCDF files on GLADE.
131+
But here, to simplify things, we will use an intermediate file that we have already created.
132+
133+
```bash
134+
ln -s /glade/u/home/zmoon/mpas-tutorial/FILE:2017-09-20_00 .
135+
```
136+
137+
:::{tip}
138+
:open: false
139+
140+
You can see what's inside an intermediate file using the `rd_intermediate` tool from WPS.
141+
On Derecho, this tool is available at
142+
143+
```
144+
/glade/u/home/wrfhelp/derecho_pre_compiled_code/wpsv4.5/util/rd_intermediate.exe
145+
```
146+
147+
(and for other WPS versions as well).
148+
:::
149+
150+
Update `namelist.init_atmosphere` with these settings:
151+
152+
| parameter | value |
153+
| ----------------------------------------------- | ----------------------------- |
154+
| `nhyd_model.config_init_case` | `7` |
155+
| `nhyd_model.config_start_time` | `'2017-09-20_00:00:00'` |
156+
| `data_sources.config_met_prefix` | `'FILE'` |
157+
| `preproc_stages.config_static_interp` | `false` |
158+
| `preproc_stages.config_native_gwd_static` | `false` |
159+
| `preproc_stages.config_native_gwd_gsl_static` | `false` |
160+
| `preproc_stages.config_vertical_grid` | `true` |
161+
| `preproc_stages.config_met_interp` | `true` |
162+
| `preproc_stages.config_input_sst` | `false` |
163+
| `preproc_stages.config_frac_seaice` | `true` |
164+
| `decomposition.config_block_decomp_file_prefix` | `'x1.10242.graph.info.part.'` |
165+
166+
Then, in `streams.init_atmosphere`, set the input file name template to `x1.10242.static.nc`
167+
(the static file we just created),
168+
and the output file name template to `x1.10242.init.nc`.
169+
170+
Submit the job.
171+
172+
```bash
173+
qsub init_real.pbs
174+
```
175+
176+
```{note}
177+
For simulations longer than a few days,
178+
you would likely want to also create SST and sea-ice update files.
179+
This uses init case 8, the "surface field initialization" case.
180+
See Section 3.2 in the [official virtual tutorial](https://www2.mmm.ucar.edu/projects/mpas/tutorial/Virtual2025/).
181+
```
182+
183+
## Run the model
184+
185+
Update `namelist.atmosphere` with these settings:
186+
187+
| parameter | value |
188+
| ----------------------------------------------- | ----------------------------- |
189+
| `nhyd_model.config_dt` | `1200.0` |
190+
| `nhyd_model.config_start_time` | `'2017-09-20_00:00:00'` |
191+
| `nhyd_model.config_run_duration` | `'3_00:00:00'` |
192+
| `nhyd_model.config_radtlw_interval` | `'01:00:00'` |
193+
| `nhyd_model.config_radtsw_interval` | `'01:00:00'` |
194+
| `decomposition.config_block_decomp_file_prefix` | `'x1.10242.graph.info.part.'` |
195+
196+
Then, in `streams.atmosphere`, set the input file name template to `x1.10242.init.nc`
197+
(the initial conditions file we just created).
198+
199+
Create a job script to run the model.
200+
201+
```{code} bash
202+
:filename: run_model.pbs
203+
204+
#!/usr/bin/env bash
205+
# Based on /glade/campaign/mmm/wmr/mpas_tutorial/job_scripts/run_model.pbs
206+
207+
#
208+
# For more information on submitting jobs to Derecho, see this documentation:
209+
# https://ncar-hpc-docs.readthedocs.io/en/latest/pbs/job-scripts/
210+
#
211+
212+
#--- Give our job a reasonable name
213+
#PBS -N run_model
214+
215+
#--- Run in the main queue
216+
#PBS -q main
217+
#PBS -l job_priority=regular
218+
219+
#--- Set the project under which the job will run
220+
#PBS -A UTAM0025
221+
222+
#--- Specify wallclock limit and resources
223+
#PBS -l walltime=00:30:00
224+
#PBS -l select=1:ncpus=128:mpiprocs=128:mem=235gb
225+
226+
source ~/mpas-modules-intel.sh
227+
228+
mpiexec ./atmosphere_model
229+
```
230+
231+
Submit the job.
232+
233+
```bash
234+
qsub run_model.pbs
235+
```
236+
237+
The main model log is `log.atmosphere.0000.out`.
238+
To follow the updates:
239+
240+
```bash
241+
tail -f log.atmosphere.0000.out
242+
```
243+
244+
A successful run will produce various model output files:
245+
246+
- `diag.*.nc` (mostly 2-d diagnostic fields; specified in `stream_list.atmosphere.diagnostics`)
247+
- `history.*.nc` (2- and 3-d prognostic and diagnostic fields; specified in `stream_list.atmosphere.output`)
248+
- `restart.*.nc` (checkpoints of the model state that we can use for a warm start)
249+
45250
# Regional
46251

47252
We will use the limited-area domain as in:
48253

49254
- https://doi.org/10.1029/2023MS004070
50255
- https://doi.org/10.1029/2024GL112341
51-
- https://doi.org/10.22541/essoar.174547926.68665522/v1
256+
- https://doi.org/10.1029/2025MS005146
52257
- https://doi.org/10.22541/essoar.174547944.43729039/v1
53-
- https://github.com/knubez/MPAS-Model/tree/isolevels-v8
54258

55-
created by rotating, moving, and cropping the `x5.8060930` global 15--3-km elliptical refinement mesh.
259+
created by rotating, moving, and cropping the `x5.8060930` global 15–3-km elliptical refinement mesh.
56260

57-
## Creating LBCs
261+
## Boundary conditions

‎myst.doi.bib‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ @article{Nunez_Ocasio_2024_b
3232

3333

3434
@article{pgw_aew_preprint,
35-
author = {Ocasio, Kelly Marie N{\' u}{\~ n}ez and Dougherty, Erin and Moon, Zachary and Davis, Christopher A},
35+
author = {N{\' u}{\~ n}ez Ocasio, Kelly and Dougherty, Erin and Moon, Zachary and Davis, Christopher A},
3636
doi = {10.22541/essoar.174547926.68665522/v1},
3737
year = {2025},
3838
month = {4},
@@ -43,7 +43,7 @@ @article{pgw_aew_preprint
4343

4444

4545
@article{pgw_mcs_preprint,
46-
author = {Ocasio, Kelly Marie N{\' u}{\~ n}ez and Dougherty, Erin and Moon, Zachary},
46+
author = {N{\' u}{\~ n}ez Ocasio, Kelly Marie and Dougherty, Erin and Moon, Zachary},
4747
doi = {10.22541/essoar.174547944.43729039/v1},
4848
year = {2025},
4949
month = {4},

‎myst.yml‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,17 @@ project:
55
title: MPAS Tutorial
66
description: A mini tutorial for standalone MPAS-A
77
keywords: [MPAS, MPAS-A]
8-
authors: [K. M. Núñez Ocasio, Z. Moon]
8+
authors:
9+
- name: K. M. Núñez Ocasio
10+
id: kelly
11+
github: knubez
12+
email: knunez.ocasio@tamu.edu
13+
url: https://sites.google.com/tamu.edu/nunez-ocasio/home
14+
- name: Z. L. Moon
15+
id: zach
16+
github: zmoon
17+
email: zmoon@tamu.edu
18+
url: https://zmoon.info
919
github: https://github.com/NunezOcasioLab/mpas-tutorial
1020
site:
1121
template: book-theme

0 commit comments

Comments
 (0)