DART+POP2 ASD Project

This documents our DART+POP2 data assimilation ASD Project.

User Guide

Logging on to the system

ssh $USER@derecho.hpc.ucar.edu

Queues

Ben Kirk posted to Slack to use the following command to test queue access:

qcmd -l select=2:ncpus=128:mpiprocs=128:mem=200G -A A####### -l walltime=0:05:00 -- "date && hostname"
Waiting on job launch; 1230726.desched1 with qsub arguments:
   qsub  -l select=2:ncpus=128:mpiprocs=128:mem=200G -A A####### -q main@desched1 -l walltime=0:05:00

Thu 15 Jun 2023 10:45:21 PM MDT
dec0476

This command gives an overview of the queues:

qstat -Qf

MPI Job Example

#!/bin/bash
#PBS -A project_code
#PBS -N mpi_job
#PBS -q main
#PBS -l walltime=01:00:00
#PBS -l select=2:ncpus=128:mpiprocs=128

# load necessary module environment
module purge
module load ncarenv craype cce cray-mpich

# Run application using cray-mpich MPI
mpiexec -n 256 -ppn 128 ./executable_name

Accelerated Scientific Discovery

Overview

Note

This project has cleared thie initial proposal stage and the final proposal is due prior to January 10, 2022.

According to the full proposal request:

Your full proposal should follow the guidelines provided for University ASD proposals at https://www2.cisl.ucar.edu/university-asd. This will be similar to the normal NSC proposal format, with additional attention to project readiness, team commitment, and your analysis and data management plan. You may omit discussion of strategic relevance since that has been assessed in Phases 1 and 2.

Note that your full proposal should be completed and submitted via the allocation submission site.

Warning

The CISL “University ASD” link above was broken when the new CISL website was unvieled in December 2021. The guidelines from that now-lost page are reproduced below.

Proposal document

Proposals should be five pages or less for Sections A–E below. The data management plan, accomplishment reports, references, and figures (Sections F–I) should be an additional five pages or less. The proposal should address each numbered item; please retain the section and item numbering and order in your responses.

Note: The request document must identify all PIs and Co-Is on any associated awards. In addition, the web submission form must include the names and affiliations of all PIs and Co-Is on the supporting NSF grant(s) and a complete list of collaborators and their institutions.

A. Project title and PIs

On the first page of the ASD proposal document please include the following:

ASD project title PIs and Co-Is on the supporting NSF award(s) along with their institution CPU core-hours and/or GPU hours requested Peak disk space needed

B. Scientific objectives

Describe the scientific problem that will be studied and its importance. This section should include discussion of related work recently completed by the PIs and other scientists along with supporting references; describe the linkage between the NSF award(s) and the proposed ASD activity (especially important if the published NSF award abstract does not clearly describe the computational component of the work funded or if the work is a “natural extension” of the funded work).

C. Computational plan

Discussion of your planned computational experiments and the resources needed should comprise the bulk of the five-page proposal.

Describe the numerical approach or models to be used. (For a non-standard or non-community model, the numerical description should briefly describe the approximations and other methods proposed to obtain valid solutions to the problem.)

Describe the computational experiments being proposed and how they relate to the scientific objectives in sufficient detail for reviewers to confirm the estimate of system resources required.

Describe the number of CPU core-hours and/or GPU hours required for each proposed experiment and the total resources for the request, including the total data written to disk, sufficient for reviewers to assess the appropriateness and efficiency of the data management plan. Explain how the number of core-hours per simulated year was determined.

Provide a table summarizing the number of CPU core-hours, GPU hours, and storage space required for each experimental configuration and the total for the proposed set of computational experiments.

D. Code requirements and readiness

The reviewers will use this section to assess the readiness of the code to run successfully and efficiently as soon as Derecho becomes available.

Describe the model or code to be used in a few sentences, including an assessment of its current readiness to run on Derecho. Please include the name of the model or code. You may cite a web page where the model or code is described.

Describe your proposed project’s special programming or runtime environment requirements. For example, required software, libraries, wall-clock time, scratch disk space needs, particularly if they differ significantly from what is currently available by default on Cheyenne today.

Benchmarks and scalability

Benchmarking system and results: Please summarize benchmark details or reference web page. Describe the model or code’s scalability on current systems and estimate the maximum number of Derecho nodes that could be used efficiently.

Other special requirements

E. Data analysis and visualization

ASD projects may be granted allocations on the Casper cluster in addition to Derecho allocations. CISL will provide help in constructing visualizations to improve understanding and presentation of results. Please describe plans or estimated needs for data analysis and visualization resources following the ASD project period on Derecho.

Supporting information

Sections F–I together should be an additional five pages or less.

F. Data management plan

Because of the potential for large-scale Derecho projects to produce extensive data output, the project’s short-term disk storage needs and the ASD project data targeted for longer-term storage should be described in Section C. The Data Management Plan should be used to provide additional explanation of the project’s data needs and plans.

Disk space

Describe the project’s medium-term disk needs for data analysis and visualization following completion of the primary HPC runs through the submission of the final report in March 2023.

Storage space

NCAR will provide storage space, commensurate with review panel recommendations, for up to one year after the ASD project ends. Describe any longer-term storage needs and/or relevant plans for sharing or managing the resulting data over the longer term.

Offsite data transfer

Discuss the amount of data to be transferred to another center and the frequency of transfers.

G. Accomplishment report on prior large allocations

The reviewers will use the accomplishment report to judge the proposing team’s experience with large-scale computations and their likelihood of ASD success on Derecho. The accomplishment report should focus on research in the past five years conducted with large (merit-reviewed) computing allocations at NCAR or other HPC centers. In addition to summarizing the computational experiments and scientific outcomes, the accomplishment report should note

the scientific publications produced,
the number of graduate students that worked on each large allocation project,
the theses produced, if any.

H. References

Please limit references to those directly related to the proposed project.

I. Figures and captions

Optional. Figures may be embedded within the first five pages of the proposal document; however, embedded figures will count against the five-page limit.

Compiling

For the ASD project, DART will be compiled with the PrgEnv-gnu programming environment and compilers recommended by HPE/Cray.

During the NERSC December 2021 GPU Hackathon, Johnson, along with DAReS Software Engineer Helen Kershaw and ASP Post-doc Chris Riedel, ported DART code to the Department of Energy’s Perlmutter machine, which is a Cray EX supercomputer (Derecho is also a Cray EX). Under the guidance of three development engineers from Nvidia, the team GPU-accelerated a kernel of DART’s source code using Nvidia’s nvfortran compiler and OpenACC directives. The code was accelerated by a factor of 15x using pinned memory, but the computed results were incorrect (there is a possible “off by one” bug in the compiler). Thus this project does not plan to use the PrgEnv-nvidia compilers.

Setup

Combining the obs_seq files into 5-day’s worth of observations

This python script proccess the WOD13 files into files containing 5-day’s worth of observations. It makes use of the f90nml packaged written by Marshall Ward.

The original file is on glade here: /glade/work/johnsonb/git/DART_derecho/models/POP/work/obs_sequence_tool_to_5day.py

#!/glade/apps/opt/modulefiles/idep python

from datetime import date, timedelta
import f90nml
import os


# To modify a namelist but preserve its comments and formatting, create a
# namelist patch and apply it to a target file using the patch function

start_year = 2010
start_month = 1
start_day = 1

obs_seq_path = '/glade/p/cisl/dares/Observations/WOD13/' # 200701/obs_seq.0Z.20070101
output_path = '/glade/derecho/scratch/johnsonb/obs_seq/'
start_date = date(start_year, start_month, start_day)
timedeltas = [timedelta(days=increment) for increment in range(0, 5)]

while start_date.year != 2017:

   # Get the list of dates in this 5-day window
   list_of_dates = [start_date + this_timedelta for this_timedelta in timedeltas]

   # Write the input files to the text file that will be read by obs_sequence_tool
   with open('./obs_sequence_tool_list.txt', 'w') as f:

      for this_date in list_of_dates:
         line = obs_seq_path + str(this_date.year).zfill(4) + str(this_date.month).zfill(2) + '/obs_seq.0Z.' + str(this_date.year).zfill(4) + str(this_date.month).zfill(2) + str(this_date.day).zfill(2)
         f.write(line + '\n')
         print(line)

   # Edit input.nml to set the output file
   filename_out = output_path + 'obs_seq.0Z.' + str(list_of_dates[0].year).zfill(4) + str(list_of_dates[0].month).zfill(2) + str(list_of_dates[0].day).zfill(2) + '_' + str(list_of_dates[-1].year).zfill(4) + str(list_of_dates[-1].month).zfill(2) + str(list_of_dates[-1].day).zfill(2)
   print('filename_out:', filename_out)

   # Patch input.nml
   patch_nml = {'obs_sequence_tool_nml': {'filename_out': filename_out}}
   f90nml.patch('./input.nml', patch_nml, './temp_input.nml')
   os.rename('./temp_input.nml', './input.nml')

   # Run obs_sequence_tool
   os.system('./obs_sequence_tool')

   # Increment to the next start date
   start_date = list_of_dates[-1] + timedelta(days=1)

Default CESM Grids

The project compares the behavior of POP2 ensembles using two different grid configurations: a low-resolution, eddy-parameterizing grid and a high-resolution, eddy-resolving grid.

The default available atmospheric grids for each are listed on the CESM Grid Resolution Definitions and are reproduced below.

Low-resolution, eddy parameterizing

The CESM grid used for the Coupled Model Intercomparison Project (CMIP) is a 1° grid with a displaced Greenland pole. It has two configurations within CESM: g16 and g17.

This is the same ocean/ice grid used for the CESM large ensemble and the DART CAM6 Reanalysis.

Each of these grids has been coupled to different atmospheric and land grids, which are listed in the Default configurations table rows below.

Grid	g16
Grid type	gx1v6 is displaced Greenland pole v6 1-deg grid
Domain file	`/glade/p/cesm/cseg/inputdata/share/domains/domain.ocn.gx1v6.090206.nc`
Domain
Default configurations	f02_g16, f05_g16, f09_g16, f19_g16, ne120_g16, ne240_f02_g16, ne30_f09_g16, ne30_f19_g16, ne30_g16, ne60_g16, T62_g16

Grid	g17
Grid type	gx1v7 is displaced Greenland pole 1-deg grid with Caspian as a land feature
Domain file	`/glade/p/cesm/cseg/inputdata/share/domains/domain.ocn.gx1v7.151008.nc`
Domain
Default configurations	f02_g17, f05_g17, f09_g17, f19_g17, f19_g17_r01, f19_g17_r05, g17_g17, ne0CONUSne30x8_g17, ne120_g17, ne120pg3_g17, ne16_g17, ne240_f02_g17, ne30_f09_g17, ne30_f19_g17, ne30_g17, ne30pg3_g17, ne60_g17, T62_g17, TL319_g17, TL639_g17

High-resolution, eddy resolving

The ~0.1° ocean/sea ice grid has two different configuratons within CESM: t12 and t13. Each of these grids has been coupled to different atmospheric and land grids. This is the same ocean/ice grid used in Johnson et al. (2016) and Deppenmeier et al. (2021).

Each of these grids has been coupled to different atmospheric and land grids, which are listed in the Default configurations table rows below.

Grid	t12
Grid type	tx0.1v2 is tripole v2 1/10-deg grid
Domain file	`/glade/p/cesm/cseg/inputdata/share/domains/domain.ocn.tx0.1v2.161014.nc`
Domain
Default configurations	f02_t12, f05_t12, ne120_t12, ne240_t12, T341_f02_t12, T62_t12

Grid	t13
Grid type	tx0.1v3 is tripole v3 1/10-deg grid with Caspian as a land feature
Domain file	`/glade/p/cesm/cseg/inputdata/share/domains/domain.ocn.tx0.1v3.170730.nc`
Domain
Default Configurations	ne120pg3_t13, T62_t13, TL319_t13

Defining a custom grid

In order to use the CAM6 reanalysis to force the t13 ocean/sea ice grid, a f09_t13 grid needs to be defined within CIME.

The Model grids page in the CIME documentation provides instructions for doing so.

Available SCRIP grid files

CIME doesn’t provide functionality to create Spherical Coordinate Remapping and Interpolation Package (SCRIP) since it is a Los Alamos product (Jones, 1999).

SCRIP files are the input files for the mapping utility, as described in Step 1 of Adding grids. But, there should be SCRIP files for f09 and t13 already on GLADE since those grids are already part of CESM. Let’s start searching.

vi /glade/p/cesm/cseg/mapping/grids/README

(2021-06-23)

Old files here have been moved to the inputdata directory:

inputdata/share/scripgrids

Moving forward, all important scrip grid files (e.g., those that relate to
standard CESM model resolutions) should be put in that directory and checked
in to the inputdata repository. (ESMF mesh files should be put in
inputdata/share/meshes.)

f09

The f09 scrip grids files are here:

/glade/p/cesm/cseg/inputdata/share/scripgrids/0.9x1.25_EW_SCRIP_desc.181018.nc
/glade/p/cesm/cseg/inputdata/share/scripgrids/0.9x1.25_NS_SCRIP_desc.181018.nc
/glade/p/cesm/cseg/inputdata/share/scripgrids/0.9x1.25_SCRIP_desc.181018.nc

t13

The t13 scrip grid files are here:

/glade/p/cesm/cseg/inputdata/share/scripgrids/tx0.1v3_211102.nc

Using f05_t12 as a starting point

The most similar existing grid combination to the f09_t13 grid that we are attempting to create is the f05_t12 grid. In /glade/work/johnsonb/cesm2_1_3/cime/config/cesm/config_grids.xml, there are two relevant sections regarding f05_T12:

<!-- Line 375 -->
<model_grid alias="f05_t12">
   <grid name="atm">0.47x0.63</grid>
   <grid name="lnd">0.47x0.63</grid>
   <grid name="ocnice">tx0.1v2</grid>
   <mask>tx0.1v2</mask>
</model_grid>

<!-- Line 1586 -->
<gridmap atm_grid="0.47x0.63" ocn_grid="tx0.1v2">
   <map name="ATM2OCN_FMAPNAME">cpl/cpl6/map_fv0.47x0.63_to_tx0.1v2_aave_da_090218.nc</map>
   <map name="ATM2OCN_SMAPNAME">cpl/cpl6/map_fv0.47x0.63_to_tx0.1v2_bilin_da_090218.nc</map>
   <map name="ATM2OCN_VMAPNAME">cpl/cpl6/map_fv0.47x0.63_to_tx0.1v2_bilin_da_090218.nc</map>
   <map name="OCN2ATM_FMAPNAME">cpl/cpl6/map_tx0.1v2_to_fv0.47x0.63_aave_da_090218.nc</map>
   <map name="OCN2ATM_SMAPNAME">cpl/cpl6/map_tx0.1v2_to_fv0.47x0.63_aave_da_090218.nc</map>
</gridmap>

It appears that we might not want patch mapping because the only ocean grids that have patch mapping in config_grids.xml are these two gx1v6 and gx3v7 grid (out of scores of other grid configurations).

<!-- Line 1578 -->
<gridmap atm_grid="0.47x0.63" ocn_grid="gx1v6">
<!-- Line 1833 -->
<gridmap atm_grid="T31" ocn_grid="gx3v7">

Interesting note: no domain file

There is no domain entry defined for the f05_t12 grid:

<!-- Line 1119 -->
<domain name="0.47x0.63">
   <nx>576</nx>  <ny>384</ny>
   <file grid="atm|lnd" mask="gx1v6">domain.lnd.fv0.47x0.63_gx1v6.090407.nc</file>
   <file grid="ocnice"  mask="gx1v6">domain.ocn.0.47x0.63_gx1v6_090408.nc</file>
   <file grid="atm|lnd" mask="gx1v7">domain.lnd.fv0.47x0.63_gx1v7.180521.nc</file>
   <file grid="ocnice"  mask="gx1v7">domain.ocn.fv0.47x0.63_gx1v7.180521.nc</file>
   <desc>0.47x0.63 is FV 1/2-deg grid:</desc>
</domain>

Generate mapping files

Mapping files should be generated for:

atm <-> ocn
atm <-> wav
lnd <-> rof
lnd <-> glc
ocn <-> wav
rof -> ocn

by calling gen_cesm_maps.sh in $CIMEROOT/tools/mapping/gen_mapping_files/.

Build the executable

The initial step is to build the ESMF_RegridWeightGenCheck executable.

module load esmf_libs/8.0.0
module load esmf-8.0.0-ncdfio-uni-O
gmake
ls -lart ../
[ ... ]
drwxr-xr-x 3 johnsonb ncar    4096 Jan  4 06:58 .
-rwxr-xr-x 1 johnsonb ncar 1624728 Jan  4 06:58 ESMF_RegridWeightGenCheck

Run the script

The next step is to run the gen_cesm_maps.sh script with the appropriate options:

cd /glade/work/johnsonb/cesm2_1_3/cime/tools/mapping/gen_mapping_files
qsub -I -l select=1:ncpus=36:mpiprocs=36 -l walltime=01:00:00 -q regular -A P86850054
./gen_cesm_maps.sh \
--fileocn /glade/p/cesm/cseg/inputdata/share/scripgrids/tx0.1v3_211102.nc \
--nameocn tx0.1v3 \
--fileatm /glade/p/cesm/cseg/inputdata/share/scripgrids/0.9x1.25_SCRIP_desc.181018.nc \
--nameatm fv0.9x1.25
[ ...  ]
Tue Jan  4 13:34:36 MST 2022
1: map_tx0.1v3_TO_fv0.9x1.25_aave.220104.nc
All           21  tests passed!
-----
2: map_tx0.1v3_TO_fv0.9x1.25_blin.220104.nc
All           14  tests passed!
-----
3: map_fv0.9x1.25_TO_tx0.1v3_aave.220104.nc
All           21  tests passed!
-----
4: map_fv0.9x1.25_TO_tx0.1v3_blin.220104.nc
All           14  tests passed!
-----
5: map_fv0.9x1.25_TO_tx0.1v3_patc.220104.nc
All           14  tests passed!
-----

So it appears the mapping files were created properly.

Resultant files

The script creates two sets of files in /glade/work/johnsonb/cesm2_1_3/cime/tools/mapping/gen_mapping_files

Mapping ocean to atmosphere

Conservative, area-averaged: map_tx0.1v3_TO_fv0.9x1.25_aave.220104.nc
Non-conservative, bilinear: map_tx0.1v3_TO_fv0.9x1.25_blin.220104.nc

Mapping atmosphere to ocean

Conservative, area-averaged: map_fv0.9x1.25_TO_tx0.1v3_aave.220104.nc
Non-conservative, bilinear: map_fv0.9x1.25_TO_tx0.1v3_blin.220104.nc
Non-conservative, patch: map_fv0.9x1.25_TO_tx0.1v3_patc.220104.nc

Caution

Before the actual experiment begins, check to see if a new rtm->ocn mapping file must be created. The Adding grids section of the CIME documentation says that the, “process for mapping from the runoff grid to the ocean grid is currently undergoing many changes…please contact Michael Levy for assistance.”

Generate domain files

Build the executable

cd /glade/work/johnsonb/cesm2_1_3/cime/tools/mapping/gen_domain_files
more INSTALL
# Read the installation instructions
cd src
../../../configure --macros-format Makefile --mpilib mpi-serial
(. ./.env_mach_specific.sh ; gmake)
ls -lart ../
[ ...  ]
drwxr-xr-x 3 johnsonb ncar    4096 Jan  4 14:52 .
-rwxr-xr-x 1 johnsonb ncar 6086848 Jan  4 14:52 gen_domain

Pass the executable the correct files

cd /glade/work/johnsonb/cesm2_1_3/cime/tools/mapping/gen_domain_files
./gen_domain -m ../gen_mapping_files/map_tx0.1v3_TO_fv0.9x1.25_aave.220104.nc -o tx0.1v3 -l fv0.9x1.25
[ ... ]
write domain.lnd.fv0.9x1.25_tx0.1v3.220104.nc
successfully created domain file domain.lnd.fv0.9x1.25_tx0.1v3.220104.nc
write domain.ocn.fv0.9x1.25_tx0.1v3.220104.nc
successfully created domain file domain.ocn.fv0.9x1.25_tx0.1v3.220104.nc

Edit config_grids.xml

Next edit config_grids.xml to add a model_grid node, a gridmap node and possibly a domain entry for every new component grid.

Define the model grid

<!-- New f09_t13 grid for DART+POP2 ASD Project -->
<model_grid alias="f09_t13">
   <grid name="atm">0.9x1.25</grid>
   <grid name="lnd">0.9x1.25</grid>
   <grid name="ocnice">tx0.1v3</grid>
   <mask>tx0.1v3</mask>
</model_grid>

Add the domain files

<!-- Line 1128 -->
<domain name="0.9x1.25">
   <nx>288</nx>  <ny>192</ny>
   <file grid="atm|lnd" mask="gx1v6">domain.lnd.fv0.9x1.25_gx1v6.090309.nc</file>
   <file grid="ocnice"  mask="gx1v6">domain.ocn.0.9x1.25_gx1v6_090403.nc</file>
   <file grid="atm|lnd" mask="gx1v7">domain.lnd.fv0.9x1.25_gx1v7.151020.nc</file>
   <file grid="ocnice"  mask="gx1v7">domain.ocn.fv0.9x1.25_gx1v7.151020.nc</file>
   <file grid="atm|lnd" mask="tn1v3">domain.lnd.fv0.9x1.25_tn1v3.160414.nc</file>
   <file grid="ocnice"  mask="tn1v3">domain.ocn.fv0.9x1.25_tn1v3.160414.nc</file>
   <file grid="atm|lnd" mask="tn0.25v3">domain.lnd.fv0.9x1.25_tn0.25v3.160721.nc</file>
   <file grid="ocnice"  mask="tn0.25v3">domain.ocn.fv0.9x1.25_tn0.25v3.160721.nc</file>
   <file grid="atm|lnd" mask="null">/glade/u/home/benedict/ys/datain/domain.aqua.fv0.9x1.25.nc</file>
   <file grid="ocnice"  mask="null">/glade/u/home/benedict/ys/datain/domain.aqua.fv0.9x1.25.nc</file>
   <!-- New f09_t13 domains for DART+POP2 ASD Project -->
   <file grid="atm|lnd" mask="tx0.1v3">/glade/work/johnsonb/cesm2_1_3/cime/tools/mapping/gen_domain_files/domain.lnd.fv0.9x1.25_tx0.1v3.220104.nc</file>
   <file grid="ocnice"  mask="tx0.1v3">/glade/work/johnsonb/cesm2_1_3/cime/tools/mapping/gen_domain_files/domain.ocn.fv0.9x1.25_tx0.1v3.220104.nc</file>
   <desc>0.9x1.25 is FV 1-deg grid:</desc>
</domain>

Declare the grid maps

<!-- New f09_t13 gridmap for DART+POP2 ASD Project -->
<gridmap atm_grid="0.9x1.25" ocn_grid="tx0.1v3">
   <map name="ATM2OCN_FMAPNAME">/glade/work/johnsonb/cesm2_1_3/cime/tools/mapping/gen_mapping_files/map_fv0.9x1.25_TO_tx0.1v3_aave.220104.nc</map>
   <map name="ATM2OCN_SMAPNAME">/glade/work/johnsonb/cesm2_1_3/cime/tools/mapping/gen_mapping_files/map_fv0.9x1.25_TO_tx0.1v3_blin.220104.nc</map>
   <map name="ATM2OCN_VMAPNAME">/glade/work/johnsonb/cesm2_1_3/cime/tools/mapping/gen_mapping_files/map_fv0.9x1.25_TO_tx0.1v3_blin.220104.nc</map>
   <map name="OCN2ATM_FMAPNAME">/glade/work/johnsonb/cesm2_1_3/cime/tools/mapping/gen_mapping_files/map_tx0.1v3_TO_fv0.9x1.25_aave.220104.nc</map>
   <map name="OCN2ATM_SMAPNAME">/glade/work/johnsonb/cesm2_1_3/cime/tools/mapping/gen_mapping_files/map_tx0.1v3_TO_fv0.9x1.25_aave.220104.nc</map>
</gridmap>

Check the XML file

Check the formatting of the XML file to ensure it is formatted properly:

cd /glade/work/johnsonb/cesm2_1_3/cime/config/cesm
xmllint --noout --schema /glade/work/johnsonb/cesm2_1_3/cime/config/xml_schemas/config_grids_v2.xsd ./config_grids.xml
./config_grids.xml validates

Redo the process for f09_t12

Scrip grid files

f09

/glade/p/cesm/cseg/inputdata/share/scripgrids/0.9x1.25_SCRIP_desc.181018.nc

t12

/glade/p/cesm/cseg/inputdata/share/scripgrids/tx0.1v2_211102.nc

cd /glade/work/johnsonb/cesm2_1_3/cime/tools/mapping/gen_mapping_files
qsub -I -l select=1:ncpus=36:mpiprocs=36 -l walltime=01:00:00 -q regular -A P86850054
./gen_cesm_maps.sh \
--fileocn /glade/p/cesm/cseg/inputdata/share/scripgrids/tx0.1v2_211102.nc \
--nameocn tx0.1v2 \
--fileatm /glade/p/cesm/cseg/inputdata/share/scripgrids/0.9x1.25_SCRIP_desc.181018.nc \
--nameatm fv0.9x1.25
[ ...  ]
Wed Jan 12 15:33:52 MST 2022
1: map_tx0.1v2_TO_fv0.9x1.25_aave.220112.nc
All           21  tests passed!
-----
2: map_tx0.1v2_TO_fv0.9x1.25_blin.220112.nc
All           14  tests passed!
-----
3: map_fv0.9x1.25_TO_tx0.1v2_aave.220112.nc
All           21  tests passed!
-----
4: map_fv0.9x1.25_TO_tx0.1v2_blin.220112.nc
All           14  tests passed!
-----
5: map_fv0.9x1.25_TO_tx0.1v2_patc.220112.nc
All           14  tests passed!
-----

Domain files

cd /glade/work/johnsonb/cesm2_1_3/cime/tools/mapping/gen_domain_files
./gen_domain -m ../gen_mapping_files/map_tx0.1v2_TO_fv0.9x1.25_aave.220112.nc -o tx0.1v2 -l fv0.9x1.25

config_grids.xml

<!-- New f09_t12 grid for DART+POP2 ASD Project -->

<model_grid alias="f09_t12">
   <grid name="atm">0.9x1.25</grid>
   <grid name="lnd">0.9x1.25</grid>
   <grid name="ocnice">tx0.1v2</grid>
   <mask>tx0.1v2</mask>
</model_grid>

<!-- New f09_t12 domains for DART+POP2 ASD Project -->

<file grid="atm|lnd" mask="tx0.1v2">/glade/work/johnsonb/cesm2_1_3/cime/tools/mapping/gen_domain_files/domain.lnd.fv0.9x1.25_tx0.1v2.220112.nc</file>
<file grid="ocnice"  mask="tx0.1v2">/glade/work/johnsonb/cesm2_1_3/cime/tools/mapping/gen_domain_files/domain.ocn.fv0.9x1.25_tx0.1v2.220112.nc</file>

<!-- New f09_t12 gridmap for DART+POP2 ASD Project -->

<gridmap atm_grid="0.9x1.25" ocn_grid="tx0.1v2">
   <map name="ATM2OCN_FMAPNAME">/glade/work/johnsonb/cesm2_1_3/cime/tools/mapping/gen_mapping_files/map_fv0.9x1.25_TO_tx0.1v2_aave.220112.nc</map>
   <map name="ATM2OCN_SMAPNAME">/glade/work/johnsonb/cesm2_1_3/cime/tools/mapping/gen_mapping_files/map_fv0.9x1.25_TO_tx0.1v2_blin.220112.nc</map>
   <map name="ATM2OCN_VMAPNAME">/glade/work/johnsonb/cesm2_1_3/cime/tools/mapping/gen_mapping_files/map_fv0.9x1.25_TO_tx0.1v2_blin.220112.nc</map>
   <map name="OCN2ATM_FMAPNAME">/glade/work/johnsonb/cesm2_1_3/cime/tools/mapping/gen_mapping_files/map_tx0.1v2_TO_fv0.9x1.25_aave.220112.nc</map>
   <map name="OCN2ATM_SMAPNAME">/glade/work/johnsonb/cesm2_1_3/cime/tools/mapping/gen_mapping_files/map_tx0.1v2_TO_fv0.9x1.25_aave.220112.nc</map>
</gridmap>

Redo the process for f09_t13

There are two SCRIP grid files for t13:

/glade/p/cesm/cseg/inputdata/share/scripgrids/tx0.1v3_211102.nc
/glade/p/cesm/cseg/inputdata/share/scripgrids/tx0.1v3_170728.nc

The tx0.1v3_211102.nc file was used in the initial attempt. The runs resulting exhibited perplexing behavior suggesting that there was something wrong with the file. This attempt rebuilds the mapping and domain files with the tx0.1v3_170728.nc file.

Build ESMF_RegridWeightGenCheck

cd /glade/work/johnsonb/cesm2_1_1/cime/tools/mapping/check_maps/src
module load esmf_libs/8.0.0
module load esmf-8.0.0-ncdfio-uni-O
gmake
ls -lart ../
drwxr-xr-x 3 johnsonb ncar    4096 Jan 13 16:11 .
-rwxr-xr-x 1 johnsonb ncar 1624728 Jan 13 16:11 ESMF_RegridWeightGenCheck

Mapping files

cd /glade/work/johnsonb/cesm2_1_1/cime/tools/mapping/gen_mapping_files/
qsub -I -l select=1:ncpus=36:mpiprocs=36 -l walltime=01:00:00 -q regular -A P86850054
./gen_cesm_maps.sh \
--fileocn /glade/p/cesm/cseg/inputdata/share/scripgrids/tx0.1v3_170728.nc \
--nameocn tx0.1v3 \
--fileatm /glade/p/cesm/cseg/inputdata/share/scripgrids/0.9x1.25_SCRIP_desc.181018.nc \
--nameatm fv0.9x1.25
[ ... ]
Thu Jan 13 16:38:27 MST 2022
1: map_tx0.1v3_TO_fv0.9x1.25_aave.220113.nc
All           21  tests passed!
-----
2: map_tx0.1v3_TO_fv0.9x1.25_blin.220113.nc
All           14  tests passed!
-----
3: map_fv0.9x1.25_TO_tx0.1v3_aave.220113.nc
All           21  tests passed!
-----
4: map_fv0.9x1.25_TO_tx0.1v3_blin.220113.nc
All           14  tests passed!
-----
5: map_fv0.9x1.25_TO_tx0.1v3_patc.220113.nc
All           14  tests passed!

Domain files

cd /glade/work/johnsonb/cesm2_1_1/cime/tools/mapping/gen_domain_files
cd src
../../../configure --macros-format Makefile --mpilib mpi-serial
(. ./.env_mach_specific.sh ; gmake)
ls -lart ../
cd /glade/work/johnsonb/cesm2_1_1/cime/tools/mapping/gen_domain_files
./gen_domain -m ../gen_mapping_files/map_tx0.1v3_TO_fv0.9x1.25_aave.220113.nc -o tx0.1v3 -l fv0.9x1.25
[ ... ]
successfully created domain file domain.lnd.fv0.9x1.25_tx0.1v3.220113.nc
write domain.ocn.fv0.9x1.25_tx0.1v3.220113.nc
successfully created domain file domain.ocn.fv0.9x1.25_tx0.1v3.220113.nc

config_grids.xml

<!-- New f09_t13 domains for DART+POP2 ASD Project -->
<domain name="0.9x1.25">
   <nx>288</nx>  <ny>192</ny>
   [ ... ]
   <file grid="atm|lnd" mask="tx0.1v3">/glade/work/johnsonb/cesm2_1_1/cime/tools/mapping/gen_domain_files/domain.lnd.fv0.9x1.25_tx0.1v3.220113.nc</file>
   <file grid="ocnice"  mask="tx0.1v3">/glade/work/johnsonb/cesm2_1_1/cime/tools/mapping/gen_domain_files/domain.ocn.fv0.9x1.25_tx0.1v3.220113.nc</file>
   [ ... ]
   <desc>0.9x1.25 is FV 1-deg grid:</desc>
 </domain>

 <!-- New f09_t13 gridmap for DART+POP2 ASD Project -->

 <gridmap atm_grid="0.9x1.25" ocn_grid="tx0.1v3">
    <map name="ATM2OCN_FMAPNAME">/glade/work/johnsonb/cesm2_1_1/cime/tools/mapping/gen_mapping_files/map_fv0.9x1.25_TO_tx0.1v3_aave.220113.nc</map>
    <map name="ATM2OCN_SMAPNAME">/glade/work/johnsonb/cesm2_1_1/cime/tools/mapping/gen_mapping_files/map_fv0.9x1.25_TO_tx0.1v3_blin.220113.nc</map>
    <map name="ATM2OCN_VMAPNAME">/glade/work/johnsonb/cesm2_1_1/cime/tools/mapping/gen_mapping_files/map_fv0.9x1.25_TO_tx0.1v3_blin.220113.nc</map>
    <map name="OCN2ATM_FMAPNAME">/glade/work/johnsonb/cesm2_1_1/cime/tools/mapping/gen_mapping_files/map_tx0.1v3_TO_fv0.9x1.25_aave.220113.nc</map>
    <map name="OCN2ATM_SMAPNAME">/glade/work/johnsonb/cesm2_1_1/cime/tools/mapping/gen_mapping_files/map_tx0.1v3_TO_fv0.9x1.25_aave.220113.nc</map>
 </gridmap>

Note

As described in List of attempts, this attempt worked.

Third attempt for f09_t13

The second attempt for f09_t13 works but after inspecting the header information for the f09_g16 and f09_g17 mapping and domain files, it seems that it would be prudent to use the exact same f09 Scrip grid file for f09_t13 as was used for the f09_g17 files.

Map files

Save the previous attempt

cd /glade/work/johnsonb/cesm2_1_3/cime/tools/mapping/gen_mapping_files
mkdir previous_attempt
mv *tx0.1v3* previous_attempt

Make a second attempt

 cd /glade/work/johnsonb/cesm2_1_3/cime/tools/mapping/gen_mapping_files/
 qsub -I -l select=1:ncpus=36:mpiprocs=36 -l walltime=01:00:00 -q regular -A P86850054
 ./gen_cesm_maps.sh \
 --fileocn /glade/p/cesm/cseg/inputdata/share/scripgrids/tx0.1v3_170728.nc \
 --nameocn tx0.1v3 \
 --fileatm /glade/p/cesm/cseg/inputdata/share/scripgrids/fv0.9x1.25_141008.nc \
 --nameatm fv0.9x1.25
 [ ... ]
 Fri Jan 14 17:34:01 MST 2022
 1: map_tx0.1v3_TO_fv0.9x1.25_aave.220114.nc
 All           21  tests passed!
 -----
 2: map_tx0.1v3_TO_fv0.9x1.25_blin.220114.nc
 All           14  tests passed!
 -----
 3: map_fv0.9x1.25_TO_tx0.1v3_aave.220114.nc
 All           21  tests passed!
 -----
4: map_fv0.9x1.25_TO_tx0.1v3_blin.220114.nc
All           14  tests passed!
-----
5: map_fv0.9x1.25_TO_tx0.1v3_patc.220114.nc
All           14  tests passed!
-----

Domain files

cd /glade/work/johnsonb/cesm2_1_3/cime/tools/mapping/gen_domain_files
./gen_domain -m ../gen_mapping_files/map_tx0.1v3_TO_fv0.9x1.25_aave.220114.nc -o tx0.1v3 -l fv0.9x1.25
[ ... ]
ls -lart
-rw-r--r-- 1 johnsonb ncar   5531800 Jan 14 17:34 domain.lnd.fv0.9x1.25_tx0.1v3.220114.nc
-rw-r--r-- 1 johnsonb ncar   5531800 Jan 14 17:34 domain.ocn.fv0.9x1.25_tx0.1v3.220114.nc

config_grid.xml

<!-- New f09_t13 domains for DART+POP2 ASD Project -->
<domain name="0.9x1.25">
   <nx>288</nx>  <ny>192</ny>
   [ ... ]
   <file grid="atm|lnd" mask="tx0.1v3">/glade/work/johnsonb/cesm2_1_3/cime/tools/mapping/gen_domain_files/domain.lnd.fv0.9x1.25_tx0.1v3.220114.nc</file>
   <file grid="ocnice"  mask="tx0.1v3">/glade/work/johnsonb/cesm2_1_3/cime/tools/mapping/gen_domain_files/domain.ocn.fv0.9x1.25_tx0.1v3.220114.nc</file>
   [ ... ]
   <desc>0.9x1.25 is FV 1-deg grid:</desc>
 </domain>

 <!-- New f09_t13 gridmap for DART+POP2 ASD Project -->

 <gridmap atm_grid="0.9x1.25" ocn_grid="tx0.1v3">
    <map name="ATM2OCN_FMAPNAME">/glade/work/johnsonb/cesm2_1_3/cime/tools/mapping/gen_mapping_files/map_fv0.9x1.25_TO_tx0.1v3_aave.220114.nc</map>
    <map name="ATM2OCN_SMAPNAME">/glade/work/johnsonb/cesm2_1_3/cime/tools/mapping/gen_mapping_files/map_fv0.9x1.25_TO_tx0.1v3_blin.220114.nc</map>
    <map name="ATM2OCN_VMAPNAME">/glade/work/johnsonb/cesm2_1_3/cime/tools/mapping/gen_mapping_files/map_fv0.9x1.25_TO_tx0.1v3_blin.220114.nc</map>
    <map name="OCN2ATM_FMAPNAME">/glade/work/johnsonb/cesm2_1_3/cime/tools/mapping/gen_mapping_files/map_tx0.1v3_TO_fv0.9x1.25_aave.220114.nc</map>
    <map name="OCN2ATM_SMAPNAME">/glade/work/johnsonb/cesm2_1_3/cime/tools/mapping/gen_mapping_files/map_tx0.1v3_TO_fv0.9x1.25_aave.220114.nc</map>
 </gridmap>

cd /glade/work/johnsonb/cesm2_1_1/cime/config/cesm
xmllint --noout --schema /glade/work/johnsonb/cesm2_1_3/cime/config/xml_schemas/config_grids_v2.xsd ./config_grids.xml
./config_grids.xml validates

Compsets

When defining non-default compsets, it is advisable to start with a default compset and adapt it.

The CESM Component Set Definitions page lists all of the defined compsets.

In this case, there are two pairs of compsets whose confugration can be studied as a starting point: the COREv2 forcing of Large and Yeager (2009), which corresponds to the GIAF and GIAF_HR compsets in CESM, and the JRA-55 forcing of Kobayashi et al. (2015), which corresponds to the GIAF_JRA and GIAF_JRA_HR compsets.

In all compsets, the land ice component is stub glacier.
In eddy-parameterizing compsets, the wave component is Wave Watch 3.
In eddy-resolving compsets, the wave component is stub wave.

GIAF

Long name: 2000_DATM%IAF_SLND_CICE_POP2_DROF%IAF_SGLC_WW3

	Value	Description
Initialization Time	2000	1850: Pre-Industrial; 2000 present day: Additional initialization times defined by components.
Atmosphere	DATM%IAF	Data driven ATM COREv2 interannual forcing
Land	SLND	Stub land component
Sea-Ice	CICE	Sea ICE (cice) model version 5
Ocean	POP2	POP2
River runoff	DROF%IAF	Data runoff modelCOREv2 interannual year forcing:
Land Ice	SGLC	Stub glacier (land ice) component
Wave	WW3	Wave Watch

GIAF_HR

Long name: 2000_DATM%IAF_SLND_CICE_POP2_DROF%IAF_SGLC_SWAV

	Value	Description
Initialization Time	2000	1850: Pre-Industrial; 2000 present day: Additional initialization times defined by components.
Atmosphere	DATM%IAF	Data driven ATM COREv2 interannual forcing
Land	SLND	Stub land component
Sea-Ice	CICE	Sea ICE (cice) model version 5
Ocean	POP2	POP2
River runoff	DROF%IAF	Data runoff modelCOREv2 interannual year forcing:
Land Ice	SGLC	Stub glacier (land ice) component
Wave	SWAV	Stub wave component

GIAF_JRA

Long name: 2000_DATM%JRA-1p4-2018_SLND_CICE_POP2_DROF%JRA-1p4-2018_SGLC_WW3

	Value	Description
Initialization Time	2000	1850: Pre-Industrial; 2000 present day: Additional initialization times defined by components.
Atmosphere	DATM%JRA-1p4-2018	Data driven ATM interannual JRA55 forcing, v1.4, through 2018
Land	SLND	Stub land component
Sea-Ice	CICE	Sea ICE (cice) model version 5
Ocean	POP2	POP2
River runoff	DROF%JRA-1p4-2018	Data runoff modelJRA55 interannual forcing, v1.4, through 2018
Land Ice	SGLC	Stub glacier (land ice) component
Wave	WW3	Wave Watch

GIAF_JRA_HR

Long name: 2000_DATM%JRA-1p4-2018_SLND_CICE%CICE4_POP2_DROF%JRA-1p4-2018_SGLC_SWAV

	Value	Description
Initialization Time	2000	1850: Pre-Industrial; 2000 present day: Additional initialization times defined by components.
Atmosphere	DATM%JRA-1p4-2018	Data driven ATM interannual JRA55 forcing, v1.4, through 2018
Land	SLND	Stub land component
Sea-Ice	CICE%CICE4	Sea ICE (cice) model version 5 :running with cice4 physics
Ocean	POP2	POP2
River runoff	DROF%JRA-1p4-2018	Data runoff modelJRA55 interannual forcing, v1.4, through 2018
Land Ice	SGLC	Stub glacier (land ice) component
Wave	SWAV	Stub wave component

Generating the initial ensemble

For the benchmarking tests, the initial ensemble does not need the spread necessary to prevent filter divergence.

Finding an initial set of restart files

Since HPSS is offline, it’s unclear where the restart files are from Johnson et al. (2016). But (luckily) it’s easier to find restart files on campaign storage than it was on HPSS.

sftp data-access.ucar.edu
cd /gpfs/csfs1/cgd/oce
find . -name "*t13*"

Copy files into an ensemble

Run a script to copy eighty sets of restart files into a restart directory on scratch.

cd /glade/u/home/johnsonb/pyscripts
python copy_restart_files.py
ls -lart /glade/scratch/johnsonb/g.e20.G.TL319_t13.control.001/rest/0001-01-01-00000
[ ... ]
-rw-r--r-- 1 johnsonb ncar 11335683072 Jan  6 17:23 g.e20.G.TL319_t13.control.001.cice_0079.r.0001-01-01-00000.nc
-rw-r--r-- 1 johnsonb ncar  8626923008 Jan  6 17:23 g.e20.G.TL319_t13.control.001.cpl_0079.r.0001-01-01-00000.nc
-rw-r--r-- 1 johnsonb ncar 35251207680 Jan  6 17:23 g.e20.G.TL319_t13.control.001.pop_0079.r.0001-01-01-00000.nc
-rw-r--r-- 1 johnsonb ncar 11335683072 Jan  6 17:23 g.e20.G.TL319_t13.control.001.cice_0080.r.0001-01-01-00000.nc
-rw-r--r-- 1 johnsonb ncar  8626923008 Jan  6 17:23 g.e20.G.TL319_t13.control.001.cpl_0080.r.0001-01-01-00000.nc
-rw-r--r-- 1 johnsonb ncar 35251207680 Jan  6 17:24 g.e20.G.TL319_t13.control.001.pop_0080.r.0001-01-01-00000.nc

t12 restart files

These are actually from Frank’s Yellowstone ASD experiment:

cp g.e12.G.T62_t12.003.pop.r.0006-02-01-00000.nc /glade/scratch/johnsonb/g.e12.G.T62_t12.003/rest/0006-02-01-00000/g.e12.G.T62_t12.003.pop_0001.r.0006-02-01-00000.nc
cp g.e12.G.T62_t12.003.cice.r.0006-02-01-00000.nc /glade/scratch/johnsonb/g.e12.G.T62_t12.003/rest/0006-02-01-00000/g.e12.G.T62_t12.003.cice_0001.r.0006-02-01-00000.nc
cp g.e12.G.T62_t12.003.cpl.r.0006-02-01-00000.nc /glade/scratch/johnsonb/g.e12.G.T62_t12.003/rest/0006-02-01-00000/g.e12.G.T62_t12.003.cpl_0001.r.0006-02-01-00000.nc

List of attempts

CESM 2.1.0

CESM 2.1.1

GIAF_HR.cesm2_1_1.T62_t13.001

Build

./create_newcase --case /glade/work/johnsonb/cesm_runs/GIAF_HR.cesm2_1_1.T62_t13.001 --compset GIAF_HR --res T62_t13 --mach cheyenne --run-unsupported --project P86850054
Compset longname is 2000_DATM%IAF_SLND_CICE_POP2_DROF%IAF_SGLC_SWAV
cd /glade/work/johnsonb/cesm_runs/GIAF_HR.cesm2_1_1.T62_t13.001
./xmlquery STOP_N
        STOP_N: 3

Result

Execution terminated Exit_status=0 resources_used.cpupercent=81 resources_used.cput=00:00:14 resources_used.mem=34468kb resources_used.ncpus=1 resources_used.vmem=64996kb resources_used.walltime=00:00:15

Note

Perplexingly, this runs to “completion” but a POP2 restart file is not produced.

GIAF_HR.cesm2_1_1.T62_t12.001

Build

./create_newcase --case /glade/work/johnsonb/cesm_runs/GIAF_HR.cesm2_1_1.T62_t12.001 --compset GIAF_HR --res T62_t12 --mach cheyenne --run-unsupported --project P86850054
Compset longname is 2000_DATM%IAF_SLND_CICE_POP2_DROF%IAF_SGLC_SWAV
cd /glade/work/johnsonb/cesm_runs/GIAF_HR.cesm2_1_1.T62_t12.001
./xmlquery STOP_N
        STOP_N: 3

Result

Execution terminated Exit_status=1 resources_used.cpupercent=98002 resources_used.cput=91:35:53 resources_used.mem=679832624kb resources_used.ncpus=1044 resources_used.vmem=5449231040kb resources_used.walltime=00:06:06

Error

Warning: Departure points out of bounds in remap [ … ] remap transport: bad departure points ERROR: remap transport: bad departure points

Chris Riedel linked to the relevant page in the CICE documentation for this problem. Editing user_nl_cice fixes the problem.

cd $CASEROOT
vim user_nl_cice_00*
! Append this line to the end of the file
ndtd = 5

G.benchmarking.cesm2_1_1.f09_t12.001

Build

./create_newcase --case /glade/work/johnsonb/cesm_runs/G.benchmarking.cesm2_1_1.f09_t12.001 --compset GIAF_HR --res f09_t12 --mach cheyenne --run-unsupported --project P86850054
cd /glade/work/johnsonb/cesm_runs/G.benchmarking.cesm2_1_1.f09_t12.001
./case.setup
qcmd -q share -l select=1 -A P86850054 -- ./case.build
[ ... ]
MODEL BUILD HAS FINISHED SUCCESSFULLY
./xmlchange STOP_N=3
./case.submit -M begin,end

Result

Execution terminated Exit_status=1 resources_used.cpupercent=95809 resources_used.cput=84:07:17 resources_used.mem=676683064kb resources_used.ncpus=1044 resources_used.vmem=5449458968kb resources_used.walltime=00:06:18

Error

Warning: Departure points out of bounds in remap [ … ] remap transport: bad departure points ERROR: remap transport: bad departure points

Note

It’s promising that the T62_t12 case and the f09_t12 both encounter the CFL-like error.

GIAF_HR.cesm2_1_1.f09_t13.002

Important

Since the G.benchmarking.cesm2_1_1.f09_t12.001 run hit the CFL-like error, it suggests that the problem with the f09_t13 setup has to do with the mapping and domain files. This GIAF_HR.cesm2_1_1.f09_t13.002 case rebuilds the mapping and domain files with a different SCRIP file: /glade/p/cesm/cseg/inputdata/share/scripgrids/tx0.1v3_170728.nc

Build

./create_newcase --case /glade/work/johnsonb/cesm_runs/GIAF_HR.cesm2_1_1.f09_t13.002 --compset GIAF_HR --res f09_t13 --mach cheyenne --run-unsupported --project P86850054
cd /glade/work/johnsonb/cesm_runs/GIAF_HR.cesm2_1_1.f09_t13.002
./case.setup
qcmd -q share -l select=1 -A P86850054 -- ./case.build
[ ... ]
MODEL BUILD HAS FINISHED SUCCESSFULLY
./xmlchange STOP_N=3
./case.submit -M begin,end

Result

Execution terminated Exit_status=0 resources_used.cpupercent=295569 resources_used.cput=364:38:03 resources_used.mem=1158844040kb resources_used.ncpus=3060 resources_used.vmem=7207103764kb resources_used.walltime=00:08:27

Note

This run ran to completion and output a POP restart file. This suggests that the SCRIP file was the issue confounding earlier t13 runs.

Working attempts for t13

Five members

G.benchmarking.startup.cesm2_1_1.f09_t13_e5.003

Ten members

G.benchmarking.startup.cesm2_1_1.f09_t13_e10.004

Fifteen members

G.benchmarking.startup.cesm2_1_1.f09_t13_e15

CESM 2.1.3

Modified buildnml script

During the DART+CAM6 Reanalysis, Kevin Raeder and Jeff Steward identified and fixed a bottleneck in one of the MCT build scripts:

/glade/work/raeder/Models/cesm2_1_relsd_m5.6/cime/src/drivers/mct/cime_config/buildnml

The script runs slowly for a multi-instance CESM case because it attempts to write multiple files simultaneously into directories that are locked.

This bottleneck is inevitable during the first submission of case.run, but it can be avoided on subsequent submissions.

cd /glade/work/johnsonb/cesm2_1_3/cime/src/drivers/mct/cime_config
git mv buildnml buildnml.old
cp /glade/work/raeder/Models/cesm2_1_relsd_m5.6/cime/src/drivers/mct/cime_config/buildnml ./

Building the case

f09_g17

Single instance

cd /glade/work/johnsonb/cesm2_1_3/cime/scripts
./create_newcase --case /glade/work/johnsonb/cesm_runs/G.CAM6-Reanalysis.f09_g17.001 --compset GIAF --res f09_g17 --mach cheyenne --run-unsupported --project P86850054
cd /glade/work/johnsonb/cesm_runs/G.CAM6-Reanalysis.f09_g17.001
./case.setup
./xmlquery --partial PE
[ ... ]
Results in group mach_pes
    NTASKS_PER_INST: ['ATM:72', 'LND:72', 'ICE:72', 'OCN:144', 'ROF:72', 'GLC:72', 'WAV:72', 'ESP:1']
    ROOTPE: ['CPL:0', 'ATM:0', 'LND:0', 'ICE:0', 'OCN:72', 'ROF:0', 'GLC:0', 'WAV:0', 'ESP:0']

Results in group mach_pes_last
    COSTPES_PER_NODE: 36
    COST_PES: 216
    MAX_MPITASKS_PER_NODE: 36
    MAX_TASKS_PER_NODE: 36
    TOTALPES: 216
[ ... ]

Eighty members

cd /glade/work/johnsonb/cesm2_1_3/cime/scripts
./create_newcase --case /glade/work/johnsonb/cesm_runs/G.CAM6-Reanalysis.f09_g17.001.e80 --compset GIAF --res f09_g17 --mach cheyenne --run-unsupported --project P86850054 --ninst 80 --multi-driver
cd /glade/work/johnsonb/cesm_runs/G.CAM6-Reanalysis.f09_g17.001.e80
./case.setup
./xmlquery --partial PE
[ ... ]
Results in group mach_pes
        NTASKS_PER_INST: ['ATM:72', 'LND:72', 'ICE:72', 'OCN:144', 'ROF:72', 'GLC:72', 'WAV:72', 'ESP:1']
        ROOTPE: ['CPL:0', 'ATM:0', 'LND:0', 'ICE:0', 'OCN:72', 'ROF:0', 'GLC:0', 'WAV:0', 'ESP:0']

Results in group mach_pes_last
        COSTPES_PER_NODE: 36
        COST_PES: 17280
        MAX_MPITASKS_PER_NODE: 36
        MAX_TASKS_PER_NODE: 36
        TOTALPES: 17280
[ ... ]

f09_t13

Use the GIAF_HR compset as a starting point for f09_t13.

Single instance

cd /glade/work/johnsonb/cesm2_1_3/cime/scripts
./create_newcase --case /glade/work/johnsonb/cesm_runs/G.CAM6-Reanalysis.f09_t13.001 --compset GIAF_HR --res f09_t13 --mach cheyenne --run-unsupported --project P86850054
cd /glade/work/johnsonb/cesm_runs/G.CAM6-Reanalysis.f09_t13.001
./case.setup
./xmlquery --partial PE
[ ... ]
Results in group mach_pes
    NTASKS_PER_INST: ['ATM:72', 'LND:72', 'ICE:828', 'OCN:2208', 'ROF:72', 'GLC:72', 'WAV:72', 'ESP:1']
    ROOTPE: ['CPL:0', 'ATM:0', 'LND:0', 'ICE:0', 'OCN:828', 'ROF:0', 'GLC:0', 'WAV:0', 'ESP:0']

Results in group mach_pes_last
    COSTPES_PER_NODE: 36
    COST_PES: 3060
    MAX_MPITASKS_PER_NODE: 36
    MAX_TASKS_PER_NODE: 36
    TOTALPES: 3036
[ ... ]

Eighty members

cd /glade/work/johnsonb/cesm2_1_3/cime/scripts
./create_newcase --case /glade/work/johnsonb/cesm_runs/G.CAM6-Reanalysis.f09_t13.001.e80 --compset GIAF_HR --res f09_t13 --mach cheyenne --run-unsupported --project P86850054 --ninst 80 --multi-driver
cd /glade/work/johnsonb/cesm_runs/G.CAM6-Reanalysis.f09_t13.001.e80
./case.setup
./xmlquery --partial PE
[ ... ]
Results in group mach_pes
        NTASKS_PER_INST: ['ATM:72', 'LND:72', 'ICE:828', 'OCN:2208', 'ROF:72', 'GLC:72', 'WAV:72', 'ESP:1']
        ROOTPE: ['CPL:0', 'ATM:0', 'LND:0', 'ICE:0', 'OCN:828', 'ROF:0', 'GLC:0', 'WAV:0', 'ESP:0']

Results in group mach_pes_last
        COSTPES_PER_NODE: 36
        COST_PES: 242892
        MAX_MPITASKS_PER_NODE: 36
        MAX_TASKS_PER_NODE: 36
        TOTALPES: 242880

Build error in CESM2.1.3

Error

A build error occurs when trying to build with CESM2.1.3. This error doesn’t occur with CESM2.1.0.

 Building case in directory /glade/work/johnsonb/cases/G.benchmarking.f09_t13_e5
 sharedlib_only is False
 model_only is False
 Setting resource.RLIMIT_STACK to -1 from (-1, -1)
 Generating component namelists as part of build
 GET_REFCASE is false, the user is expected to stage the refcase to the run directory.
 Creating component namelists
    Calling /glade/work/johnsonb/cesm2_1_3/cime/src/components/data_comps/datm/cime_config/buildnml
    Running /glade/work/johnsonb/cesm2_1_3/cime/src/components/data_comps/datm/cime_config/buildnml
Traceback (most recent call last):
   File "/glade/work/johnsonb/cesm2_1_3/cime/src/components/data_comps/datm/cime_config/buildnml", line 251, in <module>
     _main_func()
   File "/glade/work/johnsonb/cesm2_1_3/cime/src/components/data_comps/datm/cime_config/buildnml", line 247, in _main_func
     buildnml(case, caseroot, "datm")
   File "/glade/work/johnsonb/cesm2_1_3/cime/src/components/data_comps/datm/cime_config/buildnml", line 229, in buildnml
     _create_namelists(case, confdir, inst_string, namelist_infile, nmlgen, data_list_path)
   File "/glade/work/johnsonb/cesm2_1_3/cime/src/components/data_comps/datm/cime_config/buildnml", line 123, in _create_namelists
     nmlgen.create_stream_file_and_update_shr_strdata_nml(config, case.get_value("CASEROOT"),stream, stream_path, data_list_path)
   File "/glade/work/johnsonb/cesm2_1_3/cime/scripts/Tools/../../scripts/lib/CIME/nmlgen.py", line 444, in create_stream_file_and_update_shr_strdata_nml
     strmobj = Stream(infile=stream_path)
   File "/glade/work/johnsonb/cesm2_1_3/cime/scripts/Tools/../../scripts/lib/CIME/XML/stream.py", line 22, in __init__
     GenericXML.__init__(self, infile, schema=schema)
   File "/glade/work/johnsonb/cesm2_1_3/cime/scripts/Tools/../../scripts/lib/CIME/XML/generic_xml.py", line 57, in __init__
     self.read(infile, schema)
   File "/glade/work/johnsonb/cesm2_1_3/cime/scripts/Tools/../../scripts/lib/CIME/XML/generic_xml.py", line 87, in read
     self.read_fd(fd)
   File "/glade/work/johnsonb/cesm2_1_3/cime/scripts/Tools/../../scripts/lib/CIME/XML/generic_xml.py", line 112, in read_fd
     self.tree = ET.parse(fd)
   File "/glade/u/home/johnsonb/miniconda2/lib/python2.7/xml/etree/ElementTree.py", line 1182, in parse
     tree.parse(source, parser)
   File "/glade/u/home/johnsonb/miniconda2/lib/python2.7/xml/etree/ElementTree.py", line 656, in parse
     parser.feed(data)
   File "/glade/u/home/johnsonb/miniconda2/lib/python2.7/xml/etree/ElementTree.py", line 1659, in feed
     self._raiseerror(v)
   File "/glade/u/home/johnsonb/miniconda2/lib/python2.7/xml/etree/ElementTree.py", line 1523, in _raiseerror
     raise err
 xml.etree.ElementTree.ParseError: junk after document element: line 4, column 0
 ERROR: /glade/work/johnsonb/cesm2_1_3/cime/src/components/data_comps/datm/cime_config/buildnml /glade/work/johnsonb/cases/G.benchmarking.f09_t13_e5 FAILED, see above
 ERROR: Case could not be built.

Processor element layouts

As described in the Building the case page, the default processor element (PE) layout for an eighty-member CESM POP2 case on the f09_t13 grid requires 242,880 total cores and there are only 145,152 cores on Cheyenne.

In order to run the benchmarking tests, the ensemble must be squeezed onto fewer PEs.

The default PE layout is:

cd /glade/work/johnsonb/cesm_runs/G.CAM6-Reanalysis.f09_t13.001.e80
./xmlquery --partial PE
[ ...  ]
Results in group mach_pes
    NTASKS_PER_INST: ['ATM:72', 'LND:72', 'ICE:828', 'OCN:2208', 'ROF:72', 'GLC:72', 'WAV:72', 'ESP:1']
    ROOTPE: ['CPL:0', 'ATM:0', 'LND:0', 'ICE:0', 'OCN:828', 'ROF:0', 'GLC:0', 'WAV:0', 'ESP:0']

Results in group mach_pes_last
    COSTPES_PER_NODE: 36
    COST_PES: 242892
    MAX_MPITASKS_PER_NODE: 36
    MAX_TASKS_PER_NODE: 36
    TOTALPES: 242880
[ ...  ]

Two nodes (72 tasks) are set for each instance of:

ATM, LND, ROF, GLC, and WAV

23 nodes (828 tasks) are set for each instance of:

ICE

61 and 1/3 nodes are set for each instance of:

OCN

Initial attempt

For an initial f09_t13 scale down, try the following nodes:

One node for each instance of:

1 node for ATM, LND, ROF, GLC, AND WAV

11 nodes for each instance of:

ICE

30 nodes for each instance of:

OCN

Working in `DART_pop2-asd-project`

The setup_CESM_hybrid_ensemble.csh script reads in DART_params.csh. Make t13 versions of each of these scripts:

cp DART_params.csh DART_params_t13.csh
cp setup_CESM_hybrid_ensemble.csh setup_CESM_hybrid_t13_ensemble.csh
vim setup_CESM_hybrid_t13_ensemble.csh
./setup_CESM_hybrid_t13_ensemble.csh

Configuration of CIME XML files

`config_compilers.xml`

<compiler MACH="derecho">
   <CFLAGS>
     <append> -march=core-avx2 </append>
   </CFLAGS>
   <FFLAGS>
     <append> -march=core-avx2 </append>
   </FFLAGS>
   <!-- <SLIBS>
     <append> -qmkl </append>
   </SLIBS> -->
   <LDFLAGS>
      <append> -L$(NETCDF_PATH)/lib -lnetcdff -lnetcdf </append>
   </LDFLAGS>
</compiler>

`config_machines.xml`

<machine MACH="derecho">
   <DESC>NCAR AMD EPYC system</DESC>
   <NODENAME_REGEX>de.*.hpc.ucar.edu</NODENAME_REGEX>
   <OS>CNL</OS>
   <COMPILERS>intel,gnu,cray,nvhpc,intel-oneapi,intel-classic</COMPILERS>
   <MPILIBS>mpich</MPILIBS>
   <CIME_OUTPUT_ROOT>$ENV{SCRATCH}</CIME_OUTPUT_ROOT>
   <DIN_LOC_ROOT>$ENV{CESMDATAROOT}/inputdata</DIN_LOC_ROOT>
   <DIN_LOC_ROOT_CLMFORC>$ENV{CESMDATAROOT}/inputdata/atm/datm7</DIN_LOC_ROOT_CLMFORC>
   <DOUT_S_ROOT>$CIME_OUTPUT_ROOT/archive/$CASE</DOUT_S_ROOT>
   <BASELINE_ROOT>$ENV{CESMDATAROOT}/ccsm_baselines</BASELINE_ROOT>
   <CCSM_CPRNC>$ENV{CESMDATAROOT}/cprnc/cprnc</CCSM_CPRNC>
   <GMAKE_J>16</GMAKE_J>
   <BATCH_SYSTEM>pbs</BATCH_SYSTEM>
   <SUPPORTED_BY>cseg</SUPPORTED_BY>
   <MAX_TASKS_PER_NODE>128</MAX_TASKS_PER_NODE>
   <MAX_MPITASKS_PER_NODE>128</MAX_MPITASKS_PER_NODE>
   <PROJECT_REQUIRED>TRUE</PROJECT_REQUIRED>
   <mpirun mpilib="default">
      <executable>mpiexec</executable>
      <arguments>
         <arg name="label"> --label</arg>
         <arg name="buffer"> --line-buffer</arg>
         <arg name="num_tasks" > -n {{ total_tasks }}</arg>
      </arguments>
   </mpirun>
   <module_system type="module" allow_error="true">
      <!-- BKJ June 16, 2023 Trying to fix error: ERROR: module command $ENV{LMOD_ROOT}/lmod/libexec/lmod python load     cesmdev/1.0 ncarenv/23.06 -->
      <!-- <init_path lang="perl">$ENV{LMOD_ROOT}/lmod/init/perl</init_path>
      <init_path lang="python">$ENV{LMOD_ROOT}/lmod/init/env_modules_python.py</init_path>
      <init_path lang="sh">$ENV{LMOD_ROOT}/lmod/init/sh</init_path>
      <init_path lang="csh">$ENV{LMOD_ROOT}/lmod/init/csh</init_path>
      <cmd_path lang="perl">$ENV{LMOD_ROOT}/lmod/libexec/lmod perl</cmd_path>
      <cmd_path lang="python">$ENV{LMOD_ROOT}/lmod/libexec/lmod python</cmd_path> -->
      <init_path lang="perl">/glade/u/apps/derecho/23.06/spack/opt/spack/lmod/8.7.20/gcc/7.5.0/pdxb/lmod/lmod/init/perl</init_path>
      <init_path lang="python">/glade/u/apps/derecho/23.06/spack/opt/spack/lmod/8.7.20/gcc/7.5.0/pdxb/lmod/lmod/init/env_modules_python.py</init_path>
      <init_path lang="sh">/glade/u/apps/derecho/23.06/spack/opt/spack/lmod/8.7.20/gcc/7.5.0/pdxb/lmod/lmod/init/sh</init_path>
      <init_path lang="csh">/glade/u/apps/derecho/23.06/spack/opt/spack/lmod/8.7.20/gcc/7.5.0/pdxb/lmod/lmod/init/csh</init_path>
      <cmd_path lang="perl">/glade/u/apps/derecho/23.06/spack/opt/spack/lmod/8.7.20/gcc/7.5.0/pdxb/lmod/lmod/libexec/lmod perl</cmd_path>
      <cmd_path lang="python">/glade/u/apps/derecho/23.06/spack/opt/spack/lmod/8.7.20/gcc/7.5.0/pdxb/lmod/lmod/libexec/lmod python</cmd_path>
      <cmd_path lang="sh">module</cmd_path>
      <cmd_path lang="csh">module</cmd_path>
      <modules>
      <!-- BKJ June 16, 2023 Trying to fix error: ERROR: module command $ENV{LMOD_ROOT}/lmod/libexec/lmod python load       cesmdev/1.0 ncarenv/23.06 -->
      <!-- <command name="load">cesmdev/1.0</command> -->
      <!-- <command name="load">ncarenv/23.06</command> -->
      <command name="purge"/>
      <command name="load">craype</command>
      </modules>
      <modules compiler="intel">
         <command name="load">intel/2023.0.0</command>
         <command name="load">mkl</command>
      </modules>
      <modules compiler="intel-oneapi">
         <command name="load">intel-oneapi/2023.0.0</command>
         <command name="load">mkl</command>
      </modules>
      <modules compiler="intel-classic">
         <command name="load">intel-classic/2023.0.0</command>
         <command name="load">mkl</command>
      </modules>
      <modules compiler="cray">
         <command name="load">cce/15.0.1</command>
         <command name="load">cray-libsci/23.02.1.1</command>
      </modules>
      <modules compiler="gnu">
         <command name="load">gcc/12.2.0</command>
         <command name="load">cray-libsci/23.02.1.1</command>
      </modules>
      <modules compiler="nvhpc">
         <command name="load">nvhpc/23.1</command>
      </modules>
      <modules>
         <command name="load">ncarcompilers/1.0.0</command>
         <command name="load">cmake</command>
      </modules>
      <modules mpilib="mpich">
         <command name="load">cray-mpich/8.1.25</command>
      </modules>
      <modules mpilib="mpi-serial">
         <command name="load">mpi-serial/2.3.0</command>
      </modules>
      <modules mpilib="mpi-serial">
         <command name="load">netcdf/4.9.2</command>
      </modules>
      <modules mpilib="!mpi-serial">
         <command name="load">netcdf-mpi/4.9.2</command>
         <command name="load">parallel-netcdf/1.12.3</command>
      </modules>
      <modules DEBUG="TRUE">
         <command name="load">parallelio/2.6.0-debug</command>
         <command name="load">esmf/8.5.0b23-debug</command>
      </modules>
      <modules DEBUG="FALSE">
         <command name="load">parallelio/2.6.0</command>
         <!-- BKJ June 16, 2023 -->
         <!-- <command name="load">esmf/8.5.0b23</command> -->
         <command name="load">esmf/8.4.2</command>
      </modules>
   </module_system>
   <environment_variables>
      <env name="OMP_STACKSIZE">64M</env>
      <env name="FI_CXI_RX_MATCH_MODE">hybrid</env>
      <!--      <env name="MPICH_COLL_SYNC">MPI_Bcast,MPI_AllReduce,MPI_Gatherv,MPI_Scatter,MPI_Scatterv,MPI_Reduce</   env> -->
      <env name="MPICH_COLL_SYNC">1</env>
      <env name="NETCDF_PATH">/glade/u/apps/derecho/23.06/spack/opt/spack/netcdf/4.9.2/intel-oneapi-mpi/2021.8.0/oneapi/2023.0.0/pbyg</env>
   </environment_variables>
   <!-- derecho has both gpfs and lustre file systems so I think this setting may cause issues -->
   <environment_variables mpilib="mpich">
      <env name="MPICH_MPIIO_HINTS">*:romio_cb_read=enable:romio_cb_write=enable:striping_factor=24</env>
   </environment_variables>
   <environment_variables comp_interface="nuopc">
      <!-- required on all systems for timing file output -->
      <env name="ESMF_RUNTIME_PROFILE">ON</env>
      <env name="ESMF_RUNTIME_PROFILE_OUTPUT">SUMMARY</env>
   </environment_variables>
</machine>

`config_batch.xml`

<batch_system MACH="derecho" type="pbs" >
   <batch_submit>qsub</batch_submit>
   <directives>
      <directive default="/bin/bash" > -S {{ shell }}  </directive>
      <directive> -l select={{ num_nodes }}:ncpus={{ max_tasks_per_node }}:mpiprocs={{ tasks_per_node }}:ompthreads={{  thread_count }}</directive>
   </directives>
   <queues>
      <queue walltimemax="4:00:00" nodemin="1" nodemax="2488" >main</queue>
   </queues>
</batch_system>

Precursor projects

CISL’s Advanced Research Computing Division announced that Derecho will be delayed until June 2022. This could provide lead time to pursue precursor projects (on a different platform such as KAUST’s Shaheen II) to test various aspects of the experimental setup such as adaptive inflation and the influence of certain observations such as sea surface salinity.

Adaptive inflation

An experiment contrasting the behavior of the inverse-gamma adaptive inflation algorithm of Gharamti (2019) in eddy-parameterizing and eddy-resolving ocean models has not been attempted. Such an experiment would enable the team to test the ASD experimental configuration over short (model week to month) integration periods.

Sea surface salinity

Deppenmeier et al. (2021) studied the temperature structure of the Pacific Cold Tongue (PCT). The PCT is adjacent to the Eastern Tropical Pacific Fresh Pool, yet the influence of salinity on the structure of the PCT is poorly understood. A/B experiments, in which sea surface salinity is assimilated and not assimilated could be conducted to gauge its influence on the predicability of the PCT.

Creating runs for get_close caching tests

Restart files for the initial ensemble

I’ve already copied the necessary POP and CICE restart files to a directory on scratch:

/glade/scratch/johnsonb/g.e20.G.TL319_t13.control.001/rest/0001-01-01-00000

The setup scripts create the necessary POP input files that reference these restart files. If, at a later date, the files have been wiped from scratch, please let me know and I’ll rerun the script that copies them from campaign storage:

/glade/u/home/johnsonb/pyscripts/copy_restart_files.py

The restart files are from one of Alper Altuntas’s (who is a software engineer in CGD’s Ocean Section) high-resolution POP runs and there is restricted access to the Ocean Sections’ directory on campaign storage that the restart files are stored in.

Modified CESM installation

This project uses a custom grid within CESM that allows us to use the CAM6 + DART reanalysis to prescribe realistic temperature, precipitation/evaporation and momentum fluxes from the atmosphere to the ocean.

CESM provides many grid configurations. The reanalysis was computed on a 1°x1° grid in CAM which is denoted f09. POP’s mesoscale eddy-resolving high-resolution grid, t13, isn’t used very often because running ocean physics at such high-resolution is computationally intensive. So in order to use the CAM6 reanalysis and the high-resolution ocean grid together, I defined a custom grid configuration within CESM, f09_t13, to couple CAM and POP together.

This custom grid definition is stored in this xml file in this CESM repository:

/glade/work/johnsonb/cesm2_1_1/cime/config/cesm/config_grids.xml

If you are running tests on Cheyenne, the setup scripts are already set to reference this installation of CESM. If you are running the tests on another system, this entire cesm2_1_1 directory must be copied to that system.

Repository containing setup scripts

The relevant subdirectory of the DART repository containing the setup scripts is here:

/glade/work/johnsonb/git/DART_pop2-asd-project/models/POP/shell_scripts/cesm2_1

If you recursively copy this DART repository into the analagous subdirectory in your work directory on glade, /glade/work/$USER/git/DART_pop2-asd-project you won’t need to modify the setup scripts.

Only two of the scripts are actually used:

DART_params.csh
setup_CESM_startup_t13_ensemble.csh

Using DART_params.csh

Again, if you recursively copied the DART repository, the scripts should work without modification. Otherwise, you’ll likely need to change line 90 in DART_params.csh:

90 setenv DARTROOT        /glade/work/${USER}/git/DART_pop2-asd-project

Ensure CESM SourceMods are present

In order to run POP with DART, specific source code files in POP’s source code need to be overwritten. This modified source code:

Inserts flags into CESM that enable data assimilation within the model
Modifies one of the physics routines – the barotripic mode solver – to ensure that DART’s filter increments don’t crash the solver

The setup script will stage the modified source code for you. If it crashes, ensure the SourceMods are present where DART_params.csh expects them:

export cesmtag=cesm2_1_1
ls /glade/u/home/johnsonb/${cesmtag}/SourceMods

Using setup_CESM_startup_t13_ensemble.csh

Actually setting up the case should be as simple as:

./setup_CESM_startup_t13_ensemble.csh

Important

POP is a peculiar model within CESM. Historically, the initial files were binary rather than netCDF. So the setup script will configure CESM to run for two days, after which, you should run the CESM_DART_config.csh script that is placed into the case directory. It will configure the case to enable data assimilation.

After the setup script completes, you should get a success message such as:

[ ... ]
Time spent not building: 43.486701 sec
Time spent building: 1000.047169 sec
MODEL BUILD HAS FINISHED SUCCESSFULLY

Change directory to the case directory and submit the first (two model day) job.

cd $CASEROOT
./xmlquery STOP_N
     STOP_N:  2
./xmlquery STOP_OPTION
     STOP_OPTION: nday
./xmlquery --partial DATA_ASSIMILATION
     Results in group external_tools
        DATA_ASSIMILATION: ['CPL:FALSE', 'ATM:FALSE', 'LND:FALSE', 'ICE:FALSE', 'OCN:FALSE', 'ROF:FALSE', 'GLC:FALSE', 'WAV:FALSE']
        DATA_ASSIMILATION_CYCLES: 1
        DATA_ASSIMILATION_SCRIPT:
./case.submit -M begin,end

After that job completes, return to the case directory and run the CESM_DART_config.csh script. DART will complete an assimilation cycle when this run is submitted.

cd $CASEROOT
./CESM_DART_config.csh
./xmlchange STOP_N=1
./xmlquery STOP_N
     STOP_N:  1
./xmlquery --partial DATA_ASSIMILATION
     Results in group external_tools
        DATA_ASSIMILATION: ['CPL:FALSE', 'ATM:FALSE', 'LND:FALSE', 'ICE:FALSE', 'OCN:TRUE', 'ROF:FALSE', 'GLC:FALSE',  'WAV:FALSE']
        DATA_ASSIMILATION_CYCLES: 1
        DATA_ASSIMILATION_SCRIPT: $CASEROOT/assimilate.csh
./case.submit -M begin,end