Creating runs for get_close caching tests

Restart files for the initial ensemble

I’ve already copied the necessary POP and CICE restart files to a directory on scratch:

/glade/scratch/johnsonb/g.e20.G.TL319_t13.control.001/rest/0001-01-01-00000

The setup scripts create the necessary POP input files that reference these restart files. If, at a later date, the files have been wiped from scratch, please let me know and I’ll rerun the script that copies them from campaign storage:

/glade/u/home/johnsonb/pyscripts/copy_restart_files.py

The restart files are from one of Alper Altuntas’s (who is a software engineer in CGD’s Ocean Section) high-resolution POP runs and there is restricted access to the Ocean Sections’ directory on campaign storage that the restart files are stored in.

Modified CESM installation

This project uses a custom grid within CESM that allows us to use the CAM6 + DART reanalysis to prescribe realistic temperature, precipitation/evaporation and momentum fluxes from the atmosphere to the ocean.

CESM provides many grid configurations. The reanalysis was computed on a 1°x1° grid in CAM which is denoted f09. POP’s mesoscale eddy-resolving high-resolution grid, t13, isn’t used very often because running ocean physics at such high-resolution is computationally intensive. So in order to use the CAM6 reanalysis and the high-resolution ocean grid together, I defined a custom grid configuration within CESM, f09_t13, to couple CAM and POP together.

This custom grid definition is stored in this xml file in this CESM repository:

/glade/work/johnsonb/cesm2_1_1/cime/config/cesm/config_grids.xml

If you are running tests on Cheyenne, the setup scripts are already set to reference this installation of CESM. If you are running the tests on another system, this entire cesm2_1_1 directory must be copied to that system.

Repository containing setup scripts

The relevant subdirectory of the DART repository containing the setup scripts is here:

/glade/work/johnsonb/git/DART_pop2-asd-project/models/POP/shell_scripts/cesm2_1

If you recursively copy this DART repository into the analagous subdirectory in your work directory on glade, /glade/work/$USER/git/DART_pop2-asd-project you won’t need to modify the setup scripts.

Only two of the scripts are actually used:

  1. DART_params.csh

  2. setup_CESM_startup_t13_ensemble.csh

Using DART_params.csh

Again, if you recursively copied the DART repository, the scripts should work without modification. Otherwise, you’ll likely need to change line 90 in DART_params.csh:

90 setenv DARTROOT        /glade/work/${USER}/git/DART_pop2-asd-project

Ensure CESM SourceMods are present

In order to run POP with DART, specific source code files in POP’s source code need to be overwritten. This modified source code:

  1. Inserts flags into CESM that enable data assimilation within the model

  2. Modifies one of the physics routines – the barotripic mode solver – to ensure that DART’s filter increments don’t crash the solver

The setup script will stage the modified source code for you. If it crashes, ensure the SourceMods are present where DART_params.csh expects them:

export cesmtag=cesm2_1_1
ls /glade/u/home/johnsonb/${cesmtag}/SourceMods

Using setup_CESM_startup_t13_ensemble.csh

Actually setting up the case should be as simple as:

./setup_CESM_startup_t13_ensemble.csh

Important

POP is a peculiar model within CESM. Historically, the initial files were binary rather than netCDF. So the setup script will configure CESM to run for two days, after which, you should run the CESM_DART_config.csh script that is placed into the case directory. It will configure the case to enable data assimilation.

After the setup script completes, you should get a success message such as:

[ ... ]
Time spent not building: 43.486701 sec
Time spent building: 1000.047169 sec
MODEL BUILD HAS FINISHED SUCCESSFULLY

Change directory to the case directory and submit the first (two model day) job.

cd $CASEROOT
./xmlquery STOP_N
     STOP_N:  2
./xmlquery STOP_OPTION
     STOP_OPTION: nday
./xmlquery --partial DATA_ASSIMILATION
     Results in group external_tools
        DATA_ASSIMILATION: ['CPL:FALSE', 'ATM:FALSE', 'LND:FALSE', 'ICE:FALSE', 'OCN:FALSE', 'ROF:FALSE', 'GLC:FALSE', 'WAV:FALSE']
        DATA_ASSIMILATION_CYCLES: 1
        DATA_ASSIMILATION_SCRIPT:
./case.submit -M begin,end

After that job completes, return to the case directory and run the CESM_DART_config.csh script. DART will complete an assimilation cycle when this run is submitted.

cd $CASEROOT
./CESM_DART_config.csh
./xmlchange STOP_N=1
./xmlquery STOP_N
     STOP_N:  1
./xmlquery --partial DATA_ASSIMILATION
     Results in group external_tools
        DATA_ASSIMILATION: ['CPL:FALSE', 'ATM:FALSE', 'LND:FALSE', 'ICE:FALSE', 'OCN:TRUE', 'ROF:FALSE', 'GLC:FALSE',  'WAV:FALSE']
        DATA_ASSIMILATION_CYCLES: 1
        DATA_ASSIMILATION_SCRIPT: $CASEROOT/assimilate.csh
./case.submit -M begin,end