Shell Reference#

Configuration#

config.bashrc#

Configuration file for building and running nwm-rte.

Description

This file is source'd by multiple shell scripts within nwm-rte, and its variables control various aspects of the installation / setup / build as well as the runtime environment itself (e.g. mount paths).

Choices of Component Packages:

For each component package (nwm-msw-mgr, nwm-fcst-mgr, etc), you can choose from an empty string (""), or "LOCAL", or any other string.

If an empty string "" is provided, the package will be skipped (will not be installed).
If "LOCAL" is provided, the package will install from the code currently on disk.
If any other string is provided, the package will install from GitHub, and the string must be a valid tag, branch, or commit.

Environment Variables

NO_CACHE:

Docker cache directive passed to docker build call. Choose from: ["--no-cache", ""]
STAGE:

The stage of the docker build sequence to stop at. Choose from: ["ngen_rte_base", "ngen_rte_eval_verf"]. See Dockerfile.rte for more information
INSTALL_DEBUGGERS:

Passed to ./install_debuggers.sh, causes pip package debugpy and dnf package gdb to be installed. Choose from: ["NO", "YES"]
NGEN_LOG_TO_RTE:

Causes ngen's log files to be written to ./ngen_logs rather than to the realization's dir. Choose from: ["NO", "YES"]

Component Packages#
REPO_TAG_FCST_MGR:

Source for nwm-fcst-mgr.
REPO_TAG_MSW_MGR:

Source for nwm-mswm-mgr.
REPO_TAG_CAL_MGR:

Source for nwm-cal-mgr.
REPO_TAG_REGION_MGR:

Source for nwm-region-mgr.
REPO_TAG_DATA_ASSIM_ENGINE:

Source for nwm-data-assimilation.
REPO_TAG_NGEN_FORCING:

(Optional). Source for re-installing ngen-forcing Python package. Note that this does not affect the sourcing of the base image.
REPO_TAG_VERF:

Source for nwm-verf.
REPO_TAG_EVAL:

Source for nwm-eval-mgr.

Sourcing of ngen Base#
NGEN_SOURCE_MODE:
Affects the path of building, the choice of the base ngen image. Choose from: ["ghcr", "existing_local_tag", "build_from_local", "build_from_remote"]
- "ghcr" causes the ngen base image to be sourced from an existing GitHub Container Registry image that has already been published to the ngen GitHub repository.
- "existing_local_tag" causes the ngen base image to be sourced from an existing local image tag.
- "build_from_local" and "build_from_remote" cause the ngen base image to be built from the bottom up (build ngen-forcing, then build ngen, then add the RTE layers). The former uses the existing state of the codebase currently on disk for ngen-forcing and for ngen, and the latter uses the provided GH refs for sourcing the codebases of ngen-forcing and ngen.
NGEN_BASE__REMOTE_GHCR_TAG:

(Only used when NGEN_SOURCE_MODE="ghcr"). GHCR image tag to use, e.g. "latest" or a commit hash.
NGEN_BASE__EXISTING_LOCAL_TAG:

(Only used when NGEN_SOURCE_MODE="existing_local_tag"). Choose any existing local image tag.
FORCING_BASE_REMOTE_TAG:

Only used when NGEN_SOURCE_MODE="build_from_remote". Source of the ngen-forcing codebase. Choose any GitHub ref.
NGEN_BASE_REMOTE_TAG:

Only used when NGEN_SOURCE_MODE="build_from_remote". Source of the ngen codebase. Choose any GitHub ref.

Target Image Name#
TARGET_IMAGE_NAME:

Freeform name for output image tag that is built.

Misc#
THIS_SCRIPTS_GRANDPARENT_DIR:

The grandparent directory of this current file. When using REPOS_COMMON_ROOT__HOST=${THIS_SCRIPTS_GRANDPARENT_DIR}, then the other NWM repos are assumed to be siblings of the nwm-rte repo.
REPOS_COMMON_ROOT__HOST:
- ./setup_clone_repos.sh sets up this local directory and clones sibling repos into here. ./setup_data.sh downloads data into here.
- ./ngen_rte_build.sh uses this to find ngen source code when NGEN_SOURCE_MODE="build_from_local"
- ./ngen_rte_run.sh mounts various subdirectories and files from this local directory, into the container, during runtime.
- Choices for this variable:
  - A typical choice for this is ${THIS_SCRIPTS_GRANDPARENT_DIR}, which is equivalent to "${HOME}/ngwpc" if you run this from "${HOME}/ngwpc/nwm-rte" but another location such as "${HOME}/ngwpc__rte" could be used if wanting to isolate the RTE from other work.
  - Using ${THIS_SCRIPTS_GRANDPARENT_DIR} guarantees that the setup scripts (./setup_data.sh and ./setup_clone_repos.sh) will copy data and clone repos into the same locations where the build script and run script will look for them.
Mounts for data#
RUN_NGEN_ROOT__HOST:

Root of the working data associated with running realizations (dirs for realizations and for intermediary data).
MNT__RUN_NGEN__HOST:

Alias for RUN_NGEN_ROOT__HOST. Host path mounted by the container.
MNT__RUN_NGEN__CONTAINER_1:

1^st of 2 paths within the container to which the host disk path described by MNT__RUN_NGEN__HOST is mounted.
MNT__RUN_NGEN__CONTAINER_2:

2^nd of 2 paths within the container to which the host disk path described by MNT__RUN_NGEN__HOST is mounted.

Mounts for parameter files#
MNT__MODULE_PARAM_FILES_DIR__HOST:

Host path to input module parameter files from nwm-msw-mgr repository. Mounted by the container.
MNT__MODULE_PARAM_FILES_DIR__CONTAINER_1:

1^st of 2 paths within the container to which the host disk path described by MNT__MODULE_PARAM_FILES_DIR__HOST is mounted.
MNT__MODULE_PARAM_FILES_DIR__CONTAINER_2:

2^nd of 2 paths within the container to which the host disk path described by MNT__MODULE_PARAM_FILES_DIR__HOST is mounted.

Mounts for forcing configuration#
MNT__NGEN_FORCING__HOST:

Host disk path to ngen-forcing repo, for mounting configuration template files from ngen-forcing/NextGen_Forcings_Engine_BMI/BMI_NextGen_Configs/config_templates/.
MNT__NGEN_FORCING__CONTAINER_1:

1^st of 2 paths within the container to which the host disk path described by MNT__NGEN_FORCING__HOST is mounted.
MNT__NGEN_FORCING__CONTAINER_2:

2^nd of 2 paths within the container to which the host disk path described by MNT__NGEN_FORCING__HOST is mounted.

Mounts for S3 data#
S3_ROOT__HOST:

Root of input data downloaded from s3.
MNT__S3_DATA__HOST:

Alias for S3_ROOT__HOST. Host path mounted by the container.
MNT__S3_DATA__CONTAINER_1:

1^st of 2 paths within the container to which the host disk path described by MNT__S3_DATA__HOST is mounted.
MNT__S3_DATA__CONTAINER_1:

2^nd of 2 paths within the container to which the host disk path described by MNT__S3_DATA__HOST is mounted.

Mounts for regionalization data#
MNT__NWM_REGION_MGR__INPUT_DATA:

Host path to input data from nwm-region-mgr repository for regionalization workflows. Mounted by the container. Logging functions

Caveats

REPO_TAG_NGEN_FORCING provision causes the ngen-forcing Python package to be reinstalled, but does not affect other aspects of ngen-forcing (does not cause a rebuild from the ngen-forcing base). This does not affect the base image, does not reflect changes to C++ code nor changes to ngen-forcing/Dockerfile.bmi-forcings, nor changes to other ngen-forcing build instructions. It only causes an optional pip install call to be executed on the ngen-forcing Python package, after the manager packages have been installed on top of the ngen base image.

Notes

For Pattern VAR_NAME=${VAR_NAME:-"some_string"}, values set with this pattern use the existing value if it already exists, e.g. as an OS env var, otherwise they default to "some_string".

Setup#

setup_clone_repos.sh#

Clone various NWM repositories.

Usage

./setup_clone_repos.sh ssh

./setup_clone_repos.sh https shallow

Description

Clone repos from GitHub. Includes CLI option for using SSH (for r+w, requires credentials) or HTTPS (for read-only, no credentials required). Includes CLI option for a shallow (depth 1) clone or full clone. It will not alter the state of existing repos on your disk (for each clone, it skips if the folder already exists on disk).

Uses OS env var REPOS_COMMON_ROOT__HOST from config.bashrc.

Has 2 positional arguments and 0 named arguments.

Options

SSH_OR_HTTPS:

Required. Specify to clone via SSH (for read-write) or via HTTPS (for read-only). Choices: ["ssh", "https"]
SHALLOW_CLONE:

Optional. Specify to perform a shallow (--depth 1) clone or a deep clone. Choices: ["full", "shallow"]. Default: "full"

setup_data.sh#

Download various global input data, as well as specific input data for one example gage.

Usage

./setup_data.sh

./setup_data.sh -r

Description

Download input data from S3 and from APIs. Not needed for the "default" gage at the "default" realization, since those inputs are included in the repository. Calls ./setup_data_one_gage.sh for the default gage ID.

Uses various OS env vars from config.bashrc.

Has 0 positional arguments and 1 named argument.

Options

-r, --regionalization:

If provided, extra data will be downloaded for the regionalization workflow. Should be ran after cloning the nwm-region-mgr repository.

setup_data_one_gage.sh#

Download input data for one gage.

Usage

./setup_data_one_gage.sh "01121330" "CONUS"

Description

Download input hydrofabric data from S3 and from APIs, for one USGS gage. Not needed for the "default" gage at the "default" realization, since those inputs are included in the repository. Downloads all vintages of hydrofabric associated with one gage ID.

Uses various OS env vars from config.bashrc.

Has 2 positional arguments and 0 named arguments.

Options

GAGE_ID:

The USGS gage ID to be downloaded, e.g. "01121330"
DOMAIN:

The domain that the gage ID resides in, e.g. "CONUS"

Build#

ngen_rte_build.sh#

Build the ngen runtime environment Docker image.

Usage

./ngen_rte_build.sh

Description

Build the ngen runtime environment Docker image, applying various OS env vars from config.bashrc.

Can build the ngen-forcing and ngen base images if not wishing to source ngen from an existing GHCR image or an existing local image.

Components can be installed from GitHub, or from local source code, or skipped.

The image built is tagged using env var TARGET_IMAGE_NAME (typically from config.bashrc).

When the script exits, it echoes the image name and provides a command that can be used to enter the container interactively.

Alternatively, the user can run ./run.sh after the build is complete.

Has 0 positional arguments and 0 named arguments.

Requirements:

If configured to install ngwpc packages from local sources instead of from GitHub, then this script assumes this is ran from a sibling directory of those repositories (already cloned, checked out, pulled). ./setup_clone_repos.sh can be used to clone the relevant ngwpc repos.

Enter the Container Interactively#

run.sh#

Script and sourceable function for running the ngen runtime environment Docker image.

Usage

./run.sh

source ./run.sh && docker_run python "/path/to/script.py" --arg_for_script

Description

Run the ngen runtime environment Docker image. Requires that the image has already been built.

The function docker_run is sourced by and called by other scripts in the repository such as ./run_default.sh, ./run_calib.sh, ./run_fcst.sh, and ./run_tests.sh.

When run as a script, causes the user to enter the container shell interactively.

Various OS env vars are applied from config.bashrc. Notably TARGET_IMAGE_NAME is the image that is sourced (must have already been built) and launched as a container.

Various mount paths are also applied, some defined in config.bashrc and others defined in ./run.sh itself. * Input data paths * Output data paths from previous runs * Scripts from ./bin_mounted/ * Cached files from ./devcontainer/tmp/

This script has 0 positional arguments and 0 named arguments. See the description for function docker_run, which is typically sourced by other scripts and ran externally.

Requirements:

The ngen runtime environment image (defined by TARGET_IMAGE_NAME) has already been built.

The various repositories and input data needed to run are available and mountable (see ./setup_clone_repos.sh, ./setup_data.sh, ./setup_data_one_gage.sh).

Functions

docker_run#

Start the ngen runtime environment container with various mounts applied. Either run in an interactive terminal, or run a specific command and exit. If arguments are provided, then the 1^st argument is the entrypoint command, and all subsequent arguments are arguments for that entrypoint command. See usage examples in ./run_default.sh, etc.

Arguments

entrypoint:
If "/bin/sh", then start an interactive session. Else, run the provided command and provide the extra arguments as the arguments to that command.
*:
Any number of extra arguments, provided to the entrypoint command.

Run Forecasts and Calibrations#

run_default.sh#

Run "default" realizations.

Usage

./run_default.sh

Description

Source ./run.sh to call its docker_run command for running "default" realizations. It requires that the ngen runtime environment image has already been built using ./ngen_rte_build.sh.

Various OS env vars are applied from config.bashrc. Notably TARGET_IMAGE_NAME is the image that is sourced (must have already been built) and launched as a container. Various data mount paths are also applied from config.bashrc.

Has 0 positional arguments and 0 named arguments.

Requirements:

The ngen runtime environment image (defined by TARGET_IMAGE_NAME) has already been built.

The various repositories and input data needed to run are available and mountable (see ./setup_clone_repos.sh, ./setup_data.sh, ./setup_data_one_gage.sh).

run_calib.sh#

Run "calibration" realizations.

Usage

./run_calib.sh

Description

Source ./run.sh to call its docker_run command for running "calibration" realizations. It requires that the ngen runtime environment image has already been built using ./ngen_rte_build.sh.

Various OS env vars are applied from config.bashrc. Notably TARGET_IMAGE_NAME is the image that is sourced (must have already been built) and launched as a container. Various data mount paths are also applied from config.bashrc.

Has 0 positional arguments and 0 named arguments.

Requirements:

The ngen runtime environment image (defined by TARGET_IMAGE_NAME) has already been built.

The various repositories and input data needed to run are available and mountable (see ./setup_clone_repos.sh, ./setup_data.sh, ./setup_data_one_gage.sh).

run_fcst.sh#

Run "forecast" realizations.

Usage

./run_fcst.sh

Description

Source ./run.sh to call its docker_run command for running "forecast" realizations. It requires that the ngen runtime environment image has already been built using ./ngen_rte_build.sh.

Various OS env vars are applied from config.bashrc. Notably TARGET_IMAGE_NAME is the image that is sourced (must have already been built) and launched as a container. Various data mount paths are also applied from config.bashrc.

Requirements:

The ngen runtime environment image (defined by TARGET_IMAGE_NAME) has already been built.

The various repositories and input data needed to run are available and mountable (see ./setup_clone_repos.sh, ./setup_data.sh, ./setup_data_one_gage.sh).

A "calibration" realization has already been ran for the gage of interest, and its output is available at the expected location.

This script has 1 positional arguments and 0 named arguments.

Options

fcst_run_name:

Optional. Default="fcst_run". Specify to choose the name of the forecast run.

run_suite.sh#

Run realizations of various types.

Usage

./run_suite.sh

Description

Source ./run.sh to call its docker_run command for running realizations of various types ("default", "calibration", and "forecast"). It requires that the ngen runtime environment image has already been built using ./ngen_rte_build.sh.

Various OS env vars are applied from config.bashrc. Notably TARGET_IMAGE_NAME is the image that is sourced (must have already been built) and launched as a container. Various data mount paths are also applied from config.bashrc.

Has 0 positional arguments and 0 named arguments.

Requirements:

The ngen runtime environment image (defined by TARGET_IMAGE_NAME) has already been built.

The various repositories and input data needed to run are available and mountable (see ./setup_clone_repos.sh, ./setup_data.sh, ./setup_data_one_gage.sh).

run_tests.sh#

Run a series of "forecast" realizations via run_tests.py with various forcing configurations, optionally with "calibration" realizations first.

Usage

./run_tests.sh

Description

Source ./run.sh to call its docker_run command for running run_tests.py. It requires that the ngen runtime environment image has already been built using ./ngen_rte_build.sh.

When realizations fail, this program does not halt, but rather moves to the next configuration type in the list, with the goal of "trying" many different realization configurations in one call.

The status of each configuration's build step and run step is reported and written to a json file at the end.

The Python script run_tests.py includes options for stopping realizations mid-way through their run, rather than waiting for them to complete.

Various OS env vars are applied from config.bashrc. Notably TARGET_IMAGE_NAME is the image that is sourced (must have already been built) and launched as a container. Various data mount paths are also applied from config.bashrc.

Requirements:

The ngen runtime environment image (defined by TARGET_IMAGE_NAME) has already been built.

The various repositories and input data needed to run are available and mountable (see ./setup_clone_repos.sh, ./setup_data.sh, ./setup_data_one_gage.sh).

This script has 1 positional arguments and 0 named arguments.

Options

fcst_run_name:

Optional. Default="fcst_run". Specify to choose the name of the forecast run.

Additional Utiltities#

install_debuggers.sh#

Install debugger packages. Used by Dockerfile.rte.

Usage

./install_debuggers.sh YES

./install_debuggers.sh NO

Description

Install debugger packages: python package debugpy and dnf package gdb. Used by Dockerfile.rte. Optionally skip installation (noop). The choice of noop is to abstract the flow control away into a parameter to work around the non-branching nature of Docker build flow control.

Has 1 positional argument and 0 named arguments.

Options

install_debuggers:

Choose from: ["YES", "NO"]. If "NO", this is a noop.

install_package.sh#

Install a Python package, with options, and write git info json. Used by Dockerfile.rte

Usage

./install_package.sh python nwm-region-mgr development "[parreg]"

Description

Install a Python package, with options for the python environment, local vs remote, tag, and "extras". Used by Dockerfile.rte.

Also call add_git_info.py to dump the package's git info to a json file.

Treats ngen-forcing reinstall as special since it necessarily is already installed in the base image, so if it is specified here, it receives a --force-reinstall flag.

Has 4 positional arguments and 0 named arguments.

Options

python_exe:

The python executable to use for installation
repo_name:

The name of the repository to be installed
repo_remote_tag:

If "LOCAL", then install from local code. If empty string, skip installation. If other string, install that ref (tag, branch, or commit) from GitHub.
extras:

Optional extras to include in the pip install directive, e.g. pip install package_name[extras]

Run Regionalization Workflows#

For more information, see the nwm-region-mgr repository.

run_region.sh#

Script to run NWM regionalization, NGEN simulation, and evaluation workflows based on command line options.

Usage

cd [working_directory, e.g., /ngen-oe/$USER/run_region, /ngen-dev/$USER/run_region, or ~/run_region]
[NWM-RTE_ROOT]/run_region.sh [OPTIONS]

Description

Has 0 positional arguments and many named arguments.

Options

-p, --parreg:

Run parameter regionalization (includes formulation)
-f, --formreg:

Run formulation regionalization only
-n, --ngen:

Run NGEN simulation
-e, --eval:

Run evaluation
-c, --config_dirDIR:

(default: "./configs") Set config directory
-r, --reposPATH:

(default: auto-detect) Set root directory for NGWPC repos
-t, --image-tagTAG:

(default: "latest") Set Docker image tag
-i, --pull-image:

(optional switch) Pull the latest Docker image before running
-d, --delete-runtime-dir:

(optional switch) Delete runtime directory after completion
-h, --help:

Show this message and exit

Examples Run parameter regionalization

# Sample config files can be found in nwm-region-mgr repo under configs directory
/ngencerf-app/nwm-rte/run_region.sh --parreg -c configs
~/ngwpc/nwm-rte/run_region.sh --parreg -c configs
~/ngwpc/nwm-rte/run_region.sh -p -c configs
~/ngwpc/nwm-rte/run_region.sh -p -c ~/ngwpc/nwm-region-mgr/configs

Run formulation regionalization only

/ngencerf-app/nwm-rte/run_region.sh --formreg -c configs

Run NGEN simulation

/ngencerf-app/nwm-rte/run_region.sh --ngen -c configs

Run evaluation

/ngencerf-app/nwm-rte/run_region.sh --eval -c configs

Run multiple steps

/ngencerf-app/nwm-rte/run_region.sh --parreg --ngen -c configs

Run with different repos root directory

/ngencerf-app/nwm-rte/run_region.sh --ngen -c configs -r /ngen-oe/$USER

Run with different Docker image tag

/ngencerf-app/nwm-rte/run_region.sh --ngen -c configs -t pr-20-build

Run with pulling the latest Docker image before running

/ngencerf-app/nwm-rte/run_region.sh --ngen -c configs -i

Delete runtime directory after completion (default: keep for debugging)

/ngencerf-app/nwm-rte/run_region.sh -c configs -p -n -e -d

sbatch_run_region.sh#

Submits the regionalization workflow to SLURM using sbatch.

Description

Usage:

1. First cd to working directory, e.g., /ngen-oe/$USER/run_region, /ngen-dev/$USER/run_region, or ~/run_region.
2. Copy sample configs (nwm-region-mgr/configs) to working directory and edit as needed. You can also use the sample configs directly for testing purposes.
3. Then run one of the following commands depending on your setup:
     /ngencerf-app/nwm-rte/sbatch_run_region.sh CONFIG_DIR [OPTIONS] [--dry-run]
     ~/ngwpc/nwm-rte/sbatch_run_region.sh CONFIG_DIR [OPTIONS] [--dry-run]

Has 2 positional arguments and some named arguments.

Options

CONFIG_DIR:

(required) Directory containing configuration YAML files
OPTIONS:

(optional, default: "parreg") Workflow steps to runs: parreg, formreg, ngen, or eval
--dry-run:

(optional switch) If provided, print the generated SLURM script instead of submitting
--image-tagTAG:

(optional, default: "latest") Docker image tag to use for the RTE
--pull-image:

(optional switch) Pull the latest Docker image before running
--delete-runtime-dir:

(optional switch) Delete runtime directory after completion

Examples Do a dry-run to see the generated SLURM script without submitting

~/ngwpc/nwm-rte/sbatch_run_region.sh ~/ngwpc/nwm-region-mgr/configs parreg ngen eval --dry-run
/ngencerf-app/nwm-rte/sbatch_run_region.sh configs parreg ngen eval --dry-run

Submit parameter regionalization

/ngencerf-app/nwm-rte/sbatch_run_region.sh configs parreg
/ngencerf-app/nwm-rte/sbatch_run_region.sh configs

Submit formulation regionalization only

/ngencerf-app/nwm-rte/sbatch_run_region.sh configs formreg

Submit NGEN simulation with a different config directory

/ngencerf-app/nwm-rte/sbatch_run_region.sh /ngen-oe/$USER/myconfigs ngen

Submit evaluation only

/ngencerf-app/nwm-rte/sbatch_run_region.sh configs eval

Using an RTE image tag other than 'latest'

/ngencerf-app/nwm-rte/sbatch_run_region.sh configs ngen --image-tag "069ad0f6d332"

Pull the latest RTE image before running

/ngencerf-app/nwm-rte/sbatch_run_region.sh configs ngen --pull-image

Delete runtime directory after completion (default is to keep it for debugging)

/ngencerf-app/nwm-rte/sbatch_run_region.sh configs ngen --pull-image --delete-runtime-dir

Shell Reference#

Configuration#

config.bashrc#

Component Packages#

Sourcing of ngen Base#

Target Image Name#

Misc#

Mounts for data#

Mounts for parameter files#

Mounts for forcing configuration#

Mounts for S3 data#

Mounts for regionalization data#

Setup#

setup_clone_repos.sh#

setup_data.sh#

setup_data_one_gage.sh#

Build#

ngen_rte_build.sh#

Enter the Container Interactively#

run.sh#

docker_run#

Run Forecasts and Calibrations#

run_default.sh#

run_calib.sh#

run_fcst.sh#

run_suite.sh#

run_tests.sh#

Additional Utiltities#

install_debuggers.sh#

install_package.sh#

Run Regionalization Workflows#

run_region.sh#

sbatch_run_region.sh#