Skip to content

Workflow Stages#

The coastal calibration workflow consists of sequential stages, each performing a specific task in the simulation pipeline. The stage order depends on the selected model (SCHISM or SFINCS).

The run command executes all stages sequentially. All stages run natively; no containers are required. Stages execute locally on the allocated compute nodes (e.g., inside an sbatch script).

SCHISM Stage Overview#

flowchart TD
    A[download] --> B[schism_forcing_prep]
    B --> C[schism_forcing]
    C --> D[schism_sflux]
    D --> E[schism_params]
    E --> F[schism_obs]
    F --> G[schism_boundary]
    G --> H[schism_prep]
    H --> I[schism_run]
    I --> J[schism_postprocess]
    J --> K[schism_plot]

SFINCS Stage Overview#

flowchart TD
    A[download] --> B[sfincs_symlinks]
    B --> C[sfincs_data_catalog]
    C --> D[sfincs_init]
    D --> E[sfincs_timing]
    E --> F[sfincs_forcing]
    F --> G[sfincs_discharge]
    G --> H[sfincs_precip]
    H --> I[sfincs_wind]
    I --> J[sfincs_pressure]
    J --> K[sfincs_write]
    K --> L[sfincs_run]
    L --> M[sfincs_floodmap]
    M --> N[sfincs_plot]

SCHISM Stage Details#

1. download#

Purpose: Download required input data from remote sources.

Data Sources:

  • NWM meteorological forcing (LDASIN files)
  • NWM streamflow data (CHRTOUT files)
  • STOFS or GLOFS water level data (when applicable)

Runs On: Compute node (Python-only)

Outputs:

raw_download_dir/
├── meteo/nwm_ana/
│   └── *.LDASIN_DOMAIN1.nc
├── hydro/nwm/
│   └── *.CHRTOUT_DOMAIN1.nc
└── coastal/stofs/
    └── *.fields.cwl.nc

2. schism_forcing_prep#

Purpose: Prepare NWM forcing data for SCHISM.

Tasks:

  • Stage and organize downloaded NWM LDASIN and CHRTOUT files
  • Set up directory structure for forcing generation
  • Validate input data integrity

Runs On: Compute node (Python)

3. schism_forcing#

Purpose: Generate atmospheric forcing files using MPI.

Tasks:

  • Regrid NWM data to SCHISM mesh using ESMF
  • Interpolate forcing variables
  • Generate SCHISM-compatible forcing files

Runs On: Compute node (MPI parallel via mpiexec)

4. schism_sflux#

Purpose: Generate sflux atmospheric forcing files.

Tasks:

  • Generate sflux air, precipitation, and radiation files
  • Inline sea-level pressure reduction

Runs On: Compute node (Python)

5. schism_params#

Purpose: Generate SCHISM parameter file and symlink mesh files.

Tasks:

  • Symlink hgrid.gr3 and other mesh files into the work directory
  • Create param.nml with simulation parameters
  • Set time stepping configuration
  • Configure output options

Runs On: Compute node (Python)

Outputs:

work_dir/
├── hgrid.gr3 (symlink)
└── param.nml

6. schism_obs#

Purpose: Automatically discover NOAA CO-OPS water level stations within the model domain and generate a SCHISM station.in file so that SCHISM writes time-series output at those locations.

Enabled by: model_config.include_noaa_gages: true (disabled by default)

How it works:

  1. Parses hgrid.gr3 to extract the coordinates of all open boundary nodes.
  2. Computes a concave hull around the boundary points (using shapely.concave_hull with ratio=0.05).
  3. Queries the NOAA CO-OPS API to find active water level stations that fall inside the hull polygon.
  4. Writes station.in (SCHISM station definition file with elevation-only output flag) and a companion station_noaa_ids.txt that maps each station index to its NOAA station ID.

Runs On: Compute node (Python). Requires network access for the CO-OPS API call.

Outputs:

work_dir/
├── station.in             # SCHISM station definition file
└── station_noaa_ids.txt   # Index-to-NOAA-ID mapping

param.nml patching

When station.in exists, the schism_prep stage automatically patches param.nml to set iout_sta = 1 (enable station output) and nspool_sta = 18 (output interval in time-steps). The value nspool_sta = 18 is chosen because it divides all nhot_write values used across domain templates (162, 324, 8640, etc.), satisfying the SCHISM constraint mod(nhot_write, nspool_sta) == 0.

7. schism_boundary#

Purpose: Generate boundary conditions from TPXO or STOFS.

Tasks (TPXO):

  • Extract tidal constituents at boundary nodes
  • Generate harmonic boundary files
  • Create bctides.in file

Tasks (STOFS):

  • Interpolate STOFS water levels to boundary
  • Generate time-varying boundary files
  • Create elev2D.th.nc file

Runs On: Compute node (Python / MPI for ESMF regridding)

8. schism_prep#

Purpose: Final preparation before SCHISM execution.

Tasks:

  • Validate all input files present
  • Partition the mesh using METIS
  • Combine hotstart files
  • Configure MPI environment

Runs On: Compute node (Python + subprocess calls to metis_prep, gpmetis, combine_hotstart7)

9. schism_run#

Purpose: Execute the SCHISM model.

Tasks:

  • Run SCHISM with MPI via mpiexec
  • Monitor progress
  • Handle I/O scribes

Runs On: Compute node (MPI parallel, native binary)

Configuration:

  • Uses nscribes I/O processes from model config
  • OpenMP threads configured via omp_num_threads
  • Total processes = nodes * ntasks_per_node

10. schism_postprocess#

Purpose: Post-process SCHISM outputs.

Tasks:

  • Combine output files
  • Generate statistics
  • Create visualization-ready files

Runs On: Compute node (Python)

11. schism_plot#

Purpose: Compare SCHISM-simulated water levels against NOAA CO-OPS observations at every station discovered by the schism_obs stage.

Enabled by: model_config.include_noaa_gages: true

How it works:

  1. Reads the SCHISM station time-series from outputs/staout_1.
  2. Loads the station-to-NOAA-ID mapping from station_noaa_ids.txt.
  3. Fetches observed water levels from the CO-OPS API in the MLLW datum and converts to MSL using per-station datum offsets retrieved from the API.
  4. Generates 2×2 comparison plots (four stations per figure) showing simulated vs observed water levels.
  5. Saves PNG figures to the figs/ subdirectory.

Runs On: Compute node (Python-only). Requires network access for the CO-OPS API call.

Outputs:

work_dir/
└── figs/
    ├── stations_comparison_001.png
    ├── stations_comparison_002.png
    └── ...

Datum conversion

Observations are fetched in MLLW (Mean Lower Low Water) and converted to MSL (Mean Sea Level) so they share the same vertical reference as the SCHISM simulation output. The conversion offset is obtained from the CO-OPS datums endpoint for each station.

SFINCS Stage Details#

1. download#

Same as SCHISM download stage. Downloads NWM meteorological forcing, streamflow, and STOFS water level data.

Purpose: Create .nc symlinks for NWM data.

Tasks:

  • Create symlinks in the working directory pointing to downloaded NWM files
  • Organize files by type (meteo, hydro)

3. sfincs_data_catalog#

Purpose: Generate a HydroMT data catalog.

Tasks:

  • Build YAML data catalog for HydroMT-SFINCS
  • Register NWM meteo, streamflow, and STOFS data sources

4. sfincs_init#

Purpose: Initialize the SFINCS model from a pre-built template.

Tasks:

  • Copy pre-built SFINCS model from prebuilt_dir into the work directory
  • Remove stale netCDF output files from any previous run (prevents HDF5 segfaults when write_netcdf_safely encounters files with an incompatible schema)
  • Set up model directory structure

5. sfincs_timing#

Purpose: Set SFINCS model timing.

Tasks:

  • Configure simulation start/end times
  • Set output intervals

6. sfincs_forcing#

Purpose: Add water level forcing.

Tasks:

  • Read boundary point locations from sfincs.bnd
  • For TPXO: synthesize tidal water levels from TPXO constituents using harmonic reconstruction
  • For geodataset sources (STOFS): load the geodataset clipped around the boundary points, spatially interpolate to boundary locations using inverse-distance weighting (IDW), and inject into the HydroMT model
  • Apply forcing_to_mesh_offset_m to anchor the forcing signal to the correct height on the mesh datum (see note below)
  • Emit a warning if the adjusted water levels fall outside the ±15 m sanity range
  • Write boundary forcing netCDF (sfincs_netbndbzsbzifile.nc) with a zero-filled bzi (infragravity) variable required by the SFINCS binary

Forcing vertical datum offset

Tidal-only sources like TPXO provide oscillations centered on zero (MSL) but carry no information about where MSL sits on the mesh's vertical datum. The forcing_to_mesh_offset_m parameter anchors the tidal signal to the correct geodetic height on the mesh. For sources already in the mesh datum (e.g. STOFS on a NAVD88 mesh), set this to 0.0. The offset can be obtained from the NOAA VDatum API.

7. sfincs_discharge#

Purpose: Add discharge sources.

Tasks:

  • Add NWM streamflow discharge points from discharge_locations_file
  • Filter out source points that fall on inactive grid cells (prevents a SFINCS Fortran segfault caused by out-of-bounds array access)
  • Generate discharge forcing time series

8. sfincs_precip#

Purpose: Add precipitation forcing.

Tasks:

  • Add NWM precipitation data as spatially distributed forcing
  • Set the output resolution to meteo_res (or auto-derive from the quadtree grid)
  • Clip the reprojected grid to the model domain to prevent CONUS-scale inflation

9. sfincs_wind#

Purpose: Add wind forcing.

Tasks:

  • Add NWM wind data as spatially distributed forcing
  • Set the output resolution to meteo_res (or auto-derive from the quadtree grid)
  • Clip the reprojected grid to the model domain to prevent CONUS-scale inflation

Runs On: Login node (Python-only)

10. sfincs_pressure#

Purpose: Add atmospheric pressure forcing.

Tasks:

  • Add NWM atmospheric pressure data as spatially distributed forcing
  • Set the output resolution to meteo_res (or auto-derive from the quadtree grid)
  • Clip the reprojected grid to the model domain to prevent CONUS-scale inflation
  • Enable barometric pressure correction (baro=1)

Runs On: Login node (Python-only)

11. sfincs_write#

Purpose: Write the final SFINCS model.

Tasks:

  • Write all SFINCS input files (sfincs.inp, sfincs.bnd, etc.)
  • Generate boundary and forcing NetCDF files

Runs On: Login node (Python-only)

12. sfincs_run#

Purpose: Execute the SFINCS model.

Tasks:

  • Run SFINCS via compiled native binary
  • Uses single-node OpenMP parallelism (omp_num_threads)

Runs On: Compute node (OpenMP, native binary)

13. sfincs_floodmap#

Purpose: Downscale SFINCS water levels to a high-resolution flood depth map.

Tasks:

  • Read zsmax (maximum water surface elevation) from the SFINCS map output (sfincs_map.nc) via SfincsModel (handles both quadtree and regular grids)
  • Create an index COG that maps high-resolution DEM pixels to SFINCS grid cells
  • Compute flood depth (water surface minus DEM elevation) block-by-block and write a Cloud Optimized GeoTIFF (floodmap_hmax.tif)

Enabled by: model_config.floodmap_dem pointing to a high-resolution DEM. The stage is skipped when floodmap_dem is not configured, floodmap_enabled is false, sfincs_map.nc does not exist, or zsmax is not present in the map output.

Configuration:

  • floodmap_dem: path to the high-resolution DEM GeoTIFF
  • floodmap_hmin: minimum depth threshold (default: 0.05 m); shallower cells are masked out
  • floodmap_enabled: set to false to skip this stage entirely (default: true)

Runs On: Login node or compute node (Python-only)

Outputs:

model_root/
├── floodmap_hmax.tif    # Flood depth COG
└── floodmap_index.tif   # Index COG (DEM pixel → SFINCS cell mapping)

14. sfincs_plot#

Purpose: Compare simulated water levels against observations.

Tasks:

  • Read SFINCS output at observation points
  • Apply vdatum_mesh_to_msl_m to convert model output from the mesh datum to MSL
  • Fetch NOAA CO-OPS observed water levels (MLLW converted to MSL using per-station datum offsets from the CO-OPS API)
  • Generate comparison plots (simulated vs observed)
  • Save figures to the figs/ directory

Runs On: Login node or compute node (Python, requires network access)

Output datum conversion

SFINCS output inherits the vertical datum of the mesh (e.g. NAVD88). The vdatum_mesh_to_msl_m offset converts the simulated water levels to MSL so they can be compared with NOAA CO-OPS observations. This value can be obtained from the NOAA VDatum API.

Running Partial Workflows#

The run command supports --start-from and --stop-after.

CLI#

# SCHISM examples
coastal-calibration run config.yaml --stop-after download
coastal-calibration run config.yaml --start-from schism_forcing_prep --stop-after schism_sflux
coastal-calibration run config.yaml --start-from schism_boundary

# SFINCS examples
coastal-calibration run config.yaml --stop-after sfincs_write
coastal-calibration run config.yaml --start-from sfincs_run

Python API#

from coastal_calibration import CoastalCalibConfig, CoastalCalibRunner

config = CoastalCalibConfig.from_yaml("config.yaml")
runner = CoastalCalibRunner(config)

# Run specific stages
result = runner.run(start_from="schism_forcing_prep", stop_after="schism_sflux")

Error Handling#

If a stage fails:

  1. The workflow stops at the failed stage
  2. Error details are logged
  3. Subsequent stages are not executed
  4. Exit code indicates failure

To resume after fixing an issue:

# Resume from the failed stage
coastal-calibration run config.yaml --start-from <failed_stage>

Stage Timing#

When enable_timing is true in the monitoring configuration, stage durations are tracked and reported:

Stage timing:
  download: 45.2s
  schism_forcing_prep: 12.3s
  schism_forcing: 234.5s
  schism_sflux: 8.7s
  schism_params: 2.1s
  schism_obs: 3.8s
  schism_boundary: 156.8s
  schism_prep: 5.4s
  schism_run: 1823.6s
  schism_postprocess: 67.2s
  schism_plot: 15.4s
Total: 2359.4s

SFINCS Creation Stages#

The create command builds a new SFINCS quadtree model from an AOI polygon. It uses a separate configuration schema (SfincsCreateConfig) and a dedicated runner (SfincsCreator) with resumable execution. Completion is tracked in .create_status.json so that interrupted runs can be resumed with --start-from.

flowchart TD
    A[create_grid] --> B[create_fetch_data]
    B --> C[create_elevation]
    C --> D[create_mask]
    D --> E[create_boundary]
    E --> F["create_discharge (optional)"]
    F --> G[create_subgrid]
    G --> H["create_obs (optional)"]
    H --> I[create_write]

1. create_grid#

Purpose: Generate a SFINCS quadtree grid from the AOI polygon.

Tasks:

  • Read the AOI polygon (GeoJSON, Shapefile, etc.)
  • Create the base grid in the specified CRS
  • Apply quadtree refinement based on configured levels and criteria

2. create_fetch_data#

Purpose: Fetch elevation and land cover data for the AOI.

Enabled by: Any elevation dataset with a source field, or subgrid.lulc_source. Skipped when all datasets are user-provided (no auto-fetch configured).

Tasks:

  • For NOAA DEM sources: query the packaged spatial index to find the best-matching dataset based on AOI overlap, resolution, and year, download DEM tiles from S3, mosaic, and clip to the AOI extent
  • For NWS topobathy sources: fetch from the NWS icechunk S3 store clipped to the AOI bounding box
  • For Copernicus DEM / GEBCO sources: download tiles and mosaic
  • For ESA WorldCover land cover: download and mosaic land-use / land-cover tiles
  • Write a HydroMT data catalog YAML for the fetched datasets

3. create_elevation#

Purpose: Add elevation and bathymetry data to the grid.

Tasks:

  • Load configured elevation datasets (e.g., fetched NOAA DEM, NWS topobathy)
  • Interpolate elevation values onto the quadtree grid cells
  • Apply zmin/zmax filters per dataset

4. create_mask#

Purpose: Create the active cell mask.

Tasks:

  • Determine which grid cells are active based on elevation thresholds
  • Apply land/water masking criteria

5. create_boundary#

Purpose: Create water level boundary cells.

Tasks:

  • Identify grid cells along the open ocean boundary
  • Assign boundary condition flags

6. create_discharge (optional)#

Purpose: Add river discharge source points to the model.

Enabled by: Configuring a river_discharge section in the creation config. Skipped when river_discharge is not present.

Tasks:

  • Read NWM hydrofabric flowpath linestrings from a GeoPackage
  • Intersect selected flowpaths with the AOI boundary to locate discharge inflow points
  • Snap source points to the nearest active grid cell
  • Write the SFINCS .src file and a discharge locations file usable by the simulation workflow

7. create_subgrid#

Purpose: Generate subgrid lookup tables.

Tasks:

  • Compute high-resolution subgrid tables from the DEM
  • These tables allow SFINCS to use coarse computational cells while capturing fine-scale topographic detail

8. create_obs (optional)#

Purpose: Add observation points to the model.

Enabled by: Setting add_noaa_gages: true, providing observation_points, or providing an observation_locations_file in the creation config. Skipped when none of these are configured.

Tasks:

  • When add_noaa_gages is true: query NOAA CO-OPS for active water level stations within the model domain and add them as observation points
  • When observation_points or observation_locations_file is provided: add the user-specified observation points
  • Write observation point locations into the SFINCS model

9. create_write#

Purpose: Write the complete SFINCS model to disk.

Tasks:

  • Write all SFINCS input files to the configured output_dir
  • The output directory can be used as prebuilt_dir in a simulation config