Gridded data in NetCDF format

Gridded, structured data can be read from NetCDF files that follow the Climate and Forecast (CF) Metadata Conventions. There are two options to load data into Met.3D:

During program runtime, load a dataset consisting of CF-compliant NetCDF files (a dataset can be composed of a single or of multiple files) by selecting "File→New Dataset" in the Met.3D main window.
Datasets can be read on program startup (useful if you are using the same dataset again and again, or if you are using the batch mode to automatically create animations). For this function, the dataset needs to be defined in a user-defined pipeline.cfg file, which needs to be passed to Met.3D on the command line with "--pipeline=yourfile". If you are creating such a file for the first time, copy the default template and modify it to your needs. It contains descriptions of the available options: https://gitlab.com/wxmetvis/met.3d/-/blob/master/config/default_pipeline.cfg.template

Forecast variables, time steps and ensemble members can be arbitrarily distributed over the files that match the file filter. You can store all ensemble members in one file but have different files for each time step, or vice versa. Or everything can be stored in a single file.

Examples of how NetCDF files need to be structured

The following example contains a NetCDF header of a file containing an ensemble forecast on pressure levels.

netcdf somegriddeddata.pl {
dimensions:
    lon = 101 ;
    lat = 41 ;
    isobaric = 12 ;
    time = 1 ;
    ens0 = 51 ;
variables:
    float lat(lat) ;
            lat:units = "degrees_north" ;
    float lon(lon) ;
            lon:units = "degrees_east" ;
    float isobaric(isobaric) ;
            isobaric:units = "hPa" ;
            isobaric:long_name = "Isobaric surface" ;
            isobaric:positive = "down" ;
    int time(time) ;
            time:units = "Hour since 2012-10-15T00:00:00.000Z" ;
            time:standard_name = "time" ;
    int ens0(ens0) ;
            ens0:standard_name = "ensemble_member_id" ;

    float Geopotential_isobaric(time, ens0, isobaric, lat, lon) ;
            Geopotential_isobaric:long_name = "Geopotential @ Isobaric surface" ;
            Geopotential_isobaric:units = "m2.s-2" ;
    float Temperature_isobaric(time, ens0, isobaric, lat, lon) ;
            Temperature_isobaric:long_name = "Temperature @ Isobaric surface" ;
            Temperature_isobaric:units = "K" ;

    ...
}

Note the following:

The latitude and longitude dimensions are recognised according to their units keyword; cf. the longitude/latitude section in CF-conventions.

The vertical pressure level dimension requires units of pressure, as well the positive attribute being defined; cf. the vertical coordinate section in CF-conventions.

The time dimension is identified by its units attribute; cf. the time coordinate section in CF-conventions.

The time encoded in the units attribute of the time dimension is used as the forecast base/initialisation time. If you are not using forecast data (e.g. if you are using reanalysis data), Met.3D requires you to either use the same reference time in all files of your dataset, or to have the reference time equal the valid time (i.e. lead time = 0) - this essentially means that you are using one file per time step.

The ensemble dimension is optional. If you don't have an ensemble dimension, the variables should have the dimensions (time, isobaric, lat, lon).

The ensemble dimension is currently not specified in the CF conventions. Met.3D simply searches for a variable with a standard_name attribute set to ensemble_member_id. For now, the _CoordinateAxisType used by the netcdf-java library (_CoordinateAxisType = "Ensemble") is also acceptable. (See the Met.3D source code method NcCFVar::getEnsembleVar() in nccfvar.cpp).

The following example contains a NetCDF header of a file containing an ensemble forecast on hybrid sigma-pressure levels; the required keywords of the vertical variable are different; cf. Appendix D of the CF-conventions.

netcdf somegriddeddata.ml {
dimensions:
    lon = 101 ;
    lat = 41 ;
    hybrid = 62 ;
    time = 1 ;
    ens0 = 51 ;
variables:
    float lat(lat) ;
            lat:units = "degrees_north" ;
    float lon(lon) ;
            lon:units = "degrees_east" ;
    float hybrid(hybrid) ;
            hybrid:units = "sigma" ;
            hybrid:long_name = "Hybrid level" ;
            hybrid:positive = "down" ;
            hybrid:standard_name = "atmosphere_hybrid_sigma_pressure_coordinate" ;
            hybrid:formula = "p(time,level,lat,lon) = ap(level) + b(level)*ps(time,lat,lon)" ;
            hybrid:formula_terms = "ap: hyam b: hybm ps: Surface_pressure_surface" ;
    int time(time) ;
            time:units = "Hour since 2012-10-15T12:00:00.000Z" ;
            time:standard_name = "time" ;
    int ens0(ens0) ;
            ens0:_CoordinateAxisType = "Ensemble" ;

    double hyam(hybrid) ;
            hyam:long_name = "hybrid A coefficient at layer midpoints" ;
            hyam:units = "Pa" ;
    double hybm(hybrid) ;
            hybm:long_name = "hybrid B coefficient at layer midpoints" ;
            hybm:units = "1" ;

    float Temperature_hybrid(time, ens0, hybrid, lat, lon) ;
            Temperature_hybrid:long_name = "Temperature @ Hybrid level" ;
            Temperature_hybrid:units = "K" ;
    float Specific_humidity_hybrid(time, ens0, hybrid, lat, lon) ;
            Specific_humidity_hybrid:long_name = "Specific humidity @ Hybrid level" ;
            Specific_humidity_hybrid:units = "kg/kg" ;

    ...
}

Note the following:

Met.3D requires the surface pressure field as well as the ak/bk coefficients to reconstruct the 3D pressure field. The names of these variables are specified in the "hybrid:formula_terms" attribute. Make sure that this attribute lists the correct variable names. A mismatch is a common source for errors - Met.3D will not be able to find the required fields!

Gridded data in GRIB format

Met.3D provides support for GRIB files written by ECMWF’s ecCodes library (as output, e.g. by the ECMWF MARS archive or written by Metview).

In the GUI, or in your pipeline.cfg file (see above), simply change the fileFormat entry to ECMWF_GRIB:

1\fileFormat=ECMWF_GRIB

Note the following:

GRIB messages may be arbitrarily distributed over different files that match the specified fileFilter.
Support is currently only provided for a horizontally regular longitude/latitude grid (GRIB gridType = regular_ll) and either pressure (GRIB typeOfLevel = isobaricInhPa) or hybrid sigma-pressure levels (GRIB typeOfLevel = hybrid) in the vertical.
Hybrid sigma-pressure model levels additionally require the surface pressure field (GRIB shortName = sp) or its logarithm (GRIB shortName = lnsp) to be present in the dataset.
Analysis (GRIB dataType = an), forecast (GRIB dataType = fc) and perturbed/control (i.e., ensemble forecast; GRIB dataType = pf/cf) messages are interpreted.
For 3D fields, a consistent consecutive list of vertical levels must be present for all time steps and ensemble members of a dataset. Levels need not be complete (i.e., there can be missing levels at the top or bottom) but the same levels need to be provided for all data fields in a dataset.
For a given dataset, all GRIB messages must have the same horizontal extent.

Unlike NetCDF files, GRIB files do not contain global headers that summarize the information contained in the file. Hence, to know which data is stored in a file the entire file needs to be read. This, of course, is inefficient when the file is accessed multiple times from different Met.3D sessions. Met.3D hence creates index files that store such a "global header" when first accessing GRIB files. You will find these files along with the GRIB files in the directory in which the GRIB files are stored. Note that your user ID hence requires write access to the directory in which the GRIB files are stored. If you don't have write access, we recommend placing symlinks to the files in a directory to which you have access and in which the index files can be stored. When you are first loading GRIB files, creation of index files can take some time!