THE NERC MST RADAR FACILITY AT ABERYSTWYTH

NERC MSTRF: netCDF

The netCDF (network Common Data Form) file format is designed for the storage of array-oriented data, e.g. time series, altitude profiles, or time-altitude data. In order to read and write netCDF files, you will need to download and install the following freely-available software from the Unidata website: Users of netCDF data do not need to know anything about the internal file structure, which is handled by the netcdf software. Instead, any file can be accessed using a standard set of routines. This makes adapting to new files very easy. However, it is netCDF's ability to store unlimited metadata (i.e. information about the data) which makes it particularly important from a data preservation point of view. A number of user communities have defined their own "Conventions" which define the metadata standards in a discipline-specific way. For the atmospheric sciences, this is the Climate and Forecast (CF) Metadata Convention.

Creators of netCDF files should read the official netCDF and Climate and Forecast (CF) Metadata Convention documents. The following page gives sufficient details to allow someone to read a file.

The contents of any netCDF file can be viewed by typing the following command (on a Unix/Linux system):

ncdump -h netcdf_file_path

Omitting the -h option causes the data values, as well as the file structure, to be shown.

Data types
All of the fields within a netCDF file are of one of the following data types (the characters shown in brackets indicate how the "ncdump" command indicates the data type of an attribute):
BYTE (b) 8-bit signed integer
CHAR ("") 8-bit unsigned integer (strings are techically arrays of type CHAR)
SHORT (s) 16-bit signed integer
INT (i) 32-bit signed integer
FLOAT (f) 32-bit floating point
DOUBLE (d) 64-bit floating point

The new netCDF-4 storage format files additionally allow the following data types:
USHORT 16-bit unsigned integer
UINT 32-bit unsigned integer
INT64 64-bit signed integer
UINT64 64-bit unsigned integer
STRING variable length character string
BOOL (8-bit) Boolean

Global attributes
Global attributes are metdata fields which describe the file/data as a whole. File creators may add any Global Attributes that they wish and these can be of any data type. The CF convention requires that the following CHAR data type global attributes are included:
Conventions
Describes the metadata conventions used, i.e. "CF-1.3" at the time of writing
title
A succinct description of the contents of the file
institution
Details of the institution(s) responsible for acquiring, processing and storing the data
source
The instrument from which the data were recorded
history
This provides an audit trail of how and when the data were acquired, modified and recorded
references
Sources of documentation concerning the instrument, measurement technique and data processing
comment
Miscellaneous information about the data, how they have been processed, quality characteristics etc.

Dimensions
The netcdf software provides routines for determining the dimensions (e.g. time, altitude, latitude, and longitude) used by the variables, i.e. the data arrays. These are shown in red and within brackets below. Variables can be associated with one, several, or no dimensions at all. The purpose of variables with no dimensions will be clarified shortly. Each dimension has an associated one-dimensional coordinate variable, of the same name, which gives the permitted values along that dimension.

If any or all of the dimensions of a variable have the interpretations of "date or time" (T), "height or depth" (Z), "latitude" (Y), or "longitude" (X), the CF Convention recommends (but does not require) that they are used in the relative order: T, then Z, then Y, then X. All other dimensions should, whenever possible, be placed to the left of the spatio-temporal dimensions.

Refer to the coordinates variable attribute below for details of how to record latitude, longitude, and altitude for a single instrument.


Variables and their attributes
There are a number of ways to represent values of time. For files which store all the data for a given instrument and for given day, it is common to quote time in seconds since midnight (i.e. 00:00:00 UTC) of the day in question.

In addition to being associated with a data type and none, one or several dimensions, a variable can (and should) have a number of attributes associated with it. These tend to be of type CHAR but can be of any other data type. As in the case of Global Attributes, a file creator can add as many attributes to a variable as they wish. The CF Convention specifies that the following should be included wherever possible:
standard_name
The file creator is free to choose any name they wish for a variable. Where the variable relates to a commonly-used physical (or model) parameter, which has an unambiguous definition, the variable should have a CHAR type attribute standard_name, which contains a name from the standard name table. Standard names are only added to this table after a period of consultation within the discipline-specific and wider CF community. Consequently, not all parameters have a standard name. This attribute should only be added where a standard name exists. Technically dB values are dimensionless as they relate to the log of a ratio.
long_name
This is a CHAR type attribute which gives a description of variable in a format which might be used, for example, as a label on a plot. It is not necessary to include one where a standard_name attribute exists, but is essential otherwise.
units
Where possible, the value of this CHAR type attribute should be a string which can be recognised by UNIDATA's Udunits package, i.e. it is found in the udunits.dat file. Although standard names have canonical units associated with them, it is permissble to use alternatives. For example, the canonical units associated with standard_name "relative_humidity" are "1", indicating that this is a dimensionless quantity whose values range between 0.0 and 1.0. However, it is more common to refer to relative humidities as percentages between 0.0 and 100.0, in which case the value of units should be "%". Note that "dB" units are used for many radar data products, although this string does not appear in the udunits.dat file.
missing_value
This is the value which indicates a missing datum. It shares the same data type as the variable itself. This attribute is not used for variables such as reliability flags.
_FillValue
This attribute was previously used instead of missing_value. For NERC MST Radar Facility files, both attributes are given for backwards compatability. They share the same value.

The following variable attributes are common but optional:
valid_range
This is an array, of the same data type as the variable, of length 2. It contains values of valid_min and valid_max, repectively, which define the permissible range of data values. The value of missing_value should be outside of this range. The values of valid_min and valid_max can alternatively be set as two separate variable attributes.
flag_values
If an intereger-type variable is a flag, this attribute contains an array (of the same data type as the variable) of possible flag values. The corresponding meanings of these values are described by space-separated strings in the CHAR data type flag_meanings attribute.
axis
If a dimension has the interpretation of "date or time" (T), "height or depth" (Z), "latitude" (Y), or "longitude" (X), the CF Convention recommends that a CHAR data type axis attribute be given to the corresponding coordinate variable. This can hold values of "T" "Z", "Y" or "X".
coordinates
In the case of data for a fixed location and/or time, some or all of the "date or time" (T), "height or depth" (Z), "latitude" (Y), or "longitude" coordinates can be defined by a dimensionless variable which holds a single value. Parameter values are associated with this location/time information through the coordinates attribute of that parameter's variable. This a CHAR data type value which contains the names of appropriate coordinate variables separated by spaces, e.g. "latitude longitude altitude" or "time".
comment
This non-standard variable attribute is used widely for NERC MST Radar Facility data files as an aid to self-description. It is modelled on the global attribute of the same name described above.

The following non-standard variable attributes are used in some of uthe NERC MST Radar Facility data files:
estimated_accuracy
An attribute of the same data type as the variable which gives an estimate of the data accuracy
accuracy_details
A CHAR data type attribute which gives details of the data accuracy
resolution
An attribute of the same data type as the variable which indicates the resolution of the recorded data
comment
A CHAR data type attribute which is analagous to the CF Convention global attribute of the same name. It is used for storing general information about a variable.


A quick-look at the contents of a netCDF file
The output produced by the command:
ncdump -h netcdf_file_path

(for an imaginary netCDF file) looks like the following: 
dimensions:
        time = length_of_time_dimension ;
	altitude = length_of_altitude_dimension ;
variables:
        float time(time) ;
                time:standard_name = "time" ;
                time:long_name = "UTC" ;
                time:units = "seconds since YYYY-MM-DD 00:00:00 +00:00" ;
                time:axis = "T" ;
                time:comment = "No comment" ;
        float altitude(altitude) ;
                altitude:standard_name = "altitude" ;
                altitude:long_name = "Altitude above mean sea level" ;
                altitude:units = "m" ;
                altitude:axis = "Z" ;
                altitude:comment = "No comment" ;
        float latitude ;
                latitude:standard_name = "latitude" ;
                latitude:long_name = "Instrument latitude" ;
                latitude:units = "degrees_north" ;
                latitude:axis = "Y" ;
        float longitude ;
                longitude:standard_name = "longitude" ;
                longitude:long_name = "Instrument longitude" ;
                longitude:units = "degrees_east" ;
                longitude:axis = "X" ;
        float variable_1(time, altitude) ;
                variable_1:standard_name = "?" ;
                variable_1:long_name = "Long name for variable 1" ;
                variable_1:units = "m s-1" ;
                variable_1:estimated_accuracy = 2.5f ;
                variable_1:missing_value = -9999.f ;
                variable_1:_FillValue = -9999.f ;
                variable_1:coordinates = "latitude longitude" ;
		variable_1:comment = "Blah blah blah"
        float variable_2(time, altitude) ;
                variable_2:standard_name = "?" ;
                variable_2:long_name = "Long name for variable 2" ;
                variable_2:units = "hPa" ;
                variable_2:estimated_accuracy = 1.0f ;
                variable_2:missing_value = -9999.f ;
                variable_2:_FillValue = -9999.f ;
                variable_2:coordinates = "latitude longitude" ;
		variable_2:comment = "Blah blah blah"

// global attributes:
                :Conventions = "CF-1.3" ;
                :title = "Data from an imaginmary profiling instrument" ;
                :institution = "Data recorded by the NERC MST Radar Facility" ;
                :source = "The imaginary instrument" ;
		:history = "File created YYYY-MM-DD hh:mm:ss +00:00 on machine fair" ;
                :references = "http://mst.nerc.ac.uk" ;
                :comment = "Blah blah blah" ;
                :data_year = 2008s ;
                :data_month = 9s ;
                :data_day = 3s ;
NERC MST Radar Facility files which are currently in netCDF format are:
MST Radar Cartesian data - v3
MST Radar radial data - v3
MST Radar spectral data - since February 2007
Vaisala WXT510 surface met (ptu) data
Vaisala WXT510 surface met (precip) data
Vaisala WXT510 surface wind data

Internal Links:
Return to the top of the page
External Links:
The netCDF homepage on the Unidata website
Metadata conventions for netCDF
The Climate and Forecast (CF) Metadata Convention
The UNIDATA Udunits package
Page maintained by David Hooper.