Averaging over dimensions of the dataset

The average over dimensions operation makes use of clisops.core.average to process the datasets and to set the output type and the output file names.

It is possible to average over none or any number of time, longitude, latitude or level dimensions in the dataset.

[1]:

# Initialize the testing data
import clisops.utils.testing as clite

Stratus = clite.stratus(
    repo=clite.XCLIM_TEST_DATA_REPO_URL, branch=clite.XCLIM_TEST_DATA_VERSION, cache_dir=clite.XCLIM_TEST_DATA_CACHE_DIR
)

# fetch files locally or from GitHub
tas_files = [
    Stratus.fetch("cmip5/tas_Amon_HadGEM2-ES_rcp85_r1i1p1_200512-203011.nc"),
    Stratus.fetch("cmip5/tas_Amon_HadGEM2-ES_rcp85_r1i1p1_203012-205511.nc"),
    Stratus.fetch("cmip5/tas_Amon_HadGEM2-ES_rcp85_r1i1p1_205512-208011.nc"),
]

o3_file = Stratus.fetch("cmip6/o3_Amon_GFDL-ESM4_historical_r1i1p1f1_gr1_185001-194912.nc")

Downloading file 'cmip5/tas_Amon_HadGEM2-ES_rcp85_r1i1p1_200512-203011.nc' from 'https://raw.githubusercontent.com/Ouranosinc/xclim-testdata/v2024.8.23/data/cmip5/tas_Amon_HadGEM2-ES_rcp85_r1i1p1_200512-203011.nc' to '/home/docs/.cache/xclim-testdata/v2024.8.23'.
Downloading file 'cmip5/tas_Amon_HadGEM2-ES_rcp85_r1i1p1_203012-205511.nc' from 'https://raw.githubusercontent.com/Ouranosinc/xclim-testdata/v2024.8.23/data/cmip5/tas_Amon_HadGEM2-ES_rcp85_r1i1p1_203012-205511.nc' to '/home/docs/.cache/xclim-testdata/v2024.8.23'.
Downloading file 'cmip5/tas_Amon_HadGEM2-ES_rcp85_r1i1p1_205512-208011.nc' from 'https://raw.githubusercontent.com/Ouranosinc/xclim-testdata/v2024.8.23/data/cmip5/tas_Amon_HadGEM2-ES_rcp85_r1i1p1_205512-208011.nc' to '/home/docs/.cache/xclim-testdata/v2024.8.23'.
Downloading file 'cmip6/o3_Amon_GFDL-ESM4_historical_r1i1p1f1_gr1_185001-194912.nc' from 'https://raw.githubusercontent.com/Ouranosinc/xclim-testdata/v2024.8.23/data/cmip6/o3_Amon_GFDL-ESM4_historical_r1i1p1f1_gr1_185001-194912.nc' to '/home/docs/.cache/xclim-testdata/v2024.8.23'.

Parameters

Parameters taken by the average_over_dims are below:

ds: Union[xr.Dataset, str]
dims : Optional[Union[Tuple[str], DimensionParameter]]
  The dimensions over which to apply the average. If None, none of the dimensions are averaged over. Dimensions
  must be one of ["time", "level", "latitude", "longitude"].
ignore_undetected_dims: bool
  If the dimensions specified are not found in the dataset, an Exception will be raised if set to True.
  If False, an exception will not be raised and the other dimensions will be averaged over. Default = False
output_dir: Optional[Union[str, Path]] = None
output_type: {"netcdf", "nc", "zarr", "xarray"}
split_method: {"time:auto"}
file_namer: {"standard", "simple"}

The output is a list containing the outputs in the format selected.

[2]:

import xarray as xr

from clisops.exceptions import InvalidParameterValue
from clisops.ops.average import average_over_dims

[3]:

ds = xr.open_mfdataset(tas_files, decode_times=xr.coders.CFDatetimeCoder(use_cftime=True), combine="by_coords")

ds

/tmp/ipykernel_2215/1416113405.py:1: FutureWarning: In a future version of xarray the default value for data_vars will change from data_vars='all' to data_vars=None. This is likely to lead to different results when multiple datasets have matching variables with overlapping values. To opt in to new defaults and get rid of these warnings now use `set_options(use_new_combine_kwarg_defaults=True) or set data_vars explicitly.
  ds = xr.open_mfdataset(tas_files, decode_times=xr.coders.CFDatetimeCoder(use_cftime=True), combine="by_coords")

[3]:

<xarray.Dataset> Size: 94kB
Dimensions:    (time: 900, lat: 2, bnds: 2, lon: 2)
Coordinates:
  * time       (time) object 7kB 2005-12-16 00:00:00 ... 2080-11-16 00:00:00
  * lat        (lat) float64 16B -90.0 35.0
  * lon        (lon) float64 16B 0.0 187.5
    height     float64 8B 1.5
Dimensions without coordinates: bnds
Data variables:
    lat_bnds   (time, lat, bnds) float64 29kB dask.array<chunksize=(300, 2, 2), meta=np.ndarray>
    lon_bnds   (time, lon, bnds) float64 29kB dask.array<chunksize=(300, 2, 2), meta=np.ndarray>
    tas        (time, lat, lon) float32 14kB dask.array<chunksize=(300, 2, 2), meta=np.ndarray>
    time_bnds  (time, bnds) object 14kB dask.array<chunksize=(300, 2), meta=np.ndarray>
Attributes: (12/29)
    institution:            Met Office Hadley Centre, Fitzroy Road, Exeter, D...
    institute_id:           MOHC
    experiment_id:          rcp85
    source:                 HadGEM2-ES (2009) atmosphere: HadGAM2 (N96L38); o...
    model_id:               HadGEM2-ES
    forcing:                GHG, SA, Oz, LU, Sl, Vl, BC, OC, (GHG = CO2, N2O,...
    ...                     ...
    title:                  HadGEM2-ES model output prepared for CMIP5 RCP8.5
    parent_experiment:      historical
    modeling_realm:         atmos
    realization:            1
    cmor_version:           2.5.0
    NCO:                    4.7.3

Average over one dimension

[4]:

result = average_over_dims(ds, dims=["time"], ignore_undetected_dims=False, output_type="xarray")

result[0]

[4]:

<xarray.Dataset> Size: 136B
Dimensions:    (lat: 2, bnds: 2, lon: 2)
Coordinates:
  * lat        (lat) float64 16B -90.0 35.0
  * lon        (lon) float64 16B 0.0 187.5
    height     float64 8B 1.5
Dimensions without coordinates: bnds
Data variables:
    lat_bnds   (lat, bnds) float64 32B dask.array<chunksize=(2, 2), meta=np.ndarray>
    lon_bnds   (lon, bnds) float64 32B dask.array<chunksize=(2, 2), meta=np.ndarray>
    tas        (lat, lon) float32 16B dask.array<chunksize=(2, 2), meta=np.ndarray>
    time_bnds  (bnds) object 16B dask.array<chunksize=(2,), meta=np.ndarray>
Attributes: (12/29)
    institution:            Met Office Hadley Centre, Fitzroy Road, Exeter, D...
    institute_id:           MOHC
    experiment_id:          rcp85
    source:                 HadGEM2-ES (2009) atmosphere: HadGAM2 (N96L38); o...
    model_id:               HadGEM2-ES
    forcing:                GHG, SA, Oz, LU, Sl, Vl, BC, OC, (GHG = CO2, N2O,...
    ...                     ...
    title:                  HadGEM2-ES model output prepared for CMIP5 RCP8.5
    parent_experiment:      historical
    modeling_realm:         atmos
    realization:            1
    cmor_version:           2.5.0
    NCO:                    4.7.3

As you can see in the output dataset, time has been averaged over and has been removed.

Average over two dimensions

Averaging over two dimensions is just as simple as averaging over one. The dimensions to be averaged over should be passed in as a sequence.

[5]:

result = average_over_dims(ds, dims=["time", "latitude"], ignore_undetected_dims=False, output_type="xarray")

result[0]

[5]:

<xarray.Dataset> Size: 96B
Dimensions:    (bnds: 2, lon: 2)
Coordinates:
  * lon        (lon) float64 16B 0.0 187.5
    height     float64 8B 1.5
Dimensions without coordinates: bnds
Data variables:
    lat_bnds   (bnds) float64 16B dask.array<chunksize=(2,), meta=np.ndarray>
    lon_bnds   (lon, bnds) float64 32B dask.array<chunksize=(2, 2), meta=np.ndarray>
    tas        (lon) float32 8B dask.array<chunksize=(2,), meta=np.ndarray>
    time_bnds  (bnds) object 16B dask.array<chunksize=(2,), meta=np.ndarray>
Attributes: (12/29)
    institution:            Met Office Hadley Centre, Fitzroy Road, Exeter, D...
    institute_id:           MOHC
    experiment_id:          rcp85
    source:                 HadGEM2-ES (2009) atmosphere: HadGAM2 (N96L38); o...
    model_id:               HadGEM2-ES
    forcing:                GHG, SA, Oz, LU, Sl, Vl, BC, OC, (GHG = CO2, N2O,...
    ...                     ...
    title:                  HadGEM2-ES model output prepared for CMIP5 RCP8.5
    parent_experiment:      historical
    modeling_realm:         atmos
    realization:            1
    cmor_version:           2.5.0
    NCO:                    4.7.3

In this case both the time and latitude dimensions have been removed.

Allowed dimensions

It is only possible to average over longtiude, latitude, level and time. If a different dimension is provided to average over an error will be raised.

[6]:

try:
    average_over_dims(
        ds,
        dims=["incorrect_dim"],
        ignore_undetected_dims=False,
        output_type="xarray",
    )
except InvalidParameterValue as exc:
    print(exc)

Dimensions for averaging must be one of ['time', 'level', 'latitude', 'longitude', 'realization']

Dimensions not found

In the case where a dimension has been selected for averaging but it doesn’t exist in the dataset, there are 2 options.

To raise an exception when the dimension doesn’t exist, set ignore_undetected_dims = False

[7]:

try:
    average_over_dims(
        ds,
        dims=["level", "time"],
        ignore_undetected_dims=False,
        output_type="xarray",
    )
except InvalidParameterValue as exc:
    print(exc)

Requested dimensions were not found in input dataset: {'level'}.

To ignore when the dimension doesn’t exist, and average over any other requested dimensions anyway, set ignore_undetected_dims = True

[8]:

result = average_over_dims(
    ds,
    dims=["level", "time"],
    ignore_undetected_dims=True,
    output_type="xarray",
)
result[0]

[8]:

<xarray.Dataset> Size: 136B
Dimensions:    (lat: 2, bnds: 2, lon: 2)
Coordinates:
  * lat        (lat) float64 16B -90.0 35.0
  * lon        (lon) float64 16B 0.0 187.5
    height     float64 8B 1.5
Dimensions without coordinates: bnds
Data variables:
    lat_bnds   (lat, bnds) float64 32B dask.array<chunksize=(2, 2), meta=np.ndarray>
    lon_bnds   (lon, bnds) float64 32B dask.array<chunksize=(2, 2), meta=np.ndarray>
    tas        (lat, lon) float32 16B dask.array<chunksize=(2, 2), meta=np.ndarray>
    time_bnds  (bnds) object 16B dask.array<chunksize=(2,), meta=np.ndarray>
Attributes: (12/29)
    institution:            Met Office Hadley Centre, Fitzroy Road, Exeter, D...
    institute_id:           MOHC
    experiment_id:          rcp85
    source:                 HadGEM2-ES (2009) atmosphere: HadGAM2 (N96L38); o...
    model_id:               HadGEM2-ES
    forcing:                GHG, SA, Oz, LU, Sl, Vl, BC, OC, (GHG = CO2, N2O,...
    ...                     ...
    title:                  HadGEM2-ES model output prepared for CMIP5 RCP8.5
    parent_experiment:      historical
    modeling_realm:         atmos
    realization:            1
    cmor_version:           2.5.0
    NCO:                    4.7.3

In the case above, a level dimension did not exist, but this was ignored and time was averaged over anyway.

No dimensions supplied

If no dimensions are supplied, no averaging will be applied and the original dataset will be returned.

[9]:

result = average_over_dims(ds, dims=None, ignore_undetected_dims=False, output_type="xarray")

result[0]

---------------------------------------------------------------------------
InvalidParameterValue                     Traceback (most recent call last)
Cell In[9], line 1
----> 1 result = average_over_dims(ds, dims=None, ignore_undetected_dims=False, output_type="xarray")
      2
      3 result[0]

File ~/checkouts/readthedocs.org/user_builds/clisops/conda/stable/lib/python3.12/site-packages/clisops/ops/average.py:99, in average_over_dims(ds, dims, ignore_undetected_dims, output_dir, output_type, split_method, file_namer)
     58 """
     59 Calculate an average over given dimensions.
     60
   (...)     96 | file_namer: "standard"
     97 """
     98 op = Average(**locals())
---> 99 return op.process()

File ~/checkouts/readthedocs.org/user_builds/clisops/conda/stable/lib/python3.12/site-packages/clisops/ops/base_operation.py:244, in Operation.process(self)
    240 namer = self._get_file_namer()
    242 # Process the xarray Dataset - this will (usually) be lazily evaluated so
    243 # no actual data will be read
--> 244 processed_ds = self._calculate()
    246 # remove fill values from lat/lon/time if required
    247 processed_ds = self._remove_redundant_fill_values(processed_ds)

File ~/checkouts/readthedocs.org/user_builds/clisops/conda/stable/lib/python3.12/site-packages/clisops/ops/average.py:40, in Average._calculate(self)
     39 def _calculate(self):
---> 40     avg_ds = average.average_over_dims(
     41         self.ds,
     42         self.params.get("dims", None),
     43         self.params.get("ignore_undetected_dims", None),
     44     )
     46     return avg_ds

File ~/checkouts/readthedocs.org/user_builds/clisops/conda/stable/lib/python3.12/site-packages/clisops/core/average.py:172, in average_over_dims(ds, dims, ignore_undetected_dims)
    140 """
    141 Average a DataArray or Dataset over the dimensions specified.
    142
   (...)    169     prAvg = average_over_dims(pr, dims=["latitude", "longitude"], ignore_undetected_dims=True)
    170 """
    171 if not dims:
--> 172     raise InvalidParameterValue("At least one dimension for averaging must be provided")
    174 if not set(dims).issubset(set(known_coord_types)):
    175     raise InvalidParameterValue(f"Dimensions for averaging must be one of {known_coord_types}")

InvalidParameterValue: At least one dimension for averaging must be provided