`clisops` regridding functionalities - powered by `xesmf`

The regridding functionalities of clisops consist of the regridding operator/function regrid in clisops.ops, allowing one-line remapping of xarray.Datasets or xarray.DataArrays, while orchestrating the use of classes and functions in clisops.core: - the Grid and Weights classes, to check and pre-process input as well as output grids and to generate the remapping weights - a regrid function, performing the remapping by applying the generated weights on the input data

For the weight generation and the regridding, the xESMF Regridder class is used, which itself allows an easy application of many of the remapping functionalities of ESMF/ESMPy.

[1]:

# Imports

%matplotlib inline
import matplotlib.pyplot as plt
import cartopy.crs as ccrs
import psyplot.project as psy
import numpy as np
import xarray as xr
import cf_xarray as cfxr

from pathlib import Path
from git import Repo
# Set required environment variable for ESMPy
import os
os.environ['ESMFMKFILE'] = str(Path(os.__file__).parent.parent / 'esmf.mk')
import xesmf as xe

import clisops as cl # atm. the regrid-main-martin branch of clisops
import clisops.ops as clops
import clisops.core as clore
from clisops.utils import dataset_utils
from roocs_grids import get_grid_file, grid_dict, grid_annotations

print(f"Using xarray in version {xr.__version__}")
print(f"Using cf_xarray in version {cfxr.__version__}")
print(f"Using xESMF in version {xe.__version__}")
print(f"Using clisops in version {cl.__version__}")

xr.set_options(display_style='html')

## Turn off warnings?
import warnings
warnings.simplefilter("ignore")

Matplotlib is building the font cache; this may take a moment.

ERROR 1: PROJ: proj_create_from_database: Open of /home/docs/checkouts/readthedocs.org/user_builds/clisops/conda/latest/share/proj failed

Using xarray in version 2023.2.0
Using cf_xarray in version 0.9.0
Using xESMF in version 0.8.5
Using clisops in version 0.13.0

[2]:

# Initialize test data

# Initialize mini-esgf-data
MINIESGF_URL="https://github.com/roocs/mini-esgf-data"
branch = "master"
MINIESGF = Path(Path.home(),".mini-esgf-data", branch)

# Retrieve mini-esgf test data
if not os.path.isdir(MINIESGF):
    repo = Repo.clone_from(MINIESGF_URL, MINIESGF)
    repo.git.checkout(branch)
else:
    repo = Repo(MINIESGF)
    repo.git.checkout(branch)
    repo.remotes[0].pull()

MINIESGF=Path(MINIESGF,"test_data")

`clisops.core.Weights`

Create regridding weights to regrid between two grids. Supported are the following of xESMF’s remapping methods: * nearest_s2d * bilinear * conservative * patch

Create 2-degree target grid

[34]:

grid_2deg = clore.Grid(grid_id="2deg_lsm", compute_bounds=True)
grid_2deg

[34]:

clisops 90x180_cells_grid
Lat x Lon:        90 x 180
Gridcells:        16200
Format:           CF
Type:             regular_lat_lon
Extent:           global
Source:           Predefined_2deg_lsm
Bounds?           True
Collapsed cells? False
Duplicated cells? False
Permanent Mask:   None
md5 hash:         438f79b4376004d360a56fa2c0abb9f0

Create conservative remapping weights using the `clisops.core.Weights` class

grid_in and grid_out are Grid objects

[35]:

%time weights = clore.Weights(grid_in = grido, grid_out = grid_2deg, method="conservative")

CPU times: user 10.4 s, sys: 333 ms, total: 10.8 s
Wall time: 10.8 s

Local weights cache

Weights are cached on disk and do not have to be created more than once. The default cache directory is platform-dependent and set via the package platformdirs. For Linux it is '/home/my_user/.local/share/clisops/weights_dir' and can optionally be adjusted:

permanently by modifying the parameter grid_weights: local_weights_dir in the roocs.ini configuration file that can be found in the clisops installation directory
or temporarily via:

from clisops import core as clore
clore.weights_cache_init("/dir/for/weights/cache")

[36]:

from clisops.core.regrid import CONFIG
print(CONFIG["clisops:grid_weights"]["local_weights_dir"])

/home/docs/.local/share/clisops/grid_weights

[37]:

!ls -sh {CONFIG["clisops:grid_weights"]["local_weights_dir"]}

total 32M
904K grid_05603f6f046b266ba6bc164c5b3209bf.nc
 32K grid_20d38455170b1db562edb4f5e06d3cc6.nc
 24K grid_438f79b4376004d360a56fa2c0abb9f0.nc
8.6M grid_a55c642a80a2b5a8adcc71a9db4fc5f5.nc
 24K grid_cbfbd94fc4feee0c6b34841124e4adf8.nc
 20K grid_d4ebf553238ebf865140bded4c8e73b9.nc
 40K grid_e2c8b52440eb2392c600702aa1c532ef.nc
7.3M grid_ec7db0ee9226e085bfe8207b8de0672d.nc
4.0K weights_05603f6f046b266ba6bc164c5b3209bf_cbfbd94fc4feee0c6b34841124e4adf8_unperi_no-degen_bilinear.json
2.0M weights_05603f6f046b266ba6bc164c5b3209bf_cbfbd94fc4feee0c6b34841124e4adf8_unperi_no-degen_bilinear.nc
4.0K weights_20d38455170b1db562edb4f5e06d3cc6_d4ebf553238ebf865140bded4c8e73b9_peri_no-degen_bilinear.json
1.5M weights_20d38455170b1db562edb4f5e06d3cc6_d4ebf553238ebf865140bded4c8e73b9_peri_no-degen_bilinear.nc
4.0K weights_a55c642a80a2b5a8adcc71a9db4fc5f5_438f79b4376004d360a56fa2c0abb9f0_peri_skip-degen_conservative.json
7.6M weights_a55c642a80a2b5a8adcc71a9db4fc5f5_438f79b4376004d360a56fa2c0abb9f0_peri_skip-degen_conservative.nc
4.0K weights_ec7db0ee9226e085bfe8207b8de0672d_e2c8b52440eb2392c600702aa1c532ef_peri_no-degen_nearest_s2d.json
3.6M weights_ec7db0ee9226e085bfe8207b8de0672d_e2c8b52440eb2392c600702aa1c532ef_peri_no-degen_nearest_s2d.nc

[38]:

!cat {CONFIG["clisops:grid_weights"]["local_weights_dir"]}/weights_*_conservative.json

{
    "def_filename": "conservative_404x802_90x180.nc",
    "filename": "weights_a55c642a80a2b5a8adcc71a9db4fc5f5_438f79b4376004d360a56fa2c0abb9f0_peri_skip-degen_conservative.nc",
    "format": "xESMF",
    "ignore_degenerate": "True",
    "method": "conservative",
    "periodic": "True",
    "source_extent": "global",
    "source_format": "CF",
    "source_lat": "latitude",
    "source_lat_bnds": "vertices_latitude",
    "source_lon": "longitude",
    "source_lon_bnds": "vertices_longitude",
    "source_ncells": 324008,
    "source_nlat": 404,
    "source_nlon": 802,
    "source_source": "/home/docs/.mini-esgf-data/master/test_data/badc/cmip6/data/CMIP6/CMIP/MPI-M/MPI-ESM1-2-HR/historical/r1i1p1f1/Omon/tos/gn/v20190710/tos_Omon_MPI-ESM1-2-HR_historical_r1i1p1f1_gn_185001.nc",
    "source_tracking_id": "hdl:21.14100/ccd8388f-4f80-4a1e-ba47-66fe65bbeece",
    "source_type": "curvilinear",
    "source_uid": "a55c642a80a2b5a8adcc71a9db4fc5f5",
    "target_extent": "global",
    "target_format": "CF",
    "target_lat": "lat",
    "target_lat_bnds": "lat_bnds",
    "target_lon": "lon",
    "target_lon_bnds": "lon_bnds",
    "target_ncells": 16200,
    "target_nlat": 90,
    "target_nlon": 180,
    "target_source": "/home/docs/checkouts/readthedocs.org/user_builds/clisops/conda/latest/lib/python3.11/site-packages/roocs_grids/grids/land_sea_mask_2degree.nc4",
    "target_tracking_id": "",
    "target_type": "regular_lat_lon",
    "target_uid": "438f79b4376004d360a56fa2c0abb9f0",
    "tool": "xESMF_v0.8.5",
    "uid": "a55c642a80a2b5a8adcc71a9db4fc5f5_438f79b4376004d360a56fa2c0abb9f0_peri_skip-degen_conservative"
}

Now the weights will be read directly from the cache

[39]:

%time weights = clore.Weights(grid_in = grido, grid_out = grid_2deg, method="conservative")

CPU times: user 185 ms, sys: 56 ms, total: 241 ms
Wall time: 241 ms

The weights cache can be flushed, which removes all weight and grid files as well as the json files holding the metadata. To see what would be removed, one can use the dryrun=True parameter. To re-initialize the weights cache in a different directory, one can use the weights_dir_init="/new/dir/for/weights/cache" parameter. Even when re-initializing the weights cache under a new path, using clore.weights_cache_flush, no directory is getting removed, only above listed files. When dryrun is not set, the files that are getting deleted can be displayed with verbose=True.

[40]:

clore.weights_cache_flush(dryrun=True)

Flushing the clisops weights cache ('/home/docs/.local/share/clisops/grid_weights') would remove:
 - /home/docs/.local/share/clisops/grid_weights/weights_20d38455170b1db562edb4f5e06d3cc6_d4ebf553238ebf865140bded4c8e73b9_peri_no-degen_bilinear.json
 - /home/docs/.local/share/clisops/grid_weights/weights_05603f6f046b266ba6bc164c5b3209bf_cbfbd94fc4feee0c6b34841124e4adf8_unperi_no-degen_bilinear.json
 - /home/docs/.local/share/clisops/grid_weights/weights_ec7db0ee9226e085bfe8207b8de0672d_e2c8b52440eb2392c600702aa1c532ef_peri_no-degen_nearest_s2d.json
 - /home/docs/.local/share/clisops/grid_weights/weights_a55c642a80a2b5a8adcc71a9db4fc5f5_438f79b4376004d360a56fa2c0abb9f0_peri_skip-degen_conservative.json
 - /home/docs/.local/share/clisops/grid_weights/weights_20d38455170b1db562edb4f5e06d3cc6_d4ebf553238ebf865140bded4c8e73b9_peri_no-degen_bilinear.nc
 - /home/docs/.local/share/clisops/grid_weights/weights_a55c642a80a2b5a8adcc71a9db4fc5f5_438f79b4376004d360a56fa2c0abb9f0_peri_skip-degen_conservative.nc
 - /home/docs/.local/share/clisops/grid_weights/weights_05603f6f046b266ba6bc164c5b3209bf_cbfbd94fc4feee0c6b34841124e4adf8_unperi_no-degen_bilinear.nc
 - /home/docs/.local/share/clisops/grid_weights/weights_ec7db0ee9226e085bfe8207b8de0672d_e2c8b52440eb2392c600702aa1c532ef_peri_no-degen_nearest_s2d.nc
 - /home/docs/.local/share/clisops/grid_weights/grid_ec7db0ee9226e085bfe8207b8de0672d.nc
 - /home/docs/.local/share/clisops/grid_weights/grid_20d38455170b1db562edb4f5e06d3cc6.nc
 - /home/docs/.local/share/clisops/grid_weights/grid_e2c8b52440eb2392c600702aa1c532ef.nc
 - /home/docs/.local/share/clisops/grid_weights/grid_a55c642a80a2b5a8adcc71a9db4fc5f5.nc
 - /home/docs/.local/share/clisops/grid_weights/grid_d4ebf553238ebf865140bded4c8e73b9.nc
 - /home/docs/.local/share/clisops/grid_weights/grid_cbfbd94fc4feee0c6b34841124e4adf8.nc
 - /home/docs/.local/share/clisops/grid_weights/grid_438f79b4376004d360a56fa2c0abb9f0.nc
 - /home/docs/.local/share/clisops/grid_weights/grid_05603f6f046b266ba6bc164c5b3209bf.nc

[41]:

clore.weights_cache_flush(verbose=True)

Flushing the clisops weights cache ('/home/docs/.local/share/clisops/grid_weights'). Removing ...
 - /home/docs/.local/share/clisops/grid_weights/weights_20d38455170b1db562edb4f5e06d3cc6_d4ebf553238ebf865140bded4c8e73b9_peri_no-degen_bilinear.json
 - /home/docs/.local/share/clisops/grid_weights/weights_05603f6f046b266ba6bc164c5b3209bf_cbfbd94fc4feee0c6b34841124e4adf8_unperi_no-degen_bilinear.json
 - /home/docs/.local/share/clisops/grid_weights/weights_ec7db0ee9226e085bfe8207b8de0672d_e2c8b52440eb2392c600702aa1c532ef_peri_no-degen_nearest_s2d.json
 - /home/docs/.local/share/clisops/grid_weights/weights_a55c642a80a2b5a8adcc71a9db4fc5f5_438f79b4376004d360a56fa2c0abb9f0_peri_skip-degen_conservative.json
 - /home/docs/.local/share/clisops/grid_weights/weights_20d38455170b1db562edb4f5e06d3cc6_d4ebf553238ebf865140bded4c8e73b9_peri_no-degen_bilinear.nc
 - /home/docs/.local/share/clisops/grid_weights/weights_a55c642a80a2b5a8adcc71a9db4fc5f5_438f79b4376004d360a56fa2c0abb9f0_peri_skip-degen_conservative.nc
 - /home/docs/.local/share/clisops/grid_weights/weights_05603f6f046b266ba6bc164c5b3209bf_cbfbd94fc4feee0c6b34841124e4adf8_unperi_no-degen_bilinear.nc
 - /home/docs/.local/share/clisops/grid_weights/weights_ec7db0ee9226e085bfe8207b8de0672d_e2c8b52440eb2392c600702aa1c532ef_peri_no-degen_nearest_s2d.nc
 - /home/docs/.local/share/clisops/grid_weights/grid_ec7db0ee9226e085bfe8207b8de0672d.nc
 - /home/docs/.local/share/clisops/grid_weights/grid_20d38455170b1db562edb4f5e06d3cc6.nc
 - /home/docs/.local/share/clisops/grid_weights/grid_e2c8b52440eb2392c600702aa1c532ef.nc
 - /home/docs/.local/share/clisops/grid_weights/grid_a55c642a80a2b5a8adcc71a9db4fc5f5.nc
 - /home/docs/.local/share/clisops/grid_weights/grid_d4ebf553238ebf865140bded4c8e73b9.nc
 - /home/docs/.local/share/clisops/grid_weights/grid_cbfbd94fc4feee0c6b34841124e4adf8.nc
 - /home/docs/.local/share/clisops/grid_weights/grid_438f79b4376004d360a56fa2c0abb9f0.nc
 - /home/docs/.local/share/clisops/grid_weights/grid_05603f6f046b266ba6bc164c5b3209bf.nc
Initialized new weights cache at /home/docs/.local/share/clisops/grid_weights

`clisops.core.regrid`

This function allows to perform the eventual regridding and provides a resulting xarray.Dataset

def regrid(
    grid_in: Grid,
    grid_out: Grid,
    weights: Weights,
    adaptive_masking_threshold: Optional[float] = 0.5,
    keep_attrs: Optional[bool] = True,
):

grid_in and grid_out are Grid objects, weights is a Weights object.
adaptive_masking_threshold (AMT) A value within the [0., 1.] interval that defines the maximum RATIO of missing_values amongst the total number of data values contributing to the calculation of the target grid cell value. For a fraction [0., AMT[ of the contributing source data missing, the target grid cell will be set to missing_value, else, it will be re-normalized by the factor 1./(1.-RATIO). Thus, if AMT is set to 1, all source grid cells that contribute to a target grid cell must be missing in order for the target grid cell to be defined as missing itself. Values greater than 1 or less than 0 will cause adaptive masking to be turned off. This adaptive masking technique allows to reuse generated weights for differently masked data (e.g. land-sea masks or orographic masks that vary with depth / height).
keep_attrs can have the following settings:
- True : The resulting xarray.Dataset will have all attributes of grid_in.ds.attrs, despite attributes that have to be added and altered due to the new grid.
- False : The resulting xarray.Dataset will have no attributes despite attributes generated by the regridding process.
- "target" : The resulting xarray.Dataset will have all attributes of grid_out.ds.attrs, despite attributes generated by the regridding process. Not recommended.

In the following an example showing the function application and the effect of the adaptive masking.

[42]:

ds_out_amt0 = clore.regrid(grido, grid_2deg, weights, adaptive_masking_threshold=-1)

[43]:

ds_out_amt1 = clore.regrid(grido, grid_2deg, weights, adaptive_masking_threshold=0.5)

Plot the resulting data

[44]:

# Create panel plot of regridded data (global)
fig, axes = plt.subplots(ncols=2, nrows=1,
                         figsize=(18, 5), # global
                         subplot_kw={'projection': ccrs.PlateCarree()})

ds_out_amt0["tos"].isel(time=0).plot.pcolormesh(ax=axes[0], vmin=0, vmax=30, cmap="plasma")
axes[0].title.set_text("Target (2° regular lat-lon) - No adaptive masking")

ds_out_amt1["tos"].isel(time=0).plot.pcolormesh(ax=axes[1], vmin=0, vmax=30, cmap="plasma")
axes[1].title.set_text("Target (2° regular lat-lon) - Adaptive masking")

for axis in axes.flatten():
    axis.coastlines()
    axis.set_xlabel('lon')
    axis.set_ylabel('lat')

[45]:

# Create panel plot of regridded data (Japan)
fig, axes = plt.subplots(ncols=3, nrows=1,
                         figsize=(18, 4), # Japan
                         subplot_kw={'projection': ccrs.PlateCarree()})

grido.ds.tos.isel(time=0).plot.pcolormesh(ax=axes[0], x=grido.lon, y=grido.lat,
                                          vmin=0, vmax=30, cmap="plasma", shading="auto")
axes[0].title.set_text("Source - MPI-ESM1-2-HR MPIOM (TP04, ~0.4° resolution)")

ds_out_amt0["tos"].isel(time=0).plot.pcolormesh(ax=axes[1], vmin=0, vmax=30, cmap="plasma")
axes[1].title.set_text("Target - No adaptive masking")

ds_out_amt1["tos"].isel(time=0).plot.pcolormesh(ax=axes[2], vmin=0, vmax=30, cmap="plasma")
axes[2].title.set_text("Target - Adaptive masking")

for axis in axes.flatten():
    axis.coastlines()
    axis.set_xlabel('lon')
    axis.set_ylabel('lat')
    axis.set_xlim([125, 150])
    axis.set_ylim([25, 50])

clisops regridding functionalities - powered by xesmf

clisops.ops.regrid

Remap a global xarray.Dataset to a global 2.5 degree grid using the bilinear method

Load the dataset

Take a look at the grid

Remap to global 2.5 degree grid with the bilinear method

Plot the remapped data next to the source data

Remap regional xarray.Dataset to a regional grid of adaptive resolution using the bilinear method

Load the dataset

Take a look at the grid

Remap to regional regular lat-lon grid of adaptive resolution with the bilinear method

Plot the remapped data next to the source data

Remap unstructured xarray.Dataset to a global grid of adaptive resolution using the nearest neighbour method

Load the dataset

Take a look at the grid

Remap to global grid of adaptive resolution with the nearest neighbour method

Plot source data and remapped data

clisops.core.Grid

Create a grid object from an xarray.Dataset

Load the dataset

Create the Grid object

Plot the data

Create a grid object from an xarray.DataArray

Extract tos DataArray

Create Grid object for MPIOM tos dataarray:

Create a grid object using a grid_instructor

Create a grid object using a grid_id

clisops.core.Grid objects can be compared to one another

Compare the tos dataset to the tos dataarray

Compare both 0.25° ERA5 Grids

Strip clisops.core.Grid objects of all data_vars and coords unrelated to the horizontal grid

Transfer coordinate variables between clisops.core.Grid objects that are unrelated to the horizontal grid

Load the dataset

Create grid object

Transfer the coordinates to the ERA5 grid object

clisops.core.Weights

Create 2-degree target grid

Create conservative remapping weights using the clisops.core.Weights class

Local weights cache

clisops.core.regrid

In the following an example showing the function application and the effect of the adaptive masking.

Plot the resulting data

`clisops` regridding functionalities - powered by `xesmf`

`clisops.ops.regrid`

Remap a global `xarray.Dataset` to a global 2.5 degree grid using the bilinear method

Remap regional `xarray.Dataset` to a regional grid of adaptive resolution using the bilinear method

Remap unstructured `xarray.Dataset` to a global grid of adaptive resolution using the nearest neighbour method

`clisops.core.Grid`

Create a grid object from an `xarray.Dataset`

Create a grid object from an `xarray.DataArray`

Extract tos `DataArray`

Create a grid object using a `grid_instructor`

Create a grid object using a `grid_id`

`clisops.core.Grid` objects can be compared to one another

Strip `clisops.core.Grid` objects of all `data_vars` and `coords` unrelated to the horizontal grid

Transfer coordinate variables between `clisops.core.Grid` objects that are unrelated to the horizontal grid

`clisops.core.Weights`

Create conservative remapping weights using the `clisops.core.Weights` class

`clisops.core.regrid`