clisops package

CLISOPS - Climate simulation operations.

clisops.showwarning(message, *args, **kwargs)

Inject warnings from warnings.warn into loguru.

Subpackages

• clisops.core package
  • Submodules
  • clisops.core.average module
  • clisops.core.regrid module
  • clisops.core.subset module
• clisops.ops package
  • Submodules
  • clisops.ops.average module
  • clisops.ops.base_operation module
  • clisops.ops.regrid module
  • clisops.ops.subset module
• clisops.parameter package
  • Submodules
  • clisops.parameter._utils module
  • clisops.parameter.area_parameter module
  • clisops.parameter.base_parameter module
  • clisops.parameter.collection_parameter module
  • clisops.parameter.dimension_parameter module
  • clisops.parameter.level_parameter module
  • clisops.parameter.parameterise module
  • clisops.parameter.time_components_parameter module
  • clisops.parameter.time_parameter module
• clisops.utils package
  • Submodules
  • clisops.utils.common module
  • clisops.utils.dataset_utils module
  • clisops.utils.file_namers module
  • clisops.utils.file_utils module
  • clisops.utils.output_utils module
  • clisops.utils.testing module
  • clisops.utils.time_utils module

Submodules

clisops.__version__ module

The version information for the CLISOPS package.

clisops.config module

Configuration management for clisops.

clisops.config._chain_config_types(conf, keys)

clisops.config._expand_mappings(mappings, modifier, items)

Expand mappings by replacing modifier with a list of items in each case.

clisops.config._gather_config_files(package=None)

clisops.config._get_mappers(conf)

clisops.config._load_config(package=None)

clisops.config._modify_fixed_path_mappings(config, name)

Expands the contents of fixed_path_mappings based on other fixed path modifiers`.

Return type:

None

Returns:

None – Contents are changed in place.

clisops.config._post_process(config)

Post-processes the contents of the config file to modify sections based on certain rules.

Return type:

None

Returns:

None – Contents are changed in place.

clisops.config._to_boolean(i)

clisops.config._to_dict(i)

clisops.config._to_float(i)

clisops.config._to_int(i)

clisops.config._to_list(i)

clisops.config.get_config(package=None)

Return the configuration dictionary.

If the configuration has not been loaded yet, it will load it from the config file.

Parameters:

package (str or os.PathLike[str] or Path or None, optional) – The package from which to load the configuration file. If None, use the default configuration file.

Return type:

dict[str, Any]

Returns:

dict – The configuration dictionary containing all the settings from the config file.

clisops.config.reload_config(package=None)

Reload the configuration from the config file.

Used for forcibly reloading the configuration from the config file, particularly useful for pytesting mock imports.

Parameters:

package (str or os.PathLike[str] or Path or None, optional) – The package from which to load the configuration file. If None, use the default configuration file.

Return type:

dict[str, Any]

Returns:

dict – The configuration dictionary containing all the settings from the config file. Environment variables are also set based on the configuration.

clisops.exceptions module

Exceptions used in CLISOPS.

exception clisops.exceptions.InconsistencyError

Bases: Exception

Raised when there is some inconsistency which prevents files from being scanned.

exception clisops.exceptions.InvalidCollection(message='Some or all of the requested collection are not in the list of available data.')

Bases: Exception

Raised when a collection is not valid or not available in the roocs.ini file.

Parameters:

message (str) – Custom error message to be displayed.

exception clisops.exceptions.InvalidParameterValue

Bases: Exception

Raised when a parameter value is invalid or cannot be processed.

exception clisops.exceptions.InvalidProject

Bases: Exception

Raised when the project is unknown to roocs.ini.

exception clisops.exceptions.MissingParameterValue

Bases: Exception

Raised when a required parameter is missing or not provided.

clisops.project_utils module

Project utilities for CLISOPS.

class clisops.project_utils.DatasetMapper(dset, project=None, force=False)

Bases: object

Class to map to data path, dataset ID and files from any dataset input.

Dset must be a string and can be input as:

• A dataset ID: e.g. “cmip5.output1.INM.inmcm4.rcp45.mon.ocean.Omon.r1i1p1.latest.zostoga”.

• A file path: e.g. “/badc/cmip5/data/cmip5/output1/MOHC/HadGEM2-ES/rcp85/mon/atmos/Amon/r1i1p1/latest/tas/tas_Amon_HadGEM2-ES_rcp85_r1i1p1_200512-203011.nc”.

• A path to a group of files: e.g. “/badc/cmip5/data/cmip5/output1/MOHC/HadGEM2-ES/rcp85/mon/atmos/Amon/r1i1p1/latest/tas/*.nc”.

• A directory e.g. “/badc/cmip5/data/cmip5/output1/MOHC/HadGEM2-ES/rcp85/mon/atmos/Amon/r1i1p1/latest/tas”.

• An instance of the FileMapper class (that represents a set of files within a single directory).

When force=True, if the project cannot be identified, any attempt to use the base_dir of a project to resolve the data path will be ignored. Any of data_path, ds_id, and files that can be set will be set.

Parameters:

• dset (str or FileMapper) – The dataset input, which can be a Dataset/DataArray, a string representing a dataset ID or file path, or an instance of FileMapper.

• project (str, optional) – The project name to use for mapping the dataset. If not provided, it will be deduced from the dataset input.

• force (bool, optional) – If True, the function will attempt to find files even if the project of the input dataset cannot be identified. Default is False.

SUPPORTED_EXTENSIONS = ('.nc', '.gz')

_deduce_project(dset)

static _get_base_dirs_dict()

static _is_ds_id(dset)

_parse(force)

property base_dir

The base directory of the input dataset.

Returns:

str – The base directory where the dataset files are located, derived from the project configuration.

property data_path

Dataset input converted to a data path.

Returns:

str – The data path derived from the input dataset, which can be a file path or a directory path.

property ds_id

Dataset input converted to a ds id.

Returns:

str – The dataset ID derived from the input dataset, which is typically in the format “project.dataset_id”.

property files

The files found from the input dataset.

Returns:

list – A list of file paths deduced from the input dataset. If the dataset is a directory or a file pattern, it will return all matching files. If the dataset is a dataset ID, it will return files based on the dataset ID mapping.

property project

The project of the dataset input.

Returns:

str – The project name derived from the input dataset. If the project cannot be identified, it will return None.

property raw

Raw dataset input.

Returns:

xarray.Dataset or xarray.DataArray or str or FileMapper – The original dataset input provided to the DatasetMapper.

clisops.project_utils.datapath_to_dsid(datapath)

Switch from dataset path to ds id.

Parameters:

datapath (xarray.Dataset or xarray.DataArray or str or FileMapper) – The dataset input, which can be a Dataset/DataArray, a string representing a dataset ID or file path, or an instance of FileMapper.

Return type:

str

Returns:

str – The dataset id derived from the input dataset path.

clisops.project_utils.derive_ds_id(dset)

Derive the dataset id of the provided dset.

Parameters:

dset (xarray.Dataset or xarray.DataArray or str or FileMapper) – The dataset input, which can be a Dataset/DataArray, a string representing a dataset ID or file path, or an instance of FileMapper.

Return type:

str

Returns:

str – The dataset id derived from the input dataset.

clisops.project_utils.derive_dset(dset)

Derive the dataset path of the provided dset.

Parameters:

dset (xarray.Dataset or xarray.DataArray or str or FileMapper) – The dataset input, which can be a Dataset/DataArray, a string representing a dataset ID or file path, or an instance of FileMapper.

Return type:

str

Returns:

str – The dataset path derived from the input dataset.

clisops.project_utils.dset_to_filepaths(dset, force=False)

Get filepaths deduced from input dset.

Parameters:

• dset (xarray.Dataset or xarray.DataArray or str or FileMapper) – The dataset input, which can be a Dataset/DataArray, a string representing a dataset ID or file path, or an instance of FileMapper.

• force (bool, optional) – If True, the function will attempt to find files even if the project of the input dset cannot be identified. Default is False.

Returns:

list – A list of file paths deduced from the input dataset.

clisops.project_utils.dsid_to_datapath(dsid)

Switch from ds id to dataset path.

Parameters:

dsid (str) – The dataset ID, which should be in the format “project.dataset_id”.

Return type:

str

Returns:

str – The dataset path derived from the input dataset ID.

clisops.project_utils.get_data_node_dirs_dict()

Get a dictionary of the data node roots used for retrieving original files.

Return type:

dict[str, str]

Returns:

dict – A dictionary where keys are project names and values are the data node root directories.

clisops.project_utils.get_facet(facet_name, facets, project)

Get facet from project config.

Parameters:

• facet_name (str) – The name of the facet to retrieve.

• facets (dict) – A dictionary of facets from the project configuration.

• project (str) – The project name to use for mapping the facet.

Return type:

str

Returns:

str – The mapped facet value from the project configuration.

clisops.project_utils.get_project_base_dir(project)

Get the base directory of a project from the config.

Parameters:

project (str) – The name of the project for which to retrieve the base directory.

Return type:

str

Returns:

str – The base directory of the specified project.

clisops.project_utils.get_project_from_data_node_root(url)

Identify the project from data node root by identifying the data node root in the input url.

Parameters:

url (str) – The URL of the original file, which should contain the data node root.

Return type:

str

Returns:

str – The project name derived from the data node root in the input URL.

Raises:

InvalidProject – If the project cannot be identified from the URL.

clisops.project_utils.get_project_from_ds(ds)

Get the project from an xarray Dataset/DataArray.

Parameters:

ds (xarray.Dataset or xarray.DataArray) – The xarray Dataset or DataArray from which to derive the project.

Return type:

str | None

Returns:

str | None – The project derived from the input dataset.

clisops.project_utils.get_project_name(dset)

Get the project from an input dset.

Parameters:

dset (xarray.Dataset or xarray.DataArray or str or FileMapper) – The dataset input, which can be a Dataset/DataArray, a string representing a dataset ID or file path, or an instance of FileMapper.

Return type:

str | None

Returns:

str | None – The project name derived from the input dataset, or None if the project cannot be identified.

clisops.project_utils.get_projects()

Get all the projects available in the config.

Return type:

list[str]

Returns:

list of str – A list of project names derived from the configuration.

clisops.project_utils.map_facet(facet, project)

Return mapped facet value from config or facet name if not found.

Parameters:

• facet (str) – The facet name to map.

• project (str) – The project name to use for mapping.

Return type:

str

Returns:

str – The mapped facet value or the original facet name if no mapping is found.

clisops.project_utils.switch_dset(dset)

Switch between dataset path and ds id.

Parameters:

dset (xarray.Dataset or xarray.DataArray or str or FileMapper) – The dataset input, which can be a Dataset/DataArray, a string representing a dataset ID or file path, or an instance of FileMapper.

Return type:

str

Returns:

str – The dataset path or dataset ID derived from the input dataset, switched from the input.

clisops.project_utils.url_to_file_path(url)

Convert the input url of an original file to a file path.

Parameters:

url (str) – The URL of the original file, which should contain the data node root.

Return type:

str

Returns:

str – The file path derived from the input URL, based on the project’s base directory and data node root.