Subsetting

The subset operation makes use of clisops.core.subset to process the datasets and to set the output type and the output file names.

from clisops.utils import get_file
# fetch files locally or from github
tas_files = get_file([
    "cmip5/tas_Amon_HadGEM2-ES_rcp85_r1i1p1_200512-203011.nc",
    "cmip5/tas_Amon_HadGEM2-ES_rcp85_r1i1p1_203012-205511.nc",
    "cmip5/tas_Amon_HadGEM2-ES_rcp85_r1i1p1_205512-208011.nc",
    "cmip5/tas_Amon_HadGEM2-ES_rcp85_r1i1p1_208012-209912.nc",
    "cmip5/tas_Amon_HadGEM2-ES_rcp85_r1i1p1_209912-212411.nc",
    "cmip5/tas_Amon_HadGEM2-ES_rcp85_r1i1p1_212412-214911.nc",
    "cmip5/tas_Amon_HadGEM2-ES_rcp85_r1i1p1_214912-217411.nc",
    "cmip5/tas_Amon_HadGEM2-ES_rcp85_r1i1p1_217412-219911.nc",
    "cmip5/tas_Amon_HadGEM2-ES_rcp85_r1i1p1_219912-222411.nc",
    "cmip5/tas_Amon_HadGEM2-ES_rcp85_r1i1p1_222412-224911.nc",
    "cmip5/tas_Amon_HadGEM2-ES_rcp85_r1i1p1_224912-227411.nc",
    "cmip5/tas_Amon_HadGEM2-ES_rcp85_r1i1p1_227412-229911.nc",
    "cmip5/tas_Amon_HadGEM2-ES_rcp85_r1i1p1_229912-229912.nc"
])

o3_file = get_file("cmip6/o3_Amon_GFDL-ESM4_historical_r1i1p1f1_gr1_185001-194912.nc")

# remove previously created example file
import os
if os.path.exists("./output_001.nc"):
    os.remove("./output_001.nc")

from clisops.ops.subset import subset
import xarray as xr

The subset process takes several parameters:

Subsetting Parameters

ds: Union[xr.Dataset, str, Path]
time: Optional[Union[str, TimeParameter]]
area: Optional[
    Union[
        str,
        Tuple[
            Union[int, float, str],
            Union[int, float, str],
            Union[int, float, str],
            Union[int, float, str],
        ],
        AreaParameter,
    ]
]
level: Optional[
    Union[
        str, LevelParameter
    ]
]
time_components: Optional[Union[str, Dict, TimeComponentsParameter]]
output_dir: Optional[Union[str, Path]]
output_type: {"netcdf", "nc", "zarr", "xarray"}
split_method: {"time:auto"}
file_namer: {"standard"}

The output is a list containing the outputs in the format selected.

ds = xr.open_mfdataset(tas_files, use_cftime=True, combine="by_coords")

Output to xarray

There will only be one output for this example.

outputs = subset(
        ds=ds,
        time="2007-01-01T00:00:00/2200-12-30T00:00:00",
        area=(0.0, 10.0, 175.0, 90.0),
        output_type="xarray",
    )

print(f"There is only {len(outputs)} output.")
outputs[0]

There is only 1 output.

/home/docs/checkouts/readthedocs.org/user_builds/clisops/conda/stable/lib/python3.11/site-packages/clisops/core/subset.py:1331: UserWarning: "start_date" has been nudged to nearest valid time step in xarray object.
  da = subset_time(da, start_date=start_date, end_date=end_date)
/home/docs/checkouts/readthedocs.org/user_builds/clisops/conda/stable/lib/python3.11/site-packages/clisops/core/subset.py:1331: UserWarning: "end_date" has been nudged to nearest valid time step in xarray object.
  da = subset_time(da, start_date=start_date, end_date=end_date)

<xarray.Dataset>
Dimensions:    (lat: 1, time: 2329, bnds: 2, lon: 1)
Coordinates:
    height     float64 1.5
  * lat        (lat) float64 35.0
  * lon        (lon) float64 0.0
  * time       (time) object 2007-01-16 00:00:00 ... 2200-12-16 00:00:00
Dimensions without coordinates: bnds
Data variables:
    lat_bnds   (time, lat, bnds) float64 dask.array<chunksize=(287, 1, 2), meta=np.ndarray>
    lon_bnds   (time, lon, bnds) float64 dask.array<chunksize=(287, 1, 2), meta=np.ndarray>
    tas        (time, lat, lon) float32 dask.array<chunksize=(287, 1, 1), meta=np.ndarray>
    time_bnds  (time, bnds) object dask.array<chunksize=(287, 2), meta=np.ndarray>
Attributes: (12/29)
    institution:            Met Office Hadley Centre, Fitzroy Road, Exeter, D...
    institute_id:           MOHC
    experiment_id:          rcp85
    source:                 HadGEM2-ES (2009) atmosphere: HadGAM2 (N96L38); o...
    model_id:               HadGEM2-ES
    forcing:                GHG, SA, Oz, LU, Sl, Vl, BC, OC, (GHG = CO2, N2O,...
    ...                     ...
    title:                  HadGEM2-ES model output prepared for CMIP5 RCP8.5
    parent_experiment:      historical
    modeling_realm:         atmos
    realization:            1
    cmor_version:           2.5.0
    NCO:                    4.7.3

xarray.Dataset

• Dimensions:
  • lat: 1
  • time: 2329
  • bnds: 2
  • lon: 1
• Coordinates: (4)
  • height
    ()
    float64
    1.5

    units :

    m

    axis :

    Z

    positive :

    up

    long_name :

    height

    standard_name :

    height

    array(1.5)

  • lat
    (lat)
    float64
    35.0

    bounds :

    lat_bnds

    units :

    degrees_north

    axis :

    Y

    long_name :

    latitude

    standard_name :

    latitude

    array([35.])

  • lon
    (lon)
    float64
    0.0

    bounds :

    lon_bnds

    units :

    degrees_east

    axis :

    X

    long_name :

    longitude

    standard_name :

    longitude

    array([0.])

  • time
    (time)
    object
    2007-01-16 00:00:00 ... 2200-12-...

    bounds :

    time_bnds

    axis :

    T

    long_name :

    time

    standard_name :

    time

    array([cftime.Datetime360Day(2007, 1, 16, 0, 0, 0, 0, has_year_zero=True),
           cftime.Datetime360Day(2007, 2, 16, 0, 0, 0, 0, has_year_zero=True),
           cftime.Datetime360Day(2007, 3, 16, 0, 0, 0, 0, has_year_zero=True), ...,
           cftime.Datetime360Day(2200, 10, 16, 0, 0, 0, 0, has_year_zero=True),
           cftime.Datetime360Day(2200, 11, 16, 0, 0, 0, 0, has_year_zero=True),
           cftime.Datetime360Day(2200, 12, 16, 0, 0, 0, 0, has_year_zero=True)],
          dtype=object)

• Data variables: (4)
  • lat_bnds
    (time, lat, bnds)
    float64
    dask.array<chunksize=(287, 1, 2), meta=np.ndarray>

    Array Chunk Bytes 36.39 kiB 4.69 kiB Shape (2329, 1, 2) (300, 1, 2) Dask graph 9 chunks in 42 graph layers Data type float64 numpy.ndarray

  • lon_bnds
    (time, lon, bnds)
    float64
    dask.array<chunksize=(287, 1, 2), meta=np.ndarray>

    Array Chunk Bytes 36.39 kiB 4.69 kiB Shape (2329, 1, 2) (300, 1, 2) Dask graph 9 chunks in 42 graph layers Data type float64 numpy.ndarray

  • tas
    (time, lat, lon)
    float32
    dask.array<chunksize=(287, 1, 1), meta=np.ndarray>

    standard_name :

    air_temperature

    long_name :

    Near-Surface Air Temperature

    comment :

    near-surface (usually, 2 meter) air temperature.

    units :

    K

    original_name :

    mo: m01s03i236

    cell_methods :

    time: mean

    cell_measures :

    area: areacella

    history :

    2010-12-04T13:50:30Z altered by CMOR: Treated scalar dimension: 'height'. 2010-12-04T13:50:30Z altered by CMOR: replaced missing value flag (-1.07374e+09) with standard missing value (1e+20).

    associated_files :

    baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation gridspecFile: gridspec_atmos_fx_HadGEM2-ES_rcp85_r0i0p0.nc areacella: areacella_fx_HadGEM2-ES_rcp85_r0i0p0.nc

    Array Chunk Bytes 9.10 kiB 1.17 kiB Shape (2329, 1, 1) (300, 1, 1) Dask graph 9 chunks in 30 graph layers Data type float32 numpy.ndarray

  • time_bnds
    (time, bnds)
    object
    dask.array<chunksize=(287, 2), meta=np.ndarray>

    Array Chunk Bytes 36.39 kiB 4.69 kiB Shape (2329, 2) (300, 2) Dask graph 9 chunks in 28 graph layers Data type object numpy.ndarray

• Indexes: (3)
  • lat
    PandasIndex

    PandasIndex(Float64Index([35.0], dtype='float64', name='lat'))

  • lon
    PandasIndex

    PandasIndex(Float64Index([0.0], dtype='float64', name='lon'))

  • time
    PandasIndex

    PandasIndex(CFTimeIndex([2007-01-16 00:00:00, 2007-02-16 00:00:00, 2007-03-16 00:00:00,
                 2007-04-16 00:00:00, 2007-05-16 00:00:00, 2007-06-16 00:00:00,
                 2007-07-16 00:00:00, 2007-08-16 00:00:00, 2007-09-16 00:00:00,
                 2007-10-16 00:00:00,
                 ...
                 2200-03-16 00:00:00, 2200-04-16 00:00:00, 2200-05-16 00:00:00,
                 2200-06-16 00:00:00, 2200-07-16 00:00:00, 2200-08-16 00:00:00,
                 2200-09-16 00:00:00, 2200-10-16 00:00:00, 2200-11-16 00:00:00,
                 2200-12-16 00:00:00],
                dtype='object', length=2329, calendar='360_day', freq='None'))

• Attributes: (29)

  institution :

  Met Office Hadley Centre, Fitzroy Road, Exeter, Devon, EX1 3PB, UK, (http://www.metoffice.gov.uk)

  institute_id :

  MOHC

  experiment_id :

  rcp85

  source :

  HadGEM2-ES (2009) atmosphere: HadGAM2 (N96L38); ocean: HadGOM2 (lat: 1.0-0.3 lon: 1.0 L40); land-surface/vegetation: MOSES2 and TRIFFID; tropospheric chemistry: UKCA; ocean biogeochemistry: diat-HadOCC

  model_id :

  HadGEM2-ES

  forcing :

  GHG, SA, Oz, LU, Sl, Vl, BC, OC, (GHG = CO2, N2O, CH4, CFCs)

  parent_experiment_id :

  historical

  parent_experiment_rip :

  r1i1p1

  branch_time :

  52560.0

  contact :

  chris.d.jones@metoffice.gov.uk, michael.sanderson@metoffice.gov.uk

  history :

  Mon Mar 9 09:10:39 2020: ncks -d lat,,,100 -d lon,,,100 -v tas /badc/cmip5/data/cmip5/output1/MOHC/HadGEM2-ES/rcp85/mon/atmos/Amon/r1i1p1/latest/tas/tas_Amon_HadGEM2-ES_rcp85_r1i1p1_200512-203011.nc test_data/badc/cmip5/data/cmip5/output1/MOHC/HadGEM2-ES/rcp85/mon/atmos/Amon/r1i1p1/latest/tas/tas_Amon_HadGEM2-ES_rcp85_r1i1p1_200512-203011.nc MOHC pp to CMOR/NetCDF convertor (version 1.5.1) 2010-12-04T13:50:10Z CMOR rewrote data to comply with CF standards and CMIP5 requirements.

  references :

  Bellouin N. et al, (2007) Improved representation of aerosols for HadGEM2. Meteorological Office Hadley Centre, Technical Note 73, March 2007; Collins W.J. et al, (2008) Evaluation of the HadGEM2 model. Meteorological Office Hadley Centre, Technical Note 74,; Johns T.C. et al, (2006) The new Hadley Centre climate model HadGEM1: Evaluation of coupled simulations. Journal of Climate, American Meteorological Society, Vol. 19, No. 7, pages 1327-1353.; Martin G.M. et al, (2006) The physical properties of the atmosphere in the new Hadley Centre Global Environmental Model, HadGEM1 - Part 1: Model description and global climatology. Journal of Climate, American Meteorological Society, Vol. 19, No.7, pages 1274-1301.; Ringer M.A. et al, (2006) The physical properties of the atmosphere in the new Hadley Centre Global Environmental Model, HadGEM1 - Part 2: Aspects of variability and regional climate. Journal of Climate, American Meteorological Society, Vol. 19, No. 7, pages 1302-1326.

  initialization_method :

  1

  physics_version :

  1

  tracking_id :

  948b8aa2-4b1f-422a-921f-4515fcf9860b

  mo_runid :

  ajnji

  product :

  output

  experiment :

  RCP8.5

  frequency :

  mon

  creation_date :

  2010-12-04T13:50:30Z

  Conventions :

  CF-1.4

  project_id :

  CMIP5

  table_id :

  Table Amon (12 November 2010) 6e535ddfacb41fb7a252f4862fdc5766

  title :

  HadGEM2-ES model output prepared for CMIP5 RCP8.5

  parent_experiment :

  historical

  modeling_realm :

  atmos

  realization :

  1

  cmor_version :

  2.5.0

  NCO :

  4.7.3

Output to netCDF with simple namer

There is only one output as the file size is under the memory limit so does not need to be split. This example uses the simple namer which numbers output files.

outputs = subset(
        ds=ds,
        time="2007-01-01T00:00:00/2200-12-30T00:00:00",
        area=(0.0, 10.0, 175.0, 90.0),
        output_type="nc",
        output_dir=".",
        split_method="time:auto",
        file_namer="simple"
    )

/home/docs/checkouts/readthedocs.org/user_builds/clisops/conda/stable/lib/python3.11/site-packages/clisops/core/subset.py:1331: UserWarning: "start_date" has been nudged to nearest valid time step in xarray object.
  da = subset_time(da, start_date=start_date, end_date=end_date)
/home/docs/checkouts/readthedocs.org/user_builds/clisops/conda/stable/lib/python3.11/site-packages/clisops/core/subset.py:1331: UserWarning: "end_date" has been nudged to nearest valid time step in xarray object.
  da = subset_time(da, start_date=start_date, end_date=end_date)

# To open the file

subset_ds = xr.open_mfdataset("./output_001.nc", use_cftime=True, combine="by_coords")
subset_ds

<xarray.Dataset>
Dimensions:    (lat: 1, time: 2329, bnds: 2, lon: 1)
Coordinates:
    height     float64 ...
  * lat        (lat) float64 35.0
  * lon        (lon) float64 0.0
  * time       (time) object 2007-01-16 00:00:00 ... 2200-12-16 00:00:00
Dimensions without coordinates: bnds
Data variables:
    lat_bnds   (time, lat, bnds) float64 dask.array<chunksize=(2329, 1, 2), meta=np.ndarray>
    lon_bnds   (time, lon, bnds) float64 dask.array<chunksize=(2329, 1, 2), meta=np.ndarray>
    tas        (time, lat, lon) float32 dask.array<chunksize=(2329, 1, 1), meta=np.ndarray>
    time_bnds  (time, bnds) object dask.array<chunksize=(2329, 2), meta=np.ndarray>
Attributes: (12/29)
    institution:            Met Office Hadley Centre, Fitzroy Road, Exeter, D...
    institute_id:           MOHC
    experiment_id:          rcp85
    source:                 HadGEM2-ES (2009) atmosphere: HadGAM2 (N96L38); o...
    model_id:               HadGEM2-ES
    forcing:                GHG, SA, Oz, LU, Sl, Vl, BC, OC, (GHG = CO2, N2O,...
    ...                     ...
    title:                  HadGEM2-ES model output prepared for CMIP5 RCP8.5
    parent_experiment:      historical
    modeling_realm:         atmos
    realization:            1
    cmor_version:           2.5.0
    NCO:                    4.7.3

xarray.Dataset

• Dimensions:
  • lat: 1
  • time: 2329
  • bnds: 2
  • lon: 1
• Coordinates: (4)
  • height
    ()
    float64
    ...

    units :

    m

    axis :

    Z

    positive :

    up

    long_name :

    height

    standard_name :

    height

    [1 values with dtype=float64]

  • lat
    (lat)
    float64
    35.0

    bounds :

    lat_bnds

    units :

    degrees_north

    axis :

    Y

    long_name :

    latitude

    standard_name :

    latitude

    array([35.])

  • lon
    (lon)
    float64
    0.0

    bounds :

    lon_bnds

    units :

    degrees_east

    axis :

    X

    long_name :

    longitude

    standard_name :

    longitude

    array([0.])

  • time
    (time)
    object
    2007-01-16 00:00:00 ... 2200-12-...

    bounds :

    time_bnds

    axis :

    T

    long_name :

    time

    standard_name :

    time

    array([cftime.Datetime360Day(2007, 1, 16, 0, 0, 0, 0, has_year_zero=True),
           cftime.Datetime360Day(2007, 2, 16, 0, 0, 0, 0, has_year_zero=True),
           cftime.Datetime360Day(2007, 3, 16, 0, 0, 0, 0, has_year_zero=True), ...,
           cftime.Datetime360Day(2200, 10, 16, 0, 0, 0, 0, has_year_zero=True),
           cftime.Datetime360Day(2200, 11, 16, 0, 0, 0, 0, has_year_zero=True),
           cftime.Datetime360Day(2200, 12, 16, 0, 0, 0, 0, has_year_zero=True)],
          dtype=object)

• Data variables: (4)
  • lat_bnds
    (time, lat, bnds)
    float64
    dask.array<chunksize=(2329, 1, 2), meta=np.ndarray>

    Array Chunk Bytes 36.39 kiB 36.39 kiB Shape (2329, 1, 2) (2329, 1, 2) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

  • lon_bnds
    (time, lon, bnds)
    float64
    dask.array<chunksize=(2329, 1, 2), meta=np.ndarray>

    Array Chunk Bytes 36.39 kiB 36.39 kiB Shape (2329, 1, 2) (2329, 1, 2) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

  • tas
    (time, lat, lon)
    float32
    dask.array<chunksize=(2329, 1, 1), meta=np.ndarray>

    standard_name :

    air_temperature

    long_name :

    Near-Surface Air Temperature

    comment :

    near-surface (usually, 2 meter) air temperature.

    units :

    K

    original_name :

    mo: m01s03i236

    cell_methods :

    time: mean

    cell_measures :

    area: areacella

    history :

    2010-12-04T13:50:30Z altered by CMOR: Treated scalar dimension: 'height'. 2010-12-04T13:50:30Z altered by CMOR: replaced missing value flag (-1.07374e+09) with standard missing value (1e+20).

    associated_files :

    baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation gridspecFile: gridspec_atmos_fx_HadGEM2-ES_rcp85_r0i0p0.nc areacella: areacella_fx_HadGEM2-ES_rcp85_r0i0p0.nc

    Array Chunk Bytes 9.10 kiB 9.10 kiB Shape (2329, 1, 1) (2329, 1, 1) Dask graph 1 chunks in 2 graph layers Data type float32 numpy.ndarray

  • time_bnds
    (time, bnds)
    object
    dask.array<chunksize=(2329, 2), meta=np.ndarray>

    Array Chunk Bytes 36.39 kiB 36.39 kiB Shape (2329, 2) (2329, 2) Dask graph 1 chunks in 2 graph layers Data type object numpy.ndarray

• Indexes: (3)
  • lat
    PandasIndex

    PandasIndex(Float64Index([35.0], dtype='float64', name='lat'))

  • lon
    PandasIndex

    PandasIndex(Float64Index([0.0], dtype='float64', name='lon'))

  • time
    PandasIndex

    PandasIndex(CFTimeIndex([2007-01-16 00:00:00, 2007-02-16 00:00:00, 2007-03-16 00:00:00,
                 2007-04-16 00:00:00, 2007-05-16 00:00:00, 2007-06-16 00:00:00,
                 2007-07-16 00:00:00, 2007-08-16 00:00:00, 2007-09-16 00:00:00,
                 2007-10-16 00:00:00,
                 ...
                 2200-03-16 00:00:00, 2200-04-16 00:00:00, 2200-05-16 00:00:00,
                 2200-06-16 00:00:00, 2200-07-16 00:00:00, 2200-08-16 00:00:00,
                 2200-09-16 00:00:00, 2200-10-16 00:00:00, 2200-11-16 00:00:00,
                 2200-12-16 00:00:00],
                dtype='object', length=2329, calendar='360_day', freq='None'))

• Attributes: (29)

  institution :

  Met Office Hadley Centre, Fitzroy Road, Exeter, Devon, EX1 3PB, UK, (http://www.metoffice.gov.uk)

  institute_id :

  MOHC

  experiment_id :

  rcp85

  source :

  HadGEM2-ES (2009) atmosphere: HadGAM2 (N96L38); ocean: HadGOM2 (lat: 1.0-0.3 lon: 1.0 L40); land-surface/vegetation: MOSES2 and TRIFFID; tropospheric chemistry: UKCA; ocean biogeochemistry: diat-HadOCC

  model_id :

  HadGEM2-ES

  forcing :

  GHG, SA, Oz, LU, Sl, Vl, BC, OC, (GHG = CO2, N2O, CH4, CFCs)

  parent_experiment_id :

  historical

  parent_experiment_rip :

  r1i1p1

  branch_time :

  52560.0

  contact :

  chris.d.jones@metoffice.gov.uk, michael.sanderson@metoffice.gov.uk

  history :

  Mon Mar 9 09:10:39 2020: ncks -d lat,,,100 -d lon,,,100 -v tas /badc/cmip5/data/cmip5/output1/MOHC/HadGEM2-ES/rcp85/mon/atmos/Amon/r1i1p1/latest/tas/tas_Amon_HadGEM2-ES_rcp85_r1i1p1_200512-203011.nc test_data/badc/cmip5/data/cmip5/output1/MOHC/HadGEM2-ES/rcp85/mon/atmos/Amon/r1i1p1/latest/tas/tas_Amon_HadGEM2-ES_rcp85_r1i1p1_200512-203011.nc MOHC pp to CMOR/NetCDF convertor (version 1.5.1) 2010-12-04T13:50:10Z CMOR rewrote data to comply with CF standards and CMIP5 requirements.

  references :

  Bellouin N. et al, (2007) Improved representation of aerosols for HadGEM2. Meteorological Office Hadley Centre, Technical Note 73, March 2007; Collins W.J. et al, (2008) Evaluation of the HadGEM2 model. Meteorological Office Hadley Centre, Technical Note 74,; Johns T.C. et al, (2006) The new Hadley Centre climate model HadGEM1: Evaluation of coupled simulations. Journal of Climate, American Meteorological Society, Vol. 19, No. 7, pages 1327-1353.; Martin G.M. et al, (2006) The physical properties of the atmosphere in the new Hadley Centre Global Environmental Model, HadGEM1 - Part 1: Model description and global climatology. Journal of Climate, American Meteorological Society, Vol. 19, No.7, pages 1274-1301.; Ringer M.A. et al, (2006) The physical properties of the atmosphere in the new Hadley Centre Global Environmental Model, HadGEM1 - Part 2: Aspects of variability and regional climate. Journal of Climate, American Meteorological Society, Vol. 19, No. 7, pages 1302-1326.

  initialization_method :

  1

  physics_version :

  1

  tracking_id :

  948b8aa2-4b1f-422a-921f-4515fcf9860b

  mo_runid :

  ajnji

  product :

  output

  experiment :

  RCP8.5

  frequency :

  mon

  creation_date :

  2010-12-04T13:50:30Z

  Conventions :

  CF-1.4

  project_id :

  CMIP5

  table_id :

  Table Amon (12 November 2010) 6e535ddfacb41fb7a252f4862fdc5766

  title :

  HadGEM2-ES model output prepared for CMIP5 RCP8.5

  parent_experiment :

  historical

  modeling_realm :

  atmos

  realization :

  1

  cmor_version :

  2.5.0

  NCO :

  4.7.3

Output to netCDF with standard namer

There is only one output as the file size is under the memory limit so does not need to be split. This example uses the standard namer which names output filesa ccording the the input file and how it has been subsetted.

outputs = subset(
        ds=ds,
        time="2007-01-01T00:00:00/2200-12-30T00:00:00",
        area=(0.0, 10.0, 175.0, 90.0),
        output_type="nc",
        output_dir=".",
        split_method="time:auto",
        file_namer="standard"
    )

/home/docs/checkouts/readthedocs.org/user_builds/clisops/conda/stable/lib/python3.11/site-packages/clisops/core/subset.py:1331: UserWarning: "start_date" has been nudged to nearest valid time step in xarray object.
  da = subset_time(da, start_date=start_date, end_date=end_date)
/home/docs/checkouts/readthedocs.org/user_builds/clisops/conda/stable/lib/python3.11/site-packages/clisops/core/subset.py:1331: UserWarning: "end_date" has been nudged to nearest valid time step in xarray object.
  da = subset_time(da, start_date=start_date, end_date=end_date)

Subsetting by level

ds = xr.open_dataset(o3_file, use_cftime=True)

No subsetting applied

result = subset(ds=ds,
                output_type="xarray")

result[0].coords

Coordinates:
  * lat      (lat) float64 -89.5 10.5
  * lon      (lon) float64 0.625 125.6 250.6
  * plev     (plev) float64 1e+05 9.25e+04 8.5e+04 7e+04 ... 1e+03 500.0 100.0
  * time     (time) object 1850-01-16 12:00:00 ... 1949-12-16 12:00:00

Subsetting over level

# subsetting over pressure level (plev)

result = subset(ds=ds,
                level="600/100",
                output_type="xarray")

print(result[0].coords)
print(f"\nplev has been subsetted and now only has {len(result[0].coords)} values.")

Coordinates:
  * lat      (lat) float64 -89.5 10.5
  * lon      (lon) float64 0.625 125.6 250.6
  * plev     (plev) float64 500.0 100.0
  * time     (time) object 1850-01-16 12:00:00 ... 1949-12-16 12:00:00

plev has been subsetted and now only has 4 values.

/home/docs/checkouts/readthedocs.org/user_builds/clisops/conda/stable/lib/python3.11/site-packages/clisops/ops/subset.py:146: UserWarning: "first_level" has been nudged to nearest valid level in xarray object.
  result = subset_level(result, **kwargs)

Use time components

ds = xr.open_mfdataset(tas_files, use_cftime=True, combine="by_coords")

outputs = subset(
        ds=ds,
        time_components="year: 2010, 2020, 2030|month: 12, 1, 2",
        output_type="xarray",
    )

print(f"There is only {len(outputs)} output.")
outputs[0]

There is only 1 output.

<xarray.Dataset>
Dimensions:    (lat: 2, time: 9, bnds: 2, lon: 2)
Coordinates:
    height     float64 1.5
  * lat        (lat) float64 -90.0 35.0
  * lon        (lon) float64 0.0 187.5
  * time       (time) object 2010-01-16 00:00:00 ... 2030-12-16 00:00:00
Dimensions without coordinates: bnds
Data variables:
    lat_bnds   (time, lat, bnds) float64 dask.array<chunksize=(8, 2, 2), meta=np.ndarray>
    lon_bnds   (time, lon, bnds) float64 dask.array<chunksize=(8, 2, 2), meta=np.ndarray>
    tas        (time, lat, lon) float32 dask.array<chunksize=(8, 2, 2), meta=np.ndarray>
    time_bnds  (time, bnds) object dask.array<chunksize=(8, 2), meta=np.ndarray>
Attributes: (12/29)
    institution:            Met Office Hadley Centre, Fitzroy Road, Exeter, D...
    institute_id:           MOHC
    experiment_id:          rcp85
    source:                 HadGEM2-ES (2009) atmosphere: HadGAM2 (N96L38); o...
    model_id:               HadGEM2-ES
    forcing:                GHG, SA, Oz, LU, Sl, Vl, BC, OC, (GHG = CO2, N2O,...
    ...                     ...
    title:                  HadGEM2-ES model output prepared for CMIP5 RCP8.5
    parent_experiment:      historical
    modeling_realm:         atmos
    realization:            1
    cmor_version:           2.5.0
    NCO:                    4.7.3

xarray.Dataset

• Dimensions:
  • lat: 2
  • time: 9
  • bnds: 2
  • lon: 2
• Coordinates: (4)
  • height
    ()
    float64
    1.5

    units :

    m

    axis :

    Z

    positive :

    up

    long_name :

    height

    standard_name :

    height

    array(1.5)

  • lat
    (lat)
    float64
    -90.0 35.0

    bounds :

    lat_bnds

    units :

    degrees_north

    axis :

    Y

    long_name :

    latitude

    standard_name :

    latitude

    array([-90.,  35.])

  • lon
    (lon)
    float64
    0.0 187.5

    bounds :

    lon_bnds

    units :

    degrees_east

    axis :

    X

    long_name :

    longitude

    standard_name :

    longitude

    array([  0. , 187.5])

  • time
    (time)
    object
    2010-01-16 00:00:00 ... 2030-12-...

    bounds :

    time_bnds

    axis :

    T

    long_name :

    time

    standard_name :

    time

    array([cftime.Datetime360Day(2010, 1, 16, 0, 0, 0, 0, has_year_zero=True),
           cftime.Datetime360Day(2010, 2, 16, 0, 0, 0, 0, has_year_zero=True),
           cftime.Datetime360Day(2010, 12, 16, 0, 0, 0, 0, has_year_zero=True),
           cftime.Datetime360Day(2020, 1, 16, 0, 0, 0, 0, has_year_zero=True),
           cftime.Datetime360Day(2020, 2, 16, 0, 0, 0, 0, has_year_zero=True),
           cftime.Datetime360Day(2020, 12, 16, 0, 0, 0, 0, has_year_zero=True),
           cftime.Datetime360Day(2030, 1, 16, 0, 0, 0, 0, has_year_zero=True),
           cftime.Datetime360Day(2030, 2, 16, 0, 0, 0, 0, has_year_zero=True),
           cftime.Datetime360Day(2030, 12, 16, 0, 0, 0, 0, has_year_zero=True)],
          dtype=object)

• Data variables: (4)
  • lat_bnds
    (time, lat, bnds)
    float64
    dask.array<chunksize=(8, 2, 2), meta=np.ndarray>

    Array Chunk Bytes 288 B 256 B Shape (9, 2, 2) (8, 2, 2) Dask graph 2 chunks in 41 graph layers Data type float64 numpy.ndarray

  • lon_bnds
    (time, lon, bnds)
    float64
    dask.array<chunksize=(8, 2, 2), meta=np.ndarray>

    Array Chunk Bytes 288 B 256 B Shape (9, 2, 2) (8, 2, 2) Dask graph 2 chunks in 41 graph layers Data type float64 numpy.ndarray

  • tas
    (time, lat, lon)
    float32
    dask.array<chunksize=(8, 2, 2), meta=np.ndarray>

    standard_name :

    air_temperature

    long_name :

    Near-Surface Air Temperature

    comment :

    near-surface (usually, 2 meter) air temperature.

    units :

    K

    original_name :

    mo: m01s03i236

    cell_methods :

    time: mean

    cell_measures :

    area: areacella

    history :

    2010-12-04T13:50:30Z altered by CMOR: Treated scalar dimension: 'height'. 2010-12-04T13:50:30Z altered by CMOR: replaced missing value flag (-1.07374e+09) with standard missing value (1e+20).

    associated_files :

    baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation gridspecFile: gridspec_atmos_fx_HadGEM2-ES_rcp85_r0i0p0.nc areacella: areacella_fx_HadGEM2-ES_rcp85_r0i0p0.nc

    Array Chunk Bytes 144 B 128 B Shape (9, 2, 2) (8, 2, 2) Dask graph 2 chunks in 28 graph layers Data type float32 numpy.ndarray

  • time_bnds
    (time, bnds)
    object
    dask.array<chunksize=(8, 2), meta=np.ndarray>

    Array Chunk Bytes 144 B 128 B Shape (9, 2) (8, 2) Dask graph 2 chunks in 28 graph layers Data type object numpy.ndarray

• Indexes: (3)
  • lat
    PandasIndex

    PandasIndex(Float64Index([-90.0, 35.0], dtype='float64', name='lat'))

  • lon
    PandasIndex

    PandasIndex(Float64Index([0.0, 187.5], dtype='float64', name='lon'))

  • time
    PandasIndex

    PandasIndex(CFTimeIndex([2010-01-16 00:00:00, 2010-02-16 00:00:00, 2010-12-16 00:00:00,
                 2020-01-16 00:00:00, 2020-02-16 00:00:00, 2020-12-16 00:00:00,
                 2030-01-16 00:00:00, 2030-02-16 00:00:00, 2030-12-16 00:00:00],
                dtype='object', length=9, calendar='360_day', freq='None'))

• Attributes: (29)

  institution :

  Met Office Hadley Centre, Fitzroy Road, Exeter, Devon, EX1 3PB, UK, (http://www.metoffice.gov.uk)

  institute_id :

  MOHC

  experiment_id :

  rcp85

  source :

  HadGEM2-ES (2009) atmosphere: HadGAM2 (N96L38); ocean: HadGOM2 (lat: 1.0-0.3 lon: 1.0 L40); land-surface/vegetation: MOSES2 and TRIFFID; tropospheric chemistry: UKCA; ocean biogeochemistry: diat-HadOCC

  model_id :

  HadGEM2-ES

  forcing :

  GHG, SA, Oz, LU, Sl, Vl, BC, OC, (GHG = CO2, N2O, CH4, CFCs)

  parent_experiment_id :

  historical

  parent_experiment_rip :

  r1i1p1

  branch_time :

  52560.0

  contact :

  chris.d.jones@metoffice.gov.uk, michael.sanderson@metoffice.gov.uk

  history :

  Mon Mar 9 09:10:39 2020: ncks -d lat,,,100 -d lon,,,100 -v tas /badc/cmip5/data/cmip5/output1/MOHC/HadGEM2-ES/rcp85/mon/atmos/Amon/r1i1p1/latest/tas/tas_Amon_HadGEM2-ES_rcp85_r1i1p1_200512-203011.nc test_data/badc/cmip5/data/cmip5/output1/MOHC/HadGEM2-ES/rcp85/mon/atmos/Amon/r1i1p1/latest/tas/tas_Amon_HadGEM2-ES_rcp85_r1i1p1_200512-203011.nc MOHC pp to CMOR/NetCDF convertor (version 1.5.1) 2010-12-04T13:50:10Z CMOR rewrote data to comply with CF standards and CMIP5 requirements.

  references :

  Bellouin N. et al, (2007) Improved representation of aerosols for HadGEM2. Meteorological Office Hadley Centre, Technical Note 73, March 2007; Collins W.J. et al, (2008) Evaluation of the HadGEM2 model. Meteorological Office Hadley Centre, Technical Note 74,; Johns T.C. et al, (2006) The new Hadley Centre climate model HadGEM1: Evaluation of coupled simulations. Journal of Climate, American Meteorological Society, Vol. 19, No. 7, pages 1327-1353.; Martin G.M. et al, (2006) The physical properties of the atmosphere in the new Hadley Centre Global Environmental Model, HadGEM1 - Part 1: Model description and global climatology. Journal of Climate, American Meteorological Society, Vol. 19, No.7, pages 1274-1301.; Ringer M.A. et al, (2006) The physical properties of the atmosphere in the new Hadley Centre Global Environmental Model, HadGEM1 - Part 2: Aspects of variability and regional climate. Journal of Climate, American Meteorological Society, Vol. 19, No. 7, pages 1302-1326.

  initialization_method :

  1

  physics_version :

  1

  tracking_id :

  948b8aa2-4b1f-422a-921f-4515fcf9860b

  mo_runid :

  ajnji

  product :

  output

  experiment :

  RCP8.5

  frequency :

  mon

  creation_date :

  2010-12-04T13:50:30Z

  Conventions :

  CF-1.4

  project_id :

  CMIP5

  table_id :

  Table Amon (12 November 2010) 6e535ddfacb41fb7a252f4862fdc5766

  title :

  HadGEM2-ES model output prepared for CMIP5 RCP8.5

  parent_experiment :

  historical

  modeling_realm :

  atmos

  realization :

  1

  cmor_version :

  2.5.0

  NCO :

  4.7.3

Using parameter classes

from roocs_utils.parameter.param_utils import (
    level_interval,
    level_series,
    time_components,
    time_interval,
    time_series,
)

ds = xr.open_mfdataset(tas_files, use_cftime=True, combine="by_coords")

outputs = subset(
        ds=ds,
        time=time_interval("2007-01-01T00:00:00", "2200-12-30T00:00:00"),
        time_components=time_components(month=["dec", "jan", "feb"]),
        output_type="xarray",
    )

print(f"There is only {len(outputs)} output.")
outputs[0]

There is only 1 output.

/home/docs/checkouts/readthedocs.org/user_builds/clisops/conda/stable/lib/python3.11/site-packages/clisops/ops/subset.py:120: UserWarning: "start_date" has been nudged to nearest valid time step in xarray object.
  result = subset_time(self.ds, **kwargs)
/home/docs/checkouts/readthedocs.org/user_builds/clisops/conda/stable/lib/python3.11/site-packages/clisops/ops/subset.py:120: UserWarning: "end_date" has been nudged to nearest valid time step in xarray object.
  result = subset_time(self.ds, **kwargs)

<xarray.Dataset>
Dimensions:    (lat: 2, time: 583, bnds: 2, lon: 2)
Coordinates:
    height     float64 1.5
  * lat        (lat) float64 -90.0 35.0
  * lon        (lon) float64 0.0 187.5
  * time       (time) object 2007-01-16 00:00:00 ... 2200-12-16 00:00:00
Dimensions without coordinates: bnds
Data variables:
    lat_bnds   (time, lat, bnds) float64 dask.array<chunksize=(71, 2, 2), meta=np.ndarray>
    lon_bnds   (time, lon, bnds) float64 dask.array<chunksize=(71, 2, 2), meta=np.ndarray>
    tas        (time, lat, lon) float32 dask.array<chunksize=(71, 2, 2), meta=np.ndarray>
    time_bnds  (time, bnds) object dask.array<chunksize=(71, 2), meta=np.ndarray>
Attributes: (12/29)
    institution:            Met Office Hadley Centre, Fitzroy Road, Exeter, D...
    institute_id:           MOHC
    experiment_id:          rcp85
    source:                 HadGEM2-ES (2009) atmosphere: HadGAM2 (N96L38); o...
    model_id:               HadGEM2-ES
    forcing:                GHG, SA, Oz, LU, Sl, Vl, BC, OC, (GHG = CO2, N2O,...
    ...                     ...
    title:                  HadGEM2-ES model output prepared for CMIP5 RCP8.5
    parent_experiment:      historical
    modeling_realm:         atmos
    realization:            1
    cmor_version:           2.5.0
    NCO:                    4.7.3

xarray.Dataset

• Dimensions:
  • lat: 2
  • time: 583
  • bnds: 2
  • lon: 2
• Coordinates: (4)
  • height
    ()
    float64
    1.5

    units :

    m

    axis :

    Z

    positive :

    up

    long_name :

    height

    standard_name :

    height

    array(1.5)

  • lat
    (lat)
    float64
    -90.0 35.0

    bounds :

    lat_bnds

    units :

    degrees_north

    axis :

    Y

    long_name :

    latitude

    standard_name :

    latitude

    array([-90.,  35.])

  • lon
    (lon)
    float64
    0.0 187.5

    bounds :

    lon_bnds

    units :

    degrees_east

    axis :

    X

    long_name :

    longitude

    standard_name :

    longitude

    array([  0. , 187.5])

  • time
    (time)
    object
    2007-01-16 00:00:00 ... 2200-12-...

    bounds :

    time_bnds

    axis :

    T

    long_name :

    time

    standard_name :

    time

    array([cftime.Datetime360Day(2007, 1, 16, 0, 0, 0, 0, has_year_zero=True),
           cftime.Datetime360Day(2007, 2, 16, 0, 0, 0, 0, has_year_zero=True),
           cftime.Datetime360Day(2007, 12, 16, 0, 0, 0, 0, has_year_zero=True),
           ..., cftime.Datetime360Day(2200, 1, 16, 0, 0, 0, 0, has_year_zero=True),
           cftime.Datetime360Day(2200, 2, 16, 0, 0, 0, 0, has_year_zero=True),
           cftime.Datetime360Day(2200, 12, 16, 0, 0, 0, 0, has_year_zero=True)],
          dtype=object)

• Data variables: (4)
  • lat_bnds
    (time, lat, bnds)
    float64
    dask.array<chunksize=(71, 2, 2), meta=np.ndarray>

    Array Chunk Bytes 18.22 kiB 2.34 kiB Shape (583, 2, 2) (75, 2, 2) Dask graph 9 chunks in 42 graph layers Data type float64 numpy.ndarray

  • lon_bnds
    (time, lon, bnds)
    float64
    dask.array<chunksize=(71, 2, 2), meta=np.ndarray>

    Array Chunk Bytes 18.22 kiB 2.34 kiB Shape (583, 2, 2) (75, 2, 2) Dask graph 9 chunks in 42 graph layers Data type float64 numpy.ndarray

  • tas
    (time, lat, lon)
    float32
    dask.array<chunksize=(71, 2, 2), meta=np.ndarray>

    standard_name :

    air_temperature

    long_name :

    Near-Surface Air Temperature

    comment :

    near-surface (usually, 2 meter) air temperature.

    units :

    K

    original_name :

    mo: m01s03i236

    cell_methods :

    time: mean

    cell_measures :

    area: areacella

    history :

    2010-12-04T13:50:30Z altered by CMOR: Treated scalar dimension: 'height'. 2010-12-04T13:50:30Z altered by CMOR: replaced missing value flag (-1.07374e+09) with standard missing value (1e+20).

    associated_files :

    baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation gridspecFile: gridspec_atmos_fx_HadGEM2-ES_rcp85_r0i0p0.nc areacella: areacella_fx_HadGEM2-ES_rcp85_r0i0p0.nc

    Array Chunk Bytes 9.11 kiB 1.17 kiB Shape (583, 2, 2) (75, 2, 2) Dask graph 9 chunks in 29 graph layers Data type float32 numpy.ndarray

  • time_bnds
    (time, bnds)
    object
    dask.array<chunksize=(71, 2), meta=np.ndarray>

    Array Chunk Bytes 9.11 kiB 1.17 kiB Shape (583, 2) (75, 2) Dask graph 9 chunks in 29 graph layers Data type object numpy.ndarray

• Indexes: (3)
  • lat
    PandasIndex

    PandasIndex(Float64Index([-90.0, 35.0], dtype='float64', name='lat'))

  • lon
    PandasIndex

    PandasIndex(Float64Index([0.0, 187.5], dtype='float64', name='lon'))

  • time
    PandasIndex

    PandasIndex(CFTimeIndex([2007-01-16 00:00:00, 2007-02-16 00:00:00, 2007-12-16 00:00:00,
                 2008-01-16 00:00:00, 2008-02-16 00:00:00, 2008-12-16 00:00:00,
                 2009-01-16 00:00:00, 2009-02-16 00:00:00, 2009-12-16 00:00:00,
                 2010-01-16 00:00:00,
                 ...
                 2197-12-16 00:00:00, 2198-01-16 00:00:00, 2198-02-16 00:00:00,
                 2198-12-16 00:00:00, 2199-01-16 00:00:00, 2199-02-16 00:00:00,
                 2199-12-16 00:00:00, 2200-01-16 00:00:00, 2200-02-16 00:00:00,
                 2200-12-16 00:00:00],
                dtype='object', length=583, calendar='360_day', freq='None'))

• Attributes: (29)

  institution :

  Met Office Hadley Centre, Fitzroy Road, Exeter, Devon, EX1 3PB, UK, (http://www.metoffice.gov.uk)

  institute_id :

  MOHC

  experiment_id :

  rcp85

  source :

  HadGEM2-ES (2009) atmosphere: HadGAM2 (N96L38); ocean: HadGOM2 (lat: 1.0-0.3 lon: 1.0 L40); land-surface/vegetation: MOSES2 and TRIFFID; tropospheric chemistry: UKCA; ocean biogeochemistry: diat-HadOCC

  model_id :

  HadGEM2-ES

  forcing :

  GHG, SA, Oz, LU, Sl, Vl, BC, OC, (GHG = CO2, N2O, CH4, CFCs)

  parent_experiment_id :

  historical

  parent_experiment_rip :

  r1i1p1

  branch_time :

  52560.0

  contact :

  chris.d.jones@metoffice.gov.uk, michael.sanderson@metoffice.gov.uk

  history :

  Mon Mar 9 09:10:39 2020: ncks -d lat,,,100 -d lon,,,100 -v tas /badc/cmip5/data/cmip5/output1/MOHC/HadGEM2-ES/rcp85/mon/atmos/Amon/r1i1p1/latest/tas/tas_Amon_HadGEM2-ES_rcp85_r1i1p1_200512-203011.nc test_data/badc/cmip5/data/cmip5/output1/MOHC/HadGEM2-ES/rcp85/mon/atmos/Amon/r1i1p1/latest/tas/tas_Amon_HadGEM2-ES_rcp85_r1i1p1_200512-203011.nc MOHC pp to CMOR/NetCDF convertor (version 1.5.1) 2010-12-04T13:50:10Z CMOR rewrote data to comply with CF standards and CMIP5 requirements.

  references :

  Bellouin N. et al, (2007) Improved representation of aerosols for HadGEM2. Meteorological Office Hadley Centre, Technical Note 73, March 2007; Collins W.J. et al, (2008) Evaluation of the HadGEM2 model. Meteorological Office Hadley Centre, Technical Note 74,; Johns T.C. et al, (2006) The new Hadley Centre climate model HadGEM1: Evaluation of coupled simulations. Journal of Climate, American Meteorological Society, Vol. 19, No. 7, pages 1327-1353.; Martin G.M. et al, (2006) The physical properties of the atmosphere in the new Hadley Centre Global Environmental Model, HadGEM1 - Part 1: Model description and global climatology. Journal of Climate, American Meteorological Society, Vol. 19, No.7, pages 1274-1301.; Ringer M.A. et al, (2006) The physical properties of the atmosphere in the new Hadley Centre Global Environmental Model, HadGEM1 - Part 2: Aspects of variability and regional climate. Journal of Climate, American Meteorological Society, Vol. 19, No. 7, pages 1302-1326.

  initialization_method :

  1

  physics_version :

  1

  tracking_id :

  948b8aa2-4b1f-422a-921f-4515fcf9860b

  mo_runid :

  ajnji

  product :

  output

  experiment :

  RCP8.5

  frequency :

  mon

  creation_date :

  2010-12-04T13:50:30Z

  Conventions :

  CF-1.4

  project_id :

  CMIP5

  table_id :

  Table Amon (12 November 2010) 6e535ddfacb41fb7a252f4862fdc5766

  title :

  HadGEM2-ES model output prepared for CMIP5 RCP8.5

  parent_experiment :

  historical

  modeling_realm :

  atmos

  realization :

  1

  cmor_version :

  2.5.0

  NCO :

  4.7.3