Companion notebook: remote JSON/Kerchunk CMIP access

Companion notebook: remote JSON/Kerchunk CMIP access#

Demonstration companion to New Tools for Accessing CMIP Data at NERSC and Beyond#

Climate and Weather Seminar Series (CWSS)#

Sasha Ames, Stephen Po-Chedley & Tom Vo#

Lawrence Livermore National Laboratory

September 24, 2026 · 11:00 AM

This work is performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract No. DE-AC52-07NA27344.

Notebook overview#

This demonstration uses Kerchunk references to access CMIP6 data remotely, processes it with xCDAT, and compares that workflow with a local NERSC analysis. It is the remote JSON/Kerchunk demonstration component of the companion CWSS presentation.

  1. Select CMIP6 surface-temperature datasets and ECS metadata

  2. Open remote Kerchunk references and calculate warming trends

  3. Repeat the calculation with local NERSC data

  4. Compare scientific results and illustrative end-to-end timings

Environment and access requirements#

Create a conda environment containing xcdat, matplotlib, cartopy, fsspec, aiohttp, requests, zarr, jupyter, and ipykernel. The local NERSC comparison additionally requires xsearch, which is not included with xCDAT:

conda create -n cwss-kerchunk -c conda-forge xcdat matplotlib cartopy fsspec aiohttp requests zarr jupyter ipykernel
conda activate cwss-kerchunk
python -m pip install git+https://github.com/PCMDI/xsearch.git
python -m ipykernel install --user --name cwss-kerchunk --display-name "CWSS Kerchunk"

The remote portion uses ORNL-hosted Kerchunk references. The local portion requires NERSC data access and xsearch. Read the accompanying README.md for input provenance and reproducibility notes.

Imports#

[20]:

import json
import time
from collections.abc import Mapping, Sequence
from typing import Any, Literal

import cartopy.crs as ccrs
import fsspec
import matplotlib.pyplot as plt
import numpy as np
import pooch
import xarray as xr
import xcdat as xc
from cartopy.util import add_cyclic_point

Kerchunk workflow helpers#

These helper functions are defined in the notebook so the demonstration can run as a self-contained file.

[21]:

KERCHUNK_SITE_URLS = {
    "nersc": "https://g-eba899.6b7bd8.0ec8.data.globus.org/kerchunk",
    "ornl": "https://esgf-node.ornl.gov/thredds/fileServer/user_pub_work/kerchunk",
}


def find_json_files(
    catalog: Mapping[str, Mapping[str, Any]], **facets: Any
) -> dict[str, Mapping[str, Any]]:
    """Return catalog entries whose facets match every supplied value."""
    return {
        path: metadata
        for path, metadata in catalog.items()
        if all(metadata.get(facet) == value for facet, value in facets.items())
    }


def open_kerchunk_reference(
    reference_file: str, site: Literal["nersc", "ornl"] = "ornl"
) -> xr.Dataset:
    """Open a remotely hosted Kerchunk reference as an xarray dataset."""
    try:
        reference_url = f"{KERCHUNK_SITE_URLS[site]}/{reference_file}"
    except KeyError as error:
        supported_sites = ", ".join(KERCHUNK_SITE_URLS)
        raise ValueError(
            f"Unsupported Kerchunk site {site!r}. Choose one of: {supported_sites}."
        ) from error

    filesystem = fsspec.filesystem(
        "reference",
        fo=reference_url,
        remote_options={"asynchronous": True},
        remote_protocol="https",
        asynchronous=True,
    )
    return xr.open_dataset(
        filesystem.get_mapper(""), engine="zarr", consolidated=False
    )


def calculate_model_trends(
    dataset: xr.Dataset,
    variable_id: str,
    target_grid: xr.Dataset,
    expected_year_count: int,
) -> tuple[float, xr.DataArray]:
    """Calculate native-grid global and regridded cell-wise trends in K decade-1."""
    dataset = dataset.bounds.add_missing_bounds()
    if "height" in dataset.coords:
        dataset = dataset.drop_vars("height")

    annual_dataset = dataset.temporal.group_average(variable_id, freq="year")
    regridded_dataset = annual_dataset.regridder.horizontal(
        variable_id, target_grid, tool="regrid2"
    )
    decimal_year = regridded_dataset.time.dt.decimal_year
    if len(decimal_year) != expected_year_count:
        raise ValueError(
            "Annual time coordinate has "
            f"{len(decimal_year)} samples; expected {expected_year_count}."
        )

    regridded_dataset = regridded_dataset.assign_coords(time=decimal_year)
    trend_dataset = regridded_dataset[variable_id].polyfit(dim="time", deg=1)
    grid_cell_trend = trend_dataset.polyfit_coefficients.sel(degree=1) * 10

    global_mean = annual_dataset.spatial.average(variable_id)[variable_id]
    global_trend, _ = np.polyfit(decimal_year, global_mean.values, 1)

    return float(global_trend * 10), grid_cell_trend


def select_available_models(
    catalog: Mapping[str, Mapping[str, Any]],
    ecs_data: Mapping[str, float],
    excluded_models: Sequence[str] = (),
) -> list[str]:
    """Return sorted catalog models with ECS data that are not excluded."""
    excluded_model_set = set(excluded_models)
    catalog_models = {metadata["model"] for metadata in catalog.values()}

    return sorted((catalog_models & set(ecs_data)) - excluded_model_set)

The imports below support the remote workflow. xsearch is imported in the NERSC-local section so the remote portion can run without that optional dependency.

Analysis parameters#

[22]:

ANALYSIS_PERIOD = slice("1981-01-01", "2014-12-30")
LATITUDES = np.arange(-88.75, 90, 2.5)
LONGITUDES = np.arange(1.25, 360, 2.5)
EXPECTED_YEAR_COUNT = 34
TIMING_RUNS = 3
VARIABLE_ID = "tas"
EXCLUDED_MODELS = {"CIESM", "GFDL-CM4", "GFDL-ESM4", "NorCPM1"}
KERCHUNK_CATALOG_URL = (
    "https://raw.githubusercontent.com/xCDAT/xcdat-data/"
    "e31bf6cdfd478550e9a284e6d17ef35edce5ee03/resources/kerchunk_list.json"
)
KERCHUNK_CATALOG_HASH = "sha256:fe1137950015073fa6bd586d61bfda426fad6053284a9e74d7034b451e663028"
ECS_DATA_URL = (
    "https://raw.githubusercontent.com/xCDAT/xcdat-data/"
    "a727575e259499ba21d596d7ddabd8433250aac7/resources/ecsdata.json"
)
ECS_DATA_HASH = "sha256:4c06aaa54fbdf225347bc16482ab3009724246a72508578fa51832060eb481b4"

Load equilibrium climate sensitivity data#

[23]:

ecs_data_path = pooch.retrieve(
    url=ECS_DATA_URL,
    known_hash=ECS_DATA_HASH,
    path=pooch.os_cache("xcdat"),
    fname="cwss-2026-ecsdata.json",
)
with open(ecs_data_path) as file:
    ecs_data = json.load(file)

Create a common target grid#

[24]:

latitude_axis = xc.create_axis("lat", LATITUDES)
longitude_axis = xc.create_axis("lon", LONGITUDES)
target_grid = xc.create_grid(x=longitude_axis, y=latitude_axis)

Retrieve and select remote CMIP6 references#

Download the pinned Kerchunk catalog once with Pooch, then filter it to historical monthly near-surface air temperature (tas) datasets. Retain models with ECS metadata and omit models whose available data do not support this demonstration.

[25]:

catalog_path = pooch.retrieve(
    url=KERCHUNK_CATALOG_URL,
    known_hash=KERCHUNK_CATALOG_HASH,
    path=pooch.os_cache("xcdat"),
    fname="cwss-2026-kerchunk_list.json",
)
with open(catalog_path) as file:
    kerchunk_catalog = json.load(file)

remote_catalog = find_json_files(
    kerchunk_catalog,
    experiment="historical",
    variable=VARIABLE_ID,
    frequency="mon",
    cmipTable="Amon",
)
model_names = select_available_models(
    remote_catalog, ecs_data, excluded_models=EXCLUDED_MODELS
)
print(f"Selected {len(model_names)} models for analysis.")

Selected 38 models for analysis.

Remote Kerchunk workflow#

Run the processing workflow three times after the catalog is available locally and report the median duration. The timing therefore measures remote reference access, data loading, and xCDAT processing without including the one-time catalog download; it remains sensitive to network and cache conditions.

[26]:

# Timing begins in the processing cell.

Local NERSC workflow#

Repeat the same analysis against locally accessible NERSC datasets. This section requires NERSC access and xsearch; it is not expected to run in a general public environment.

[32]:

# Timing begins after local dataset discovery.

Discover local CMIP6 data with xsearch#

[33]:

import xsearch as xs

local_catalog = xs.findPaths(
    experiment="historical",
    variable=VARIABLE_ID,
    frequency="mon",
    cmipTable="Amon",
    mip_era="CMIP6",
)

Compare workflow timings#

Each timing is the median of three runs of data loading and analysis after catalog discovery or reference selection. They are examples from one environment, not portable performance benchmarks.

[38]:

local_elapsed_time = time.strftime(
    "%M:%S", time.gmtime(local_elapsed_median_seconds)
)
local_elapsed_times = [
    time.strftime("%M:%S", time.gmtime(seconds))
    for seconds in local_elapsed_seconds
]
print(
    f"Remote workflow median ({TIMING_RUNS} runs): {remote_elapsed_time}; "
    f"local NERSC workflow median ({TIMING_RUNS} runs): {local_elapsed_time}."
    f"\nRemote runs: {', '.join(remote_elapsed_times)}."
    f"\nLocal runs: {', '.join(local_elapsed_times)}."
)

Remote workflow median (3 runs): 03:20; local NERSC workflow median (3 runs): 02:40.
Remote runs: 03:20, 03:13, 03:21.
Local runs: 03:10, 02:38, 02:40.

Timings can vary between runs due to network conditions, filesystem caching, system load, and remote data-service availability.