TidyRun Serialization Guide

Overview

This page covers serialization only: storing and retrieving Python objects (serialize, deserialize, metadata, encoders, and LazyDict).

For deferred-compute APIs (DAG, Job, ParametrizedJob) and DAG execution patterns, see the dedicated DAG Guide.

The TidyRun serialization framework provides a comprehensive, extensible system for storing and retrieving Python objects, including nested dictionaries, DataFrames, Series, and arbitrary Python objects, using a filesystem hierarchy.

Key Features

• Type-Aware Encoding: Automatically selects the best format (folder, parquet, HDF5, JSON, or pickle) based on value type
• Metadata Sidecars: Each output is accompanied by a .tidyrun metadata file recording encoding format, version, and checksum
• Lazy Evaluation: Directories deserialize into LazyDict objects that load values on-demand on each access
• Unified Path Handling: Paths are normalized via cloudpathlib.AnyPath for local and cloud-backed locations
• Optional S3 Support: s3://... locations work when the boto3 extra is installed
• Recursive Concatenation: LazyDict.concat() method provides pandas-style aggregation across nested structures
• Parallel Leaf Loading: LazyDict.concat(max_workers=...) can load selected leaf values concurrently
• Fallback Chain: Intelligent fallback routing (e.g., parquet → HDF5 when parquet encoding fails)
• Extensible Pipeline: Users can provide custom encoder sequences and compose encoders
• Direct Import Path: Use tidyrun.serialization for the serialization API

Quick Start

Basic Serialization

from tidyrun import serialize, deserialize
import pandas as pd

# Serialize a simple value
data = {"experiment": "run_1", "results": 42}
serialize(data, "./output/my_result")

# Deserialize back (returns LazyDict for folders)
loaded = deserialize("./output/my_result")
print(loaded["experiment"])  # "run_1" — loaded on access
print(loaded["results"])     # 42

DataFrame Support

import pandas as pd

df = pd.DataFrame({"x": [1, 2, 3], "y": ["a", "b", "c"]})

# Serializes to Parquet (with metadata sidecar)
serialize(df, "./output/dataframe")

# Deserializes directly as DataFrame
loaded_df = deserialize("./output/dataframe")

Nested Data with LazyDict

# Nested dictionary
nested = {
    "run_1": {"metrics": pd.DataFrame(...)},
    "run_2": {"metrics": pd.DataFrame(...)},
}

serialize(nested, "./outputs/runs")

# Deserialize as LazyDict — values load on access
loaded = deserialize("./outputs/runs")

# Lazy access
metrics_1 = loaded["run_1"]["metrics"]  # Loaded only when accessed

# Recursive materialization
full_dict = loaded.to_dict()  # Materializes all nested values

DAG APIs

For deferred-compute APIs (DAG, Job, ParametrizedJob) and local multithreaded DAG evaluation, see the dedicated DAG page: DAG Guide.

Optional S3 Support

TidyRun can read and write s3://bucket/prefix/... locations when the optional S3 dependency is installed:

pip install tidyrun[s3]

Under the hood, S3 serialization stages through a local temporary directory and then uploads the generated files. Deserialization downloads the object tree to a temporary local directory and then reuses the normal local loader.

For tests and local development, the S3 round-trip test suite uses the moto mock backend.

Concatenation Across Runs

# Given a nested structure like:
# outputs/
#   run_1/
#     metrics.parquet
#   run_2/
#     metrics.parquet
#   ...

loaded = deserialize("./outputs")

# Concatenate all DataFrames, keyed by run ID
combined = loaded.concat(names=["run_id"])
# Returns: DataFrame with multi-index (run_id) and all metrics stacked

# With transformation (e.g., add timestamp)
combined = loaded.concat(
    names=["run_id"],
    transform=lambda df: df.assign(loaded_at=pd.Timestamp.now())
)

# Transform to scalar values (wrapped as a one-row "value" column)
totals = loaded.concat(
    names=["run_id"],
    transform=lambda df: df["metric"].sum(),
)

# With filtering (load only specific runs)
combined = loaded.concat(
    names=["run_id"],
    select=lambda path: path[0] in ["run_1", "run_2"]
)

# Parallel loading of leaf values (useful when leaf files are large)
combined = loaded.concat(
    names=["run_id"],
    max_workers=8,
)

# select(path) argument:
# - path: tuple of keys to the leaf (e.g. ("run_1", "metrics"))
# Note: select is evaluated at multiple depths while traversing nested folders.
# If you index deeper elements like path[1], guard with len(path) first.

Quick Reference

One-Minute Overview

TidyRun serialization provides save/load behavior with automatic format selection:

from tidyrun import serialize, deserialize

serialize({"df": my_dataframe, "config": settings}, "./output/result")

result = deserialize("./output/result")
df = result["df"]

What Gets Stored Where

Type	Format	File	Notes
dict	Folder tree	key/name/	Keys encoded via TOML; values recurse
pd.DataFrame	Parquet	*.parquet	Falls back to HDF5 on failure
pd.Series	Parquet	*.parquet	Falls back to HDF5 on failure
Scalar (int, str, bool, float, date, datetime, time)	JSON	*.json	JSON-serializable scalars
Other objects	Pickle	*.pickle	Last resort

Every output gets a .tidyrun metadata sidecar recording the selected format and checksum.

LazyDict in Practice

Directories deserialize as LazyDict objects for on-demand loading:

result = deserialize("./large_output")
value = result["key"]
full_dict = result.to_dict()

Encoder Fallback Chain

If one encoder cannot serialize a value, the next compatible encoder is tried:

1. dict -> folder tree
2. DataFrame -> parquet, then HDF5
3. Series -> parquet, then HDF5
4. JSON-serializable values -> JSON
5. Everything else -> pickle

Common Patterns

Save experiment results:

serialize(
    {
        "config": hyperparams,
        "metrics": pd.DataFrame(training_log),
        "model": trained_model,
    },
    "./experiments/exp_001",
)

Load and compare runs:

runs = deserialize("./experiments")
comparison_table = runs.concat(names=["exp_id"])

API Summary

Function	Purpose
serialize(value, target, encoders=None)	Save a value to disk and return ChecksumInfo
deserialize(source, encoders=None)	Load a value from disk
LazyDict.to_dict()	Materialize a nested LazyDict
LazyDict.concat(names, transform, select, max_workers=None)	Recursively concatenate DataFrames with optional parallel leaf loading
encode_key(key)	Encode a Python type to a filename-safe key
decode_key(name)	Decode a stored key name

Architecture

Module Structure

src/tidyrun/serialization/
├── __init__.py          # Public API exports
├── types.py             # Type definitions, exceptions, constants
├── paths.py             # Path/location helpers
├── metadata.py          # Metadata I/O and format mapping
├── encoders.py          # Encoder implementations (dict, parquet, hdf5, etc.)
├── lazy_dict.py         # LazyDict class
└── api.py               # Main serialize/deserialize functions

Encoder Pipeline

The default encoder pipeline tries encoders in this order:

1. dict-folder: Maps dictionaries to directory trees (keys become folder names)
2. dataframe-parquet: Stores DataFrames as .parquet files
3. series-parquet: Stores pandas Series as .parquet files
4. pandas-hdf5: Fallback for DataFrames/Series (HDF5 format with key "data")
5. fallback-json: JSON serialization for scalar types (int, float, str, list, dict, etc.)
6. fallback-pickle: Last resort for arbitrary Python objects

Encoders are tried in order; the first whose predicate returns True is used.

Fallback Mechanism

When an encoder fails (e.g., parquet cannot serialize a multi-index DataFrame), it raises GoToNextEncoderException to signal "skip me and try the next one." This allows:

• DataFrame with multi-index → fails parquet → tries HDF5 ✓
• Series when parquet engine unavailable → fails parquet → tries HDF5 ✓
• Custom object → fails all structured formats → falls back to pickle ✓

Metadata Sidecars

Every serialized value gets a .tidyrun metadata file:

# output.tidyrun
version = 1
encoding = "dataframe-parquet"
suffix = ".parquet"

[checksum]
algorithm = "sha256"
digest = "..."

This metadata:

• Tracks the encoding format used
• Enables schema versioning for future compatibility
• Allows deserialization without requiring file extension guessing

API Reference

Core Functions

tidyrun.serialization.api.serialize(value, target, encoders=None)

Serialize a Python value to disk using the configured encoder pipeline.

Parameters:

Name	Type	Description	Default
value	Any	The value to serialize (dict, DataFrame, Series, scalar, etc.).	required
target	str | Path | CloudPath	Where to write the output. Extension-free; the encoder appends the appropriate suffix. Accepts local paths or s3:// URIs (requires the optional boto3 dependency installed via pip install tidyrun[s3]).	required
encoders	Iterable[EncoderSpec] | None	Custom encoder pipeline. Defaults to :func:default_encoders.	None

Returns:

Type	Description
ChecksumInfo	Checksum (algorithm, digest) for the serialized payload.

Raises:

Type	Description
TidyRunSerializationError	When no encoder matches the value type.
NotImplementedError	When an S3 target is requested without the optional dependency.

Source code in src/tidyrun/serialization/api.py

def serialize(
    value: Any,
    target: str | Path | CloudPath,
    encoders: Iterable[EncoderSpec] | None = None,
) -> ChecksumInfo:
    """Serialize a Python value to disk using the configured encoder pipeline.

    Parameters
    ----------
    value :
        The value to serialize (dict, DataFrame, Series, scalar, etc.).
    target :
        Where to write the output. Extension-free; the encoder appends the
        appropriate suffix. Accepts local paths or ``s3://`` URIs (requires
        the optional ``boto3`` dependency installed via ``pip install tidyrun[s3]``).
    encoders :
        Custom encoder pipeline. Defaults to :func:`default_encoders`.

    Returns
    -------
    ChecksumInfo
        Checksum (``algorithm``, ``digest``) for the serialized payload.

    Raises
    ------
    TidyRunSerializationError
        When no encoder matches the value type.
    NotImplementedError
        When an S3 target is requested without the optional dependency.
    """
    path = AnyPath(target)
    encoder_list = tuple(default_encoders() if encoders is None else encoders)
    selected_encoder: EncoderSpec | None = None
    checksum: ChecksumInfo | None = None
    for encoder in encoder_list:
        if not encoder.predicate(value):
            continue

        try:
            checksum = encoder.serializer(value, path)
        except GoToNextEncoderException:
            continue

        selected_encoder = encoder
        break

    if selected_encoder is None:
        raise TidyRunSerializationError(
            f"No encoder found for value of type {type(value).__name__!r}"
        )

    assert checksum is not None, "Encoder did not return checksum"

    symlink_target: str | None = None
    if selected_encoder.name == "symlink":
        import os

        symlink_target = cast(str, os.fspath(value))

    write_metadata(
        path,
        encoding=selected_encoder.name,
        suffix=suffix_for_encoder(selected_encoder.name),
        checksum=checksum,
        symlink_target=symlink_target,
    )
    return checksum

tidyrun.serialization.api.deserialize(source, encoders=None)

Deserialize a value from disk using metadata to determine the format.

Directories encoded as dict-folder are returned as :class:LazyDict objects whose values are loaded on first access.

Parameters:

Name	Type	Description	Default
source	str | Path | CloudPath	Location to read from. Accepts local paths or s3:// URIs (requires the optional boto3 dependency installed via pip install tidyrun[s3]).	required
encoders	Iterable[EncoderSpec] | None	Custom encoder pipeline. Defaults to :func:default_encoders.	None

Returns:

Type	Description
Any	LazyDict for dict-folder outputs; pd.DataFrame / pd.Series for tabular outputs; the original Python object for scalar or pickle outputs.

Raises:

Type	Description
TidyRunDeserializationError	When metadata is missing, the encoder name is unknown, or the payload cannot be read.

Source code in src/tidyrun/serialization/api.py

def deserialize(
    source: str | Path | CloudPath, encoders: Iterable[EncoderSpec] | None = None
) -> Any:
    """Deserialize a value from disk using metadata to determine the format.

    Directories encoded as ``dict-folder`` are returned as :class:`LazyDict`
    objects whose values are loaded on first access.

    Parameters
    ----------
    source :
        Location to read from. Accepts local paths or ``s3://`` URIs (requires
        the optional ``boto3`` dependency installed via ``pip install tidyrun[s3]``).
    encoders :
        Custom encoder pipeline. Defaults to :func:`default_encoders`.

    Returns
    -------
    Any
        ``LazyDict`` for dict-folder outputs; ``pd.DataFrame`` / ``pd.Series``
        for tabular outputs; the original Python object for scalar or pickle
        outputs.

    Raises
    ------
    TidyRunDeserializationError
        When metadata is missing, the encoder name is unknown, or the payload
        cannot be read.
    """
    path = AnyPath(source)
    encoder_list = tuple(default_encoders() if encoders is None else encoders)
    if not metadata_exists(path):
        return _deserialize_without_metadata(path, encoder_list)

    metadata = read_metadata(path)
    encoder_name = metadata["encoding"]
    encoder_map = encoder_by_name(encoder_list)
    encoder = encoder_map.get(encoder_name)
    if encoder is None:
        raise TidyRunDeserializationError(
            f"Unknown encoder in metadata: {encoder_name!r}"
        )

    checksum = metadata.get("checksum")
    assert checksum is None or isinstance(checksum, ChecksumInfo)

    # Symlink metadata can point to another serialized location.
    if encoder_name == "symlink":
        target = metadata.get("symlink_target")
        if isinstance(target, str):
            if "://" in target:
                target_path = AnyPath(target)
            else:
                target_path = AnyPath(path.parent / Path(target))
            return deserialize(target_path, encoders=encoder_list)

    return encoder.deserializer(path, checksum)

LazyDict

tidyrun.serialization.lazy_dict.LazyDict

Bases: Mapping[Key, Any]

Dictionary-like object that loads each child value on first access.

Source code in src/tidyrun/serialization/lazy_dict.py

class LazyDict(Mapping[Key, Any]):
    """Dictionary-like object that loads each child value on first access."""

    def __init__(
        self,
        serialized_path: Path | CloudPath,
        checksum: ChecksumInfo | None = None,
    ) -> None:
        self.__serialized_path__ = serialized_path
        self.__checksum__ = checksum

    """Marker that this LazyDict can be serialized as a symlink reference."""

    def __fspath__(self) -> str:
        """Return the path this LazyDict was loaded from.

        This enables symlink serialization via os.fspath() protocol.
        """
        return str(self.__serialized_path__)

    def _encoded_entries(self) -> list[str]:
        if not self.__serialized_path__.is_dir():
            raise TidyRunDeserializationError(
                f"Expected directory, got: {self.__serialized_path__}"
            )

        serialized_path = cast(Any, self.__serialized_path__)
        entries: list[str] = []
        metadata_named: set[str] = set()

        metadata_files = sorted(
            serialized_path.glob(f"*{TIDYRUN_METADATA_EXTENSION}"), key=lambda p: p.name
        )
        for metadata_file in metadata_files:
            encoded_name = metadata_file.name[: -len(TIDYRUN_METADATA_EXTENSION)]
            entries.append(encoded_name)
            metadata_named.add(encoded_name)

        if metadata_files:
            # When metadata files are present also include bare subdirectories
            # not already covered by a metadata sidecar.  This handles output
            # trees where parametrised-job group directories sit alongside
            # individual job outputs that do have metadata sidecars.
            for entry in sorted(serialized_path.iterdir(), key=lambda p: p.name):
                if entry.is_dir() and entry.name not in metadata_named:
                    entries.append(entry.name)
        else:
            # No metadata — fall back to scanning all payload candidates.
            for entry in sorted(serialized_path.iterdir(), key=lambda p: p.name):
                encoded_name = _decoded_name_from_payload_name(entry.name)
                if encoded_name is None:
                    continue
                entries.append(encoded_name)

        return entries

    def __getitem__(self, key: Key) -> Any:
        from .api import deserialize

        if isinstance(key, str) and ("/" in key or "\\" in key):
            # This is a convenience shortcut that allows users to
            # load nested data by passing a path-like key
            name = key
        else:
            name = encode_key(key)

        key_dir = self.__serialized_path__ / name
        return deserialize(key_dir)

    def __iter__(self) -> Iterator[Key]:
        for encoded_name in self._encoded_entries():
            yield decode_key(encoded_name)

    def __len__(self) -> int:
        return sum(1 for _ in self)

    def _ipython_key_completions_(self) -> list[str]:
        """Return string keys for bracket-completion in IPython/Jupyter."""
        return [key for key in self if isinstance(key, str)]

    def to_dict(self) -> dict[Key, Any]:
        result: dict[Key, Any] = {}
        for key in self:
            value = self[key]
            if isinstance(value, LazyDict):
                result[key] = value.to_dict()
            else:
                result[key] = value
        return result

    def concat(
        self,
        names: list[str | None] | None = None,
        transform: Callable[[Any], Any] | None = None,
        select: Callable[[tuple[Key, ...]], bool] | None = None,
        max_workers: int | None = None,
    ) -> Any:
        """Concatenate leaf values from a nested LazyDict.

        Parameters
        ----------
        names :
            Names for the MultiIndex levels built from nested keys. Pass
            ``None`` for a level to drop it from the index and concatenate
            without that level.
        transform :
            Optional function applied to each selected leaf value before
            concatenation. The result may be a ``pd.DataFrame``,
            ``pd.Series``, or a scalar (which is wrapped as a one-row
            ``"value"`` column).
        select :
            Optional predicate called as ``select(path)`` where ``path`` is a
            tuple of keys leading to a node (e.g. ``("run_001", "metrics")``).
            Evaluated before loading child values, so filtered paths are never
            deserialized.
        max_workers :
            When set, leaf values are loaded in parallel using a
            :class:`~concurrent.futures.ThreadPoolExecutor` with this many
            worker threads. Useful when leaves are large files (e.g. Parquet)
            and I/O dominates. When ``None`` (default), loading is sequential.

        Returns
        -------
        pd.DataFrame
            Result of ``pd.concat`` with a MultiIndex built from the selected
            leaf paths.

        Raises
        ------
        ValueError
            When a ``LazyDict`` node is encountered but ``names`` does not have
            enough levels to reach leaf values, or when no values are selected.
        """
        import pandas as pd  # pyright: ignore[reportMissingTypeStubs]

        pd = cast(Any, pd)

        def _identity(value: Any) -> Any:
            return value

        def _select_all(_path: tuple[Key, ...]) -> bool:
            return True

        selected_transform: Callable[[Any], Any] = (
            transform if transform is not None else _identity
        )
        selected_filter: Callable[[tuple[Key, ...]], bool] = (
            select if select is not None else _select_all
        )

        # Phase 1 (serial): walk the tree to discover leaf (path, node, key)
        # without loading the leaf values.  Intermediate LazyDict nodes are
        # detected cheaply via metadata peek; only directories that resolve to
        # another LazyDict are recursed into.
        leaf_refs: list[tuple[tuple[Key, ...], LazyDict, Key]] = []

        def _collect_refs(node: LazyDict, prefix: tuple[Key, ...]) -> None:
            for key in node:
                current_path = prefix + (key,)
                if not selected_filter(current_path):
                    continue

                encoded_name = encode_key(key)
                if _entry_is_lazy_dict(node.__serialized_path__, encoded_name):
                    if names is not None and len(current_path) >= len(names):
                        if transform is not None:
                            leaf_refs.append((current_path, node, key))
                            continue
                        raise ValueError(
                            f"Encountered LazyDict at depth {len(current_path)}, "
                            f"but names only has {len(names)} levels. "
                            f"Provide more levels in names to reach leaf values."
                        )
                    child = node[key]
                    _collect_refs(child, current_path)
                else:
                    leaf_refs.append((current_path, node, key))

        _collect_refs(self, ())

        if not leaf_refs:
            raise ValueError("No values selected for concatenation")

        # Phase 2: load leaf values — parallel when max_workers is set.
        def _load(ref: tuple[tuple[Key, ...], LazyDict, Key]) -> Any:
            _, node, key = ref
            value = node[key]
            transformed = selected_transform(value)
            if isinstance(transformed, LazyDict):
                raise ValueError(
                    f"Transform returned LazyDict for path {ref[0]}, but "
                    f"concat expects leaf values. Adjust transform or provide "
                    f"more levels in names to reach leaf values."
                )
            if isinstance(transformed, (pd.Series, pd.DataFrame)):
                return transformed
            return pd.Series([transformed], name="value").to_frame()

        if max_workers is not None and len(leaf_refs) > 1:
            with ThreadPoolExecutor(max_workers=max_workers) as executor:
                frames = list(executor.map(_load, leaf_refs))
        else:
            frames = [_load(ref) for ref in leaf_refs]

        # Phase 3: apply transform and build frames.
        keys: list[tuple[Key, ...]] = []
        values: list[Any] = []
        for (current_path, _, _key), frame in zip(leaf_refs, frames):
            keys.append(current_path)
            values.append(frame)

        # Handle None values in names by dropping those levels
        if names is not None and any(n is None for n in names):
            # Create filtered keys and names, keeping only non-None levels
            keep_indices = [i for i, n in enumerate(names) if n is not None]
            filtered_names = [names[i] for i in keep_indices]

            if not filtered_names:
                # All levels are dropped, just concatenate without keys
                return pd.concat(values)
            else:
                # Some levels remain
                filtered_keys = [tuple(k[i] for i in keep_indices) for k in keys]
                return pd.concat(values, keys=filtered_keys, names=filtered_names)
        else:
            return pd.concat(values, keys=keys, names=names)

    def __repr__(self) -> str:
        return f"LazyDict(keys={list(self)!r})"

__getitem__(key)

Source code in src/tidyrun/serialization/lazy_dict.py

def __getitem__(self, key: Key) -> Any:
    from .api import deserialize

    if isinstance(key, str) and ("/" in key or "\\" in key):
        # This is a convenience shortcut that allows users to
        # load nested data by passing a path-like key
        name = key
    else:
        name = encode_key(key)

    key_dir = self.__serialized_path__ / name
    return deserialize(key_dir)

to_dict()

Source code in src/tidyrun/serialization/lazy_dict.py

def to_dict(self) -> dict[Key, Any]:
    result: dict[Key, Any] = {}
    for key in self:
        value = self[key]
        if isinstance(value, LazyDict):
            result[key] = value.to_dict()
        else:
            result[key] = value
    return result

concat(names=None, transform=None, select=None, max_workers=None)

Concatenate leaf values from a nested LazyDict.

Parameters:

Name	Type	Description	Default
names	list[str | None] | None	Names for the MultiIndex levels built from nested keys. Pass None for a level to drop it from the index and concatenate without that level.	None
transform	Callable[[Any], Any] | None	Optional function applied to each selected leaf value before concatenation. The result may be a pd.DataFrame, pd.Series, or a scalar (which is wrapped as a one-row "value" column).	None
select	Callable[[tuple[Key, ...]], bool] | None	Optional predicate called as select(path) where path is a tuple of keys leading to a node (e.g. ("run_001", "metrics")). Evaluated before loading child values, so filtered paths are never deserialized.	None
max_workers	int | None	When set, leaf values are loaded in parallel using a :class:~concurrent.futures.ThreadPoolExecutor with this many worker threads. Useful when leaves are large files (e.g. Parquet) and I/O dominates. When None (default), loading is sequential.	None

Returns:

Type	Description
DataFrame	Result of pd.concat with a MultiIndex built from the selected leaf paths.

Raises:

Type	Description
ValueError	When a LazyDict node is encountered but names does not have enough levels to reach leaf values, or when no values are selected.

Source code in src/tidyrun/serialization/lazy_dict.py

def concat(
    self,
    names: list[str | None] | None = None,
    transform: Callable[[Any], Any] | None = None,
    select: Callable[[tuple[Key, ...]], bool] | None = None,
    max_workers: int | None = None,
) -> Any:
    """Concatenate leaf values from a nested LazyDict.

    Parameters
    ----------
    names :
        Names for the MultiIndex levels built from nested keys. Pass
        ``None`` for a level to drop it from the index and concatenate
        without that level.
    transform :
        Optional function applied to each selected leaf value before
        concatenation. The result may be a ``pd.DataFrame``,
        ``pd.Series``, or a scalar (which is wrapped as a one-row
        ``"value"`` column).
    select :
        Optional predicate called as ``select(path)`` where ``path`` is a
        tuple of keys leading to a node (e.g. ``("run_001", "metrics")``).
        Evaluated before loading child values, so filtered paths are never
        deserialized.
    max_workers :
        When set, leaf values are loaded in parallel using a
        :class:`~concurrent.futures.ThreadPoolExecutor` with this many
        worker threads. Useful when leaves are large files (e.g. Parquet)
        and I/O dominates. When ``None`` (default), loading is sequential.

    Returns
    -------
    pd.DataFrame
        Result of ``pd.concat`` with a MultiIndex built from the selected
        leaf paths.

    Raises
    ------
    ValueError
        When a ``LazyDict`` node is encountered but ``names`` does not have
        enough levels to reach leaf values, or when no values are selected.
    """
    import pandas as pd  # pyright: ignore[reportMissingTypeStubs]

    pd = cast(Any, pd)

    def _identity(value: Any) -> Any:
        return value

    def _select_all(_path: tuple[Key, ...]) -> bool:
        return True

    selected_transform: Callable[[Any], Any] = (
        transform if transform is not None else _identity
    )
    selected_filter: Callable[[tuple[Key, ...]], bool] = (
        select if select is not None else _select_all
    )

    # Phase 1 (serial): walk the tree to discover leaf (path, node, key)
    # without loading the leaf values.  Intermediate LazyDict nodes are
    # detected cheaply via metadata peek; only directories that resolve to
    # another LazyDict are recursed into.
    leaf_refs: list[tuple[tuple[Key, ...], LazyDict, Key]] = []

    def _collect_refs(node: LazyDict, prefix: tuple[Key, ...]) -> None:
        for key in node:
            current_path = prefix + (key,)
            if not selected_filter(current_path):
                continue

            encoded_name = encode_key(key)
            if _entry_is_lazy_dict(node.__serialized_path__, encoded_name):
                if names is not None and len(current_path) >= len(names):
                    if transform is not None:
                        leaf_refs.append((current_path, node, key))
                        continue
                    raise ValueError(
                        f"Encountered LazyDict at depth {len(current_path)}, "
                        f"but names only has {len(names)} levels. "
                        f"Provide more levels in names to reach leaf values."
                    )
                child = node[key]
                _collect_refs(child, current_path)
            else:
                leaf_refs.append((current_path, node, key))

    _collect_refs(self, ())

    if not leaf_refs:
        raise ValueError("No values selected for concatenation")

    # Phase 2: load leaf values — parallel when max_workers is set.
    def _load(ref: tuple[tuple[Key, ...], LazyDict, Key]) -> Any:
        _, node, key = ref
        value = node[key]
        transformed = selected_transform(value)
        if isinstance(transformed, LazyDict):
            raise ValueError(
                f"Transform returned LazyDict for path {ref[0]}, but "
                f"concat expects leaf values. Adjust transform or provide "
                f"more levels in names to reach leaf values."
            )
        if isinstance(transformed, (pd.Series, pd.DataFrame)):
            return transformed
        return pd.Series([transformed], name="value").to_frame()

    if max_workers is not None and len(leaf_refs) > 1:
        with ThreadPoolExecutor(max_workers=max_workers) as executor:
            frames = list(executor.map(_load, leaf_refs))
    else:
        frames = [_load(ref) for ref in leaf_refs]

    # Phase 3: apply transform and build frames.
    keys: list[tuple[Key, ...]] = []
    values: list[Any] = []
    for (current_path, _, _key), frame in zip(leaf_refs, frames):
        keys.append(current_path)
        values.append(frame)

    # Handle None values in names by dropping those levels
    if names is not None and any(n is None for n in names):
        # Create filtered keys and names, keeping only non-None levels
        keep_indices = [i for i, n in enumerate(names) if n is not None]
        filtered_names = [names[i] for i in keep_indices]

        if not filtered_names:
            # All levels are dropped, just concatenate without keys
            return pd.concat(values)
        else:
            # Some levels remain
            filtered_keys = [tuple(k[i] for i in keep_indices) for k in keys]
            return pd.concat(values, keys=filtered_keys, names=filtered_names)
    else:
        return pd.concat(values, keys=keys, names=names)

Exceptions

TidyRunSerializationError

Raised when serialization fails (e.g., no encoder matches the value type).

from tidyrun.serialization import TidyRunSerializationError

try:
    serialize(some_unsupported_type(), "./output")
except TidyRunSerializationError as e:
    print(f"Cannot serialize: {e}")

TidyRunDeserializationError

Raised when deserialization fails (e.g., missing metadata, invalid format).

from tidyrun.serialization import TidyRunDeserializationError

try:
    deserialize("./invalid_path")
except TidyRunDeserializationError as e:
    print(f"Cannot deserialize: {e}")

Key Encoding

TidyRun keys (used as folder/file names in the hierarchy) are encoded using TOML for type safety.

tidyrun.keys.encode_key(key)

Encode a Python key value to a path-safe string using TOML.

Supported types are str, int, float, bool, date, datetime, and time. Plain strings that round-trip unambiguously through TOML are left unquoted; strings that would otherwise be interpreted as another type (e.g. "true", "42") are TOML-quoted.

Parameters:

Name	Type	Description	Default
key	Key	The key value to encode.	required

Returns:

Type	Description
str	A non-empty string suitable for use as a filesystem path component.

Raises:

Type	Description
TidyRunKeyEncodingError	When the key type is not supported or the resulting name violates path constraints (empty, contains / or \, starts with ., or ends with .tidyrun).

Examples:

>>> encode_key(42)
'42'
>>> encode_key("hello")
'hello'
>>> encode_key("true")
'"true"'
>>> encode_key(True)
'true'

Source code in src/tidyrun/keys.py

def encode_key(key: Key) -> str:
    """Encode a Python key value to a path-safe string using TOML.

    Supported types are ``str``, ``int``, ``float``, ``bool``, ``date``,
    ``datetime``, and ``time``. Plain strings that round-trip unambiguously
    through TOML are left unquoted; strings that would otherwise be
    interpreted as another type (e.g. ``"true"``, ``"42"``) are
    TOML-quoted.

    Parameters
    ----------
    key :
        The key value to encode.

    Returns
    -------
    str
        A non-empty string suitable for use as a filesystem path component.

    Raises
    ------
    TidyRunKeyEncodingError
        When the key type is not supported or the resulting name violates
        path constraints (empty, contains ``/`` or ``\\``, starts with
        ``.``, or ends with ``.tidyrun``).

    Examples
    --------
    >>> encode_key(42)
    '42'
    >>> encode_key("hello")
    'hello'
    >>> encode_key("true")
    '"true"'
    >>> encode_key(True)
    'true'
    """
    if not _is_supported_key(key):
        raise TidyRunKeyEncodingError(f"Unsupported key type: {type(key).__name__}")

    if isinstance(key, str):
        _validate_name(key, error_type=TidyRunKeyEncodingError)

        # Keep plain strings unquoted unless parsing them as TOML would
        # coerce them to a different type (e.g. int/bool/date).
        try:
            toml_module = cast(Any, toml)
            parsed = cast(dict[str, Any], toml_module.loads(f"{_KEY_NAME} = {key}\n"))[
                _KEY_NAME
            ]
        except toml.TomlDecodeError:
            return key

        if parsed == key and isinstance(parsed, str):
            return key

    try:
        toml_module = cast(Any, toml)
        toml_doc = cast(str, toml_module.dumps({_KEY_NAME: key}))
    except (TypeError, ValueError) as exc:
        raise TidyRunKeyEncodingError(
            f"Unsupported key type: {type(key).__name__}"
        ) from exc

    prefix = f"{_KEY_NAME} = "
    assert toml_doc.startswith(prefix)
    assert toml_doc.endswith("\n")
    name = toml_doc[len(prefix) : -1]

    _validate_name(name, error_type=TidyRunKeyEncodingError)
    return name

tidyrun.keys.decode_key(name)

Decode a stored key name back to its original Python type.

Reverses the encoding produced by :func:encode_key.

Parameters:

Name	Type	Description	Default
name	str	The encoded key name (a filesystem path component).	required

Returns:

Type	Description
Key	The original Python value (str, int, float, bool, date, datetime, or time).

Raises:

Type	Description
TidyRunKeyDecodingError	When the name is empty, contains path separators, or cannot be decoded to a supported key type.

Examples:

>>> decode_key("hello")
'hello'
>>> decode_key('"true"')
'true'
>>> decode_key("42")
42

Source code in src/tidyrun/keys.py

def decode_key(name: str) -> Key:
    """Decode a stored key name back to its original Python type.

    Reverses the encoding produced by :func:`encode_key`.

    Parameters
    ----------
    name :
        The encoded key name (a filesystem path component).

    Returns
    -------
    Key
        The original Python value (``str``, ``int``, ``float``, ``bool``,
        ``date``, ``datetime``, or ``time``).

    Raises
    ------
    TidyRunKeyDecodingError
        When the name is empty, contains path separators, or cannot be
        decoded to a supported key type.

    Examples
    --------
    >>> decode_key("hello")
    'hello'
    >>> decode_key('"true"')
    'true'
    >>> decode_key("42")
    42
    """
    _validate_name(name, error_type=TidyRunKeyDecodingError)

    try:
        toml_module = cast(Any, toml)
        value = cast(dict[str, Any], toml_module.loads(f"{_KEY_NAME} = {name}\n"))[
            _KEY_NAME
        ]
    except toml.TomlDecodeError as exc:
        if name.startswith('"') or name.startswith("'"):
            raise TidyRunKeyDecodingError(f"Invalid encoded key: {name!r}") from exc
        return name

    if not _is_supported_key(value):
        raise TidyRunKeyDecodingError(
            f"Decoded value has unsupported type: {type(value).__name__}"
        )

    return value

Customization

Custom Encoder

To add support for a custom type:

from tidyrun.serialization import EncoderSpec, serialize

def is_my_type(value):
    return isinstance(value, MyType)

def encode_my_type(value, target):
    # Write value to target location
    ...

def decode_my_type(source):
    # Read and return value from source location
    ...

my_encoder = EncoderSpec(
    name="my-custom-type",
    predicate=is_my_type,
    serializer=encode_my_type,
    deserializer=decode_my_type,
)

# Use custom encoder
from tidyrun.serialization import default_encoders

custom_pipeline = (my_encoder,) + default_encoders()
serialize(my_value, "./output", encoders=custom_pipeline)

Custom Encoder Pipeline

To override the default pipeline order:

from tidyrun.serialization import default_encoders, EncoderSpec

# Reorder: put HDF5 before Parquet
encoders = default_encoders()
reordered = (
    encoders[0],  # dict-folder
    encoders[3],  # pandas-hdf5
    encoders[1],  # dataframe-parquet
    *encoders[2:],  # rest
)

serialize(df, "./output", encoders=reordered)

Performance Considerations

Lazy Loading

LazyDict does not load values until accessed:

loaded = deserialize("./large_structure")  # Fast: only reads metadata

result = loaded["expensive_dataframe"]  # Slow: loads large file here
result = loaded["expensive_dataframe"]  # Loaded again on access

Concatenation Memory

concat() materializes all selected leaves into memory before concatenating. For very large structures, consider filtering with the select parameter:

# Load all 1000 experiments at once (high memory)
combined = loaded.concat()

# Load only 10 specific experiments (low memory)
combined = loaded.concat(
    select=lambda path: path[0] in experiments[:10]
)

Parquet Engine Selection

Parquet serialization uses pyarrow by default; fastparquet is tried if pyarrow is unavailable. HDF5 is tried if parquet encoding fails.

Limitations and Future Work

Current Limitations

1. Remote Storage: S3 is supported as an optional backend via tidyrun[s3]. Other cloud storage providers such as GCS and Azure Blob Storage are not yet supported.
2. Schema Evolution: No automatic schema migration for DataFrames. Users must handle schema changes manually (e.g., use transform in concat).
3. Parquet Multi-Index: Multi-index DataFrames cannot be serialized to parquet and fall back to HDF5.

Planned Features

• Custom Metadata: Allow users to store arbitrary metadata alongside outputs

Testing

Run the full test suite:

pixi run pytest tests/serialization tests/test_keys.py

Key test modules:

• tests/serialization/test_api.py: End-to-end serialize/deserialize, metadata, fallback sequencing, S3 round-trip
• tests/serialization/test_encoders.py: Encoder predicates and detection logic
• tests/serialization/test_lazy_dict.py: LazyDict access patterns and concatenation
• tests/test_keys.py: Key encoding and decoding

Examples

Example 1: Saving Experiment Results

import pandas as pd
from tidyrun import serialize, deserialize

# After running an experiment
results = {
    "config": {"lr": 0.001, "epochs": 100},
    "metrics": pd.DataFrame({
        "epoch": [1, 2, 3],
        "loss": [0.5, 0.3, 0.2]
    }),
    "model_weights": some_large_array,  # Will pickle
}

serialize(results, "./experiments/exp_001")

# Later, load with lazy access
loaded = deserialize("./experiments/exp_001")
print(loaded["config"])  # Loaded on access
model = loaded["model_weights"]  # Pickled data

Example 2: Comparing Multiple Runs

runs = {
    "run_a": {
        "metrics": pd.DataFrame({"accuracy": [0.8, 0.85, 0.9]}),
    },
    "run_b": {
        "metrics": pd.DataFrame({"accuracy": [0.75, 0.82, 0.88]}),
    },
}

serialize(runs, "./comparison")

# Load and aggregate
loaded = deserialize("./comparison")
combined = loaded.concat(names=["run_id"])
print(combined)
# Output:
#             accuracy
# run_id
# run_a   0        0.80
#         1        0.85
#         2        0.90
# run_b   0        0.75
#         1        0.82
#         2        0.88

Example 3: Filtered Concatenation

# Load results from multiple time periods
results = deserialize("./results_by_month")

# Concatenate only 2026 results, adding month info
combined = results.concat(
    names=["month"],
    select=lambda path: path[0].startswith("2026"),
    transform=lambda df: df.assign(period="2026")
)