Skip to content
This site documents the development version of Orcapod (the main branch), which may differ from the latest release. For the releases, see PyPI and the changelog.

Databases

Database backends for persistent storage of pipeline computation results.

class · orcapod.databases.delta_lake_databases.DeltaTableDatabase

class DeltaTableDatabase(
base_path: str | Path | UPath,
storage_options: dict[str, str] | None = None,
create_base_path: bool = True,
batch_size: int = 1000,
max_hierarchy_depth: int = 10,
allow_schema_evolution: bool = True,
_path_prefix: tuple[str, ...] = (),
_root: 'DeltaTableDatabase | None' = None,
_scoped_path: tuple[str, ...] = (),
_shared_pending_batches: 'dict[str, pa.Table] | None' = None,
_shared_pending_record_ids: 'defaultdict[str, set[bytes]] | None' = None,
_shared_delta_table_cache: 'dict[str, deltalake.DeltaTable] | None' = None,
_shared_snapshots: 'dict[str, _CommittedSnapshot] | None' = None,
_shared_pending_indexes: 'dict[str, KeyIndex] | None' = None,
_shared_absent_tables: 'set[str] | None' = None,
)

A Delta table database with clear insert vs update semantics.

  • insert(): Never overwrites existing records by default. Can skip duplicates if requested. Can be batched for performance. Supports composite keys.

  • update(): Always overwrites existing records. Executes immediately. Requires pending batches to be flushed first (or use force_flush=True).

Supports both single column and composite (multi-column) record IDs.

method · orcapod.databases.delta_lake_databases.DeltaTableDatabase.list_table_paths

def list_table_paths(prefix: tuple[str, ...] = ()) -> list[tuple[str, ...]]

List the paths of the Delta tables stored under prefix.

Walks the directories below prefix: a directory that holds a _delta_log directory is a table. The walk goes on below a table, because tables can be nested (Orcapod v0.3 stores a table one level below where v0.2 stored it), and never enters _delta_log. No table is opened. Only flushed tables are listed, as with table_exists.

A local database is listed exactly. A cloud database is listed through fsspec with one list request per directory: s3:// and s3a:// through s3fs with the credentials, endpoint and region of this database's storage options; gs://, az:// and abfs:// only when the matching fsspec package is installed and reaches the database with its default credentials.

Parameters:

  • prefix (tuple[str, ...]) default (): Leading path components, relative to self.base_path.

Returns:

  • list[tuple[str, ...]]: The table paths, relative to self.base_path, sorted.

Raises:

  • TableListingNotSupportedError: If the database is on cloud storage and could not be listed; the reason is chained.
  • OrcapodDatabaseReadError: If a local directory could not be listed.

method · orcapod.databases.delta_lake_databases.DeltaTableDatabase.table_exists

def table_exists(record_path: tuple[str, ...]) -> bool

Return True if a Delta table exists at record_path.

This is a cheap existence check that does NOT load any records.

Parameters:

  • record_path (tuple[str, ...]): Path components identifying the table, relative to self.base_path.

Returns:

  • bool: True if a table exists at the given path, False otherwise.

method · orcapod.databases.delta_lake_databases.DeltaTableDatabase.add_records

def add_records(
record_path: tuple[str, ...],
records: pa.Table,
record_id_column: str | None = None,
skip_duplicates: bool = False,
flush: bool = False,
schema_handling: Literal['merge', 'error', 'coerce'] = 'error',
) -> None

Insert new records. By default, never overwrites existing records.

Parameters:

  • schema_handling (Literal['merge', 'error', 'coerce']) default 'error': How to handle schema differences
  • skip_duplicates (bool) default False: If True, skip records with IDs that already exist. If False, raise error on duplicates.
  • flush (bool) default False: Whether to flush immediately after the insert

Raises:

  • ValueError: If any record IDs already exist and skip_duplicates=False

method · orcapod.databases.delta_lake_databases.DeltaTableDatabase.get_all_records

def get_all_records(
record_path: tuple[str, ...],
record_id_column: str | None = None,
retrieve_pending: bool = True,
) -> pa.Table | None

Get all records from both pending batch and Delta table.

Every read method returns the rows a flush would leave: the stored rows plus the pending rows whose record id is not stored yet, as seen through any handle of this at() family.

Returns:

  • pa.Table | None: Combined Arrow table with all records, or None if no records exist

Raises:

  • OrcapodDatabaseReadError: If the Delta table exists but could not be read.

method · orcapod.databases.delta_lake_databases.DeltaTableDatabase.get_record_by_id

def get_record_by_id(
record_path: tuple[str, ...],
record_id: str | bytes,
record_id_column: str | None = None,
flush: bool = False,
) -> 'pa.Table | None'

Get a specific record by record_id with schema preservation.

Parameters:

  • record_path (tuple[str, ...]): Tuple of path components
  • record_id (str | bytes): Unique identifier for the record

Returns:

  • 'pa.Table | None': Arrow table for the record or None if not found

method · orcapod.databases.delta_lake_databases.DeltaTableDatabase.get_records_by_ids

def get_records_by_ids(
record_path: tuple[str, ...],
record_ids: Collection[str | bytes] | pl.Series | pa.Array,
record_id_column: str | None = None,
flush: bool = False,
) -> 'pa.Table | None'

Retrieve records by entry IDs as a single table with schema preservation.

Parameters:

  • record_path (tuple[str, ...]): Tuple of path components
  • record_ids (Collection[str | bytes] | pl.Series | pa.Array): Entry IDs to retrieve

Returns:

  • 'pa.Table | None': Arrow table containing all found records with original schema, or None if no records found

method · orcapod.databases.delta_lake_databases.DeltaTableDatabase.to_config

def to_config(db_registry: Any = None) -> dict[str, Any]

Serialize database configuration to a JSON-compatible dict.

classmethod · orcapod.databases.delta_lake_databases.DeltaTableDatabase.from_config

def from_config(config: dict[str, Any]) -> 'DeltaTableDatabase'

Reconstruct a DeltaTableDatabase from a config dict.

Supports both the current format ("root_uri" for the storage root, "base_path" as a list for the scoping prefix) and the legacy format produced before ENG-341 ("base_path" as a URI string, no prefix).

property · orcapod.databases.delta_lake_databases.DeltaTableDatabase.keeps_field_metadata

Type: bool

Whether this database keeps Arrow field metadata: it does, on every write path including schema evolution (ITL-1019).

method · orcapod.databases.delta_lake_databases.DeltaTableDatabase.flush

def flush() -> None

Flush all pending batches.

Every pending batch is attempted, even after one fails. A batch that fails stays queued, so a later flush tries it again, and each failure is logged.

Raises:

  • OrcapodDatabaseReadError: If a batch's existing table could not be loaded. It is raised as itself; every other batch that failed in this flush is added to it as a note.
  • InputValidationError: If no batch failed with OrcapodDatabaseReadError but a batch's column types differ from its stored table's. Other failed batches are added to it as notes.
  • RuntimeError: If batches failed with other errors only, naming the failed batches, with the first failure as its __cause__.

method · orcapod.databases.delta_lake_databases.DeltaTableDatabase.flush_batch

def flush_batch(record_path: tuple[str, ...]) -> None

Flush pending batch for a specific source path.

Parameters:

  • record_path (tuple[str, ...]): Tuple of path components (relative to this instance's prefix)

property · orcapod.databases.delta_lake_databases.DeltaTableDatabase.base_path

Type: tuple[str, ...]

The current relative root of this database view (always () for root instances).

method · orcapod.databases.delta_lake_databases.DeltaTableDatabase.at

def at(*path_components: str) -> 'DeltaTableDatabase'

Return a new DeltaTableDatabase scoped to the given sub-path.

The returned instance uses the same underlying filesystem root but all reads and writes are relative to the extended prefix. The scoped instance shares the root's _pending_batches and _pending_record_ids dicts so that a flush() on the root is visible to — and can flush — all scoped instances.

Raises:

  • TypeError: If any component is not a str.
  • ValueError: If any component is empty, is '.' or '..', or contains filesystem-unsafe characters (/, \, *, ?, ", <, >, |, \0).

method · orcapod.databases.delta_lake_databases.DeltaTableDatabase.list_sources

def list_sources() -> list[tuple[str, ...]]

List all record paths that contain a valid Delta table under base_path.

Returns:

  • list[tuple[str, ...]]: List of record_path tuples (e.g. [("alpha",), ("beta", "sub")]).

Raises:

  • NotImplementedError: For cloud base paths (filesystem walk not supported).

class · orcapod.databases.in_memory_databases.InMemoryArrowDatabase

class InMemoryArrowDatabase(
max_hierarchy_depth: int = 10,
_path_prefix: tuple[str, ...] = (),
_shared_tables: 'dict[str, pa.Table] | None' = None,
_shared_pending_batches: 'dict[str, pa.Table] | None' = None,
_shared_pending_record_ids: 'dict[str, set[bytes]] | None' = None,
_shared_indexes: 'dict[str, KeyIndex] | None' = None,
_root: InMemoryArrowDatabase | None = None,
_scoped_path: tuple[str, ...] = (),
)

A pure in-memory implementation of the ArrowDatabaseProtocol protocol.

Records are stored in PyArrow tables held in process memory. Data is lost when the process exits — intended for tests and ephemeral use.

Supports the same pending-batch semantics as DeltaTableDatabase: records are buffered in a pending batch and become part of the committed database only after flush() is called (or flush=True is passed to a write method).

Lookups by record ID or by a binary or string column value use hash indexes over the committed rows, and each flush appends without copying the stored rows, so the cost of a per-row write, flush or lookup does not grow with the rows already stored.

property · orcapod.databases.in_memory_databases.InMemoryArrowDatabase.keeps_field_metadata

Type: bool

Whether this database keeps Arrow field metadata: it does (ITL-1019).

property · orcapod.databases.in_memory_databases.InMemoryArrowDatabase.base_path

Type: tuple[str, ...]

The current relative root of this database view (always () for root instances).

method · orcapod.databases.in_memory_databases.InMemoryArrowDatabase.at

def at(*path_components: str) -> 'InMemoryArrowDatabase'

Return a new InMemoryArrowDatabase scoped to the given sub-path.

The returned instance shares the underlying storage dicts (_tables, _pending_batches, _pending_record_ids) by reference, so writes through any view are visible to all views of the same root database.

Raises:

  • ValueError: If any path component is empty, not a str, or contains '/' or '\0' (which would corrupt the '/'-separated record key scheme).

method · orcapod.databases.in_memory_databases.InMemoryArrowDatabase.list_sources

def list_sources() -> list[tuple[str, ...]]

Return all stored record paths under this database's prefix.

Returns a sorted list of record path tuples, each relative to this instance's prefix (consistent with DeltaTableDatabase.list_sources).

method · orcapod.databases.in_memory_databases.InMemoryArrowDatabase.list_table_paths

def list_table_paths(prefix: tuple[str, ...] = ()) -> list[tuple[str, ...]]

List the paths of the tables stored under prefix.

Exact: every table, flushed or still pending, whose path starts with prefix. Nothing is read.

Parameters:

  • prefix (tuple[str, ...]) default (): Leading path components, relative to self.base_path.

Returns:

  • list[tuple[str, ...]]: The table paths, relative to self.base_path, sorted.

method · orcapod.databases.in_memory_databases.InMemoryArrowDatabase.table_exists

def table_exists(record_path: tuple[str, ...]) -> bool

Return True if a table exists at record_path.

This is a cheap existence check that does NOT load any records.

Parameters:

  • record_path (tuple[str, ...]): Path components identifying the table, relative to self.base_path.

Returns:

  • bool: True if a table exists at the given path, False otherwise.

method · orcapod.databases.in_memory_databases.InMemoryArrowDatabase.to_config

def to_config(db_registry: DatabaseRegistry | None = None) -> dict[str, Any]

Serialize database configuration to a JSON-compatible dict.

When called on a scoped instance (self._root is not None) with a db_registry, emits a compact scoped format that references the root database by registry key plus the scoped path. The root config is registered in the registry, not inlined.

When called without a registry, or on a root instance, emits the existing flat format unchanged (backward compatible).

classmethod · orcapod.databases.in_memory_databases.InMemoryArrowDatabase.from_config

def from_config(config: dict[str, Any]) -> 'InMemoryArrowDatabase'

Reconstruct an InMemoryArrowDatabase from a config dict.

class · orcapod.databases.noop_database.NoOpArrowDatabase

class NoOpArrowDatabase(
_path_prefix: tuple[str, ...] = (),
_root: NoOpArrowDatabase | None = None,
_scoped_path: tuple[str, ...] = (),
)

An ArrowDatabaseProtocol implementation that performs no real storage.

All write operations are silently discarded. All read operations return None (empty / not found). Useful as a placeholder where a database dependency is required by an interface but persistence is unwanted — e.g. dry-run pipelines, testing that code paths execute without I/O, or benchmarking pure compute overhead.

It keeps no Arrow field metadata (keeps_field_metadata is False), so a node that would store a data-tag refuses it (InputValidationError when the database is attached): a function node over a stream with data-tags, or an operator node with data-tags in LOG or REPLAY mode (ITL-1019). Keep the tags plain (tag_only_columns) or use the cache mode OFF to dry-run such a node.

property · orcapod.databases.noop_database.NoOpArrowDatabase.keeps_field_metadata

Type: bool

Whether this database keeps Arrow field metadata: it keeps nothing (ITL-1019).

method · orcapod.databases.noop_database.NoOpArrowDatabase.list_table_paths

def list_table_paths(prefix: tuple[str, ...] = ()) -> list[tuple[str, ...]]

Return no path: this no-op implementation never stores a table.

Parameters:

  • prefix (tuple[str, ...]) default (): Leading path components, relative to self.base_path.

Returns:

  • list[tuple[str, ...]]: Always an empty list.

method · orcapod.databases.noop_database.NoOpArrowDatabase.table_exists

def table_exists(record_path: tuple[str, ...]) -> bool

Return False — no tables are ever persisted by this no-op implementation.

Parameters:

  • record_path (tuple[str, ...]): Path components identifying the table, relative to self.base_path.

Returns:

  • bool: Always False.

property · orcapod.databases.noop_database.NoOpArrowDatabase.base_path

Type: tuple[str, ...]

The current relative root of this database view (always () for root instances).

method · orcapod.databases.noop_database.NoOpArrowDatabase.at

def at(*path_components: str) -> 'NoOpArrowDatabase'

Return a new NoOpArrowDatabase scoped to the given sub-path.

All reads and writes are still discarded; the prefix only affects the reported base_path of the returned instance.

method · orcapod.databases.noop_database.NoOpArrowDatabase.to_config

def to_config(db_registry: Any = None) -> dict[str, Any]

Serialize database configuration to a JSON-compatible dict.

classmethod · orcapod.databases.noop_database.NoOpArrowDatabase.from_config

def from_config(config: dict[str, Any]) -> 'NoOpArrowDatabase'

Reconstruct a NoOpArrowDatabase from a config dict.