Databases
Database backends for persistent storage of pipeline computation results.
DeltaTableDatabase
Section titled “DeltaTableDatabase”class · orcapod.databases.delta_lake_databases.DeltaTableDatabase
class DeltaTableDatabase( base_path: str | Path | UPath, storage_options: dict[str, str] | None = None, create_base_path: bool = True, batch_size: int = 1000, max_hierarchy_depth: int = 10, allow_schema_evolution: bool = True, _path_prefix: tuple[str, ...] = (), _root: 'DeltaTableDatabase | None' = None, _scoped_path: tuple[str, ...] = (), _shared_pending_batches: 'dict[str, pa.Table] | None' = None, _shared_pending_record_ids: 'defaultdict[str, set[bytes]] | None' = None, _shared_delta_table_cache: 'dict[str, deltalake.DeltaTable] | None' = None, _shared_snapshots: 'dict[str, _CommittedSnapshot] | None' = None, _shared_pending_indexes: 'dict[str, KeyIndex] | None' = None, _shared_absent_tables: 'set[str] | None' = None,)A Delta table database with clear insert vs update semantics.
-
insert(): Never overwrites existing records by default. Can skip duplicates if requested. Can be batched for performance. Supports composite keys.
-
update(): Always overwrites existing records. Executes immediately. Requires pending batches to be flushed first (or use force_flush=True).
Supports both single column and composite (multi-column) record IDs.
list_table_paths
Section titled “list_table_paths”method · orcapod.databases.delta_lake_databases.DeltaTableDatabase.list_table_paths
def list_table_paths(prefix: tuple[str, ...] = ()) -> list[tuple[str, ...]]List the paths of the Delta tables stored under prefix.
Walks the directories below prefix: a directory that holds a
_delta_log directory is a table. The walk goes on below a table, because
tables can be nested (Orcapod v0.3 stores a table one level below where
v0.2 stored it), and never enters _delta_log. No table is opened. Only
flushed tables are listed, as with table_exists.
A local database is listed exactly. A cloud database is listed through fsspec
with one list request per directory: s3:// and s3a:// through
s3fs with the credentials, endpoint and region of this database's
storage options; gs://, az:// and abfs:// only when the matching
fsspec package is installed and reaches the database with its default
credentials.
Parameters:
- prefix (
tuple[str, ...]) default(): Leading path components, relative toself.base_path.
Returns:
list[tuple[str, ...]]: The table paths, relative toself.base_path, sorted.
Raises:
TableListingNotSupportedError: If the database is on cloud storage and could not be listed; the reason is chained.OrcapodDatabaseReadError: If a local directory could not be listed.
table_exists
Section titled “table_exists”method · orcapod.databases.delta_lake_databases.DeltaTableDatabase.table_exists
def table_exists(record_path: tuple[str, ...]) -> boolReturn True if a Delta table exists at record_path.
This is a cheap existence check that does NOT load any records.
Parameters:
- record_path (
tuple[str, ...]): Path components identifying the table, relative toself.base_path.
Returns:
bool:Trueif a table exists at the given path,Falseotherwise.
add_records
Section titled “add_records”method · orcapod.databases.delta_lake_databases.DeltaTableDatabase.add_records
def add_records( record_path: tuple[str, ...], records: pa.Table, record_id_column: str | None = None, skip_duplicates: bool = False, flush: bool = False, schema_handling: Literal['merge', 'error', 'coerce'] = 'error',) -> NoneInsert new records. By default, never overwrites existing records.
Parameters:
- schema_handling (
Literal['merge', 'error', 'coerce']) default'error': How to handle schema differences - skip_duplicates (
bool) defaultFalse: If True, skip records with IDs that already exist. If False, raise error on duplicates. - flush (
bool) defaultFalse: Whether to flush immediately after the insert
Raises:
ValueError: If any record IDs already exist and skip_duplicates=False
get_all_records
Section titled “get_all_records”method · orcapod.databases.delta_lake_databases.DeltaTableDatabase.get_all_records
def get_all_records( record_path: tuple[str, ...], record_id_column: str | None = None, retrieve_pending: bool = True,) -> pa.Table | NoneGet all records from both pending batch and Delta table.
Every read method returns the rows a flush would leave: the stored rows
plus the pending rows whose record id is not stored yet, as seen through
any handle of this at() family.
Returns:
pa.Table | None: Combined Arrow table with all records, or None if no records exist
Raises:
OrcapodDatabaseReadError: If the Delta table exists but could not be read.
get_record_by_id
Section titled “get_record_by_id”method · orcapod.databases.delta_lake_databases.DeltaTableDatabase.get_record_by_id
def get_record_by_id( record_path: tuple[str, ...], record_id: str | bytes, record_id_column: str | None = None, flush: bool = False,) -> 'pa.Table | None'Get a specific record by record_id with schema preservation.
Parameters:
- record_path (
tuple[str, ...]): Tuple of path components - record_id (
str | bytes): Unique identifier for the record
Returns:
'pa.Table | None': Arrow table for the record or None if not found
get_records_by_ids
Section titled “get_records_by_ids”method · orcapod.databases.delta_lake_databases.DeltaTableDatabase.get_records_by_ids
def get_records_by_ids( record_path: tuple[str, ...], record_ids: Collection[str | bytes] | pl.Series | pa.Array, record_id_column: str | None = None, flush: bool = False,) -> 'pa.Table | None'Retrieve records by entry IDs as a single table with schema preservation.
Parameters:
- record_path (
tuple[str, ...]): Tuple of path components - record_ids (
Collection[str | bytes] | pl.Series | pa.Array): Entry IDs to retrieve
Returns:
'pa.Table | None': Arrow table containing all found records with original schema, or None if no records found
to_config
Section titled “to_config”method · orcapod.databases.delta_lake_databases.DeltaTableDatabase.to_config
def to_config(db_registry: Any = None) -> dict[str, Any]Serialize database configuration to a JSON-compatible dict.
from_config
Section titled “from_config”classmethod · orcapod.databases.delta_lake_databases.DeltaTableDatabase.from_config
def from_config(config: dict[str, Any]) -> 'DeltaTableDatabase'Reconstruct a DeltaTableDatabase from a config dict.
Supports both the current format ("root_uri" for the storage root,
"base_path" as a list for the scoping prefix) and the legacy format
produced before ENG-341 ("base_path" as a URI string, no prefix).
keeps_field_metadata
Section titled “keeps_field_metadata”property · orcapod.databases.delta_lake_databases.DeltaTableDatabase.keeps_field_metadata
Type: bool
Whether this database keeps Arrow field metadata: it does, on every write path including schema evolution (ITL-1019).
method · orcapod.databases.delta_lake_databases.DeltaTableDatabase.flush
def flush() -> NoneFlush all pending batches.
Every pending batch is attempted, even after one fails. A batch that fails stays queued, so a later flush tries it again, and each failure is logged.
Raises:
OrcapodDatabaseReadError: If a batch's existing table could not be loaded. It is raised as itself; every other batch that failed in this flush is added to it as a note.InputValidationError: If no batch failed withOrcapodDatabaseReadErrorbut a batch's column types differ from its stored table's. Other failed batches are added to it as notes.RuntimeError: If batches failed with other errors only, naming the failed batches, with the first failure as its__cause__.
flush_batch
Section titled “flush_batch”method · orcapod.databases.delta_lake_databases.DeltaTableDatabase.flush_batch
def flush_batch(record_path: tuple[str, ...]) -> NoneFlush pending batch for a specific source path.
Parameters:
- record_path (
tuple[str, ...]): Tuple of path components (relative to this instance's prefix)
base_path
Section titled “base_path”property · orcapod.databases.delta_lake_databases.DeltaTableDatabase.base_path
Type: tuple[str, ...]
The current relative root of this database view (always () for root instances).
method · orcapod.databases.delta_lake_databases.DeltaTableDatabase.at
def at(*path_components: str) -> 'DeltaTableDatabase'Return a new DeltaTableDatabase scoped to the given sub-path.
The returned instance uses the same underlying filesystem root but
all reads and writes are relative to the extended prefix. The scoped
instance shares the root's _pending_batches and
_pending_record_ids dicts so that a flush() on the root is
visible to — and can flush — all scoped instances.
Raises:
TypeError: If any component is not a str.ValueError: If any component is empty, is'.'or'..', or contains filesystem-unsafe characters (/,\,*,?,",<,>,|,\0).
list_sources
Section titled “list_sources”method · orcapod.databases.delta_lake_databases.DeltaTableDatabase.list_sources
def list_sources() -> list[tuple[str, ...]]List all record paths that contain a valid Delta table under base_path.
Returns:
list[tuple[str, ...]]: List of record_path tuples (e.g. [("alpha",), ("beta", "sub")]).
Raises:
NotImplementedError: For cloud base paths (filesystem walk not supported).
InMemoryArrowDatabase
Section titled “InMemoryArrowDatabase”class · orcapod.databases.in_memory_databases.InMemoryArrowDatabase
class InMemoryArrowDatabase( max_hierarchy_depth: int = 10, _path_prefix: tuple[str, ...] = (), _shared_tables: 'dict[str, pa.Table] | None' = None, _shared_pending_batches: 'dict[str, pa.Table] | None' = None, _shared_pending_record_ids: 'dict[str, set[bytes]] | None' = None, _shared_indexes: 'dict[str, KeyIndex] | None' = None, _root: InMemoryArrowDatabase | None = None, _scoped_path: tuple[str, ...] = (),)A pure in-memory implementation of the ArrowDatabaseProtocol protocol.
Records are stored in PyArrow tables held in process memory. Data is lost when the process exits — intended for tests and ephemeral use.
Supports the same pending-batch semantics as DeltaTableDatabase: records are buffered in a pending batch and become part of the committed database only after flush() is called (or flush=True is passed to a write method).
Lookups by record ID or by a binary or string column value use hash indexes over the committed rows, and each flush appends without copying the stored rows, so the cost of a per-row write, flush or lookup does not grow with the rows already stored.
keeps_field_metadata
Section titled “keeps_field_metadata”property · orcapod.databases.in_memory_databases.InMemoryArrowDatabase.keeps_field_metadata
Type: bool
Whether this database keeps Arrow field metadata: it does (ITL-1019).
base_path
Section titled “base_path”property · orcapod.databases.in_memory_databases.InMemoryArrowDatabase.base_path
Type: tuple[str, ...]
The current relative root of this database view (always () for root instances).
method · orcapod.databases.in_memory_databases.InMemoryArrowDatabase.at
def at(*path_components: str) -> 'InMemoryArrowDatabase'Return a new InMemoryArrowDatabase scoped to the given sub-path.
The returned instance shares the underlying storage dicts (_tables, _pending_batches, _pending_record_ids) by reference, so writes through any view are visible to all views of the same root database.
Raises:
ValueError: If any path component is empty, not a str, or contains'/'or'\0'(which would corrupt the'/'-separated record key scheme).
list_sources
Section titled “list_sources”method · orcapod.databases.in_memory_databases.InMemoryArrowDatabase.list_sources
def list_sources() -> list[tuple[str, ...]]Return all stored record paths under this database's prefix.
Returns a sorted list of record path tuples, each relative to
this instance's prefix (consistent with DeltaTableDatabase.list_sources).
list_table_paths
Section titled “list_table_paths”method · orcapod.databases.in_memory_databases.InMemoryArrowDatabase.list_table_paths
def list_table_paths(prefix: tuple[str, ...] = ()) -> list[tuple[str, ...]]List the paths of the tables stored under prefix.
Exact: every table, flushed or still pending, whose path starts with
prefix. Nothing is read.
Parameters:
- prefix (
tuple[str, ...]) default(): Leading path components, relative toself.base_path.
Returns:
list[tuple[str, ...]]: The table paths, relative toself.base_path, sorted.
table_exists
Section titled “table_exists”method · orcapod.databases.in_memory_databases.InMemoryArrowDatabase.table_exists
def table_exists(record_path: tuple[str, ...]) -> boolReturn True if a table exists at record_path.
This is a cheap existence check that does NOT load any records.
Parameters:
- record_path (
tuple[str, ...]): Path components identifying the table, relative toself.base_path.
Returns:
bool:Trueif a table exists at the given path,Falseotherwise.
to_config
Section titled “to_config”method · orcapod.databases.in_memory_databases.InMemoryArrowDatabase.to_config
def to_config(db_registry: DatabaseRegistry | None = None) -> dict[str, Any]Serialize database configuration to a JSON-compatible dict.
When called on a scoped instance (self._root is not None) with a
db_registry, emits a compact scoped format that references the root
database by registry key plus the scoped path. The root config is
registered in the registry, not inlined.
When called without a registry, or on a root instance, emits the existing flat format unchanged (backward compatible).
from_config
Section titled “from_config”classmethod · orcapod.databases.in_memory_databases.InMemoryArrowDatabase.from_config
def from_config(config: dict[str, Any]) -> 'InMemoryArrowDatabase'Reconstruct an InMemoryArrowDatabase from a config dict.
NoOpArrowDatabase
Section titled “NoOpArrowDatabase”class · orcapod.databases.noop_database.NoOpArrowDatabase
class NoOpArrowDatabase( _path_prefix: tuple[str, ...] = (), _root: NoOpArrowDatabase | None = None, _scoped_path: tuple[str, ...] = (),)An ArrowDatabaseProtocol implementation that performs no real storage.
All write operations are silently discarded. All read operations return None (empty / not found). Useful as a placeholder where a database dependency is required by an interface but persistence is unwanted — e.g. dry-run pipelines, testing that code paths execute without I/O, or benchmarking pure compute overhead.
It keeps no Arrow field metadata (keeps_field_metadata is False), so a
node that would store a data-tag refuses it (InputValidationError when the
database is attached): a function node over a stream with data-tags, or an
operator node with data-tags in LOG or REPLAY mode (ITL-1019). Keep the
tags plain (tag_only_columns) or use the cache mode OFF to dry-run such a
node.
keeps_field_metadata
Section titled “keeps_field_metadata”property · orcapod.databases.noop_database.NoOpArrowDatabase.keeps_field_metadata
Type: bool
Whether this database keeps Arrow field metadata: it keeps nothing (ITL-1019).
list_table_paths
Section titled “list_table_paths”method · orcapod.databases.noop_database.NoOpArrowDatabase.list_table_paths
def list_table_paths(prefix: tuple[str, ...] = ()) -> list[tuple[str, ...]]Return no path: this no-op implementation never stores a table.
Parameters:
- prefix (
tuple[str, ...]) default(): Leading path components, relative toself.base_path.
Returns:
list[tuple[str, ...]]: Always an empty list.
table_exists
Section titled “table_exists”method · orcapod.databases.noop_database.NoOpArrowDatabase.table_exists
def table_exists(record_path: tuple[str, ...]) -> boolReturn False — no tables are ever persisted by this no-op implementation.
Parameters:
- record_path (
tuple[str, ...]): Path components identifying the table, relative toself.base_path.
Returns:
bool: AlwaysFalse.
base_path
Section titled “base_path”property · orcapod.databases.noop_database.NoOpArrowDatabase.base_path
Type: tuple[str, ...]
The current relative root of this database view (always () for root instances).
method · orcapod.databases.noop_database.NoOpArrowDatabase.at
def at(*path_components: str) -> 'NoOpArrowDatabase'Return a new NoOpArrowDatabase scoped to the given sub-path.
All reads and writes are still discarded; the prefix only affects the reported base_path of the returned instance.
to_config
Section titled “to_config”method · orcapod.databases.noop_database.NoOpArrowDatabase.to_config
def to_config(db_registry: Any = None) -> dict[str, Any]Serialize database configuration to a JSON-compatible dict.
from_config
Section titled “from_config”classmethod · orcapod.databases.noop_database.NoOpArrowDatabase.from_config
def from_config(config: dict[str, Any]) -> 'NoOpArrowDatabase'Reconstruct a NoOpArrowDatabase from a config dict.