Skip to content
This site documents the development version of Orcapod (the main branch), which may differ from the latest release. For the releases, see PyPI and the changelog.

Sources

Source classes provide the entry point for external data into Orcapod pipelines. All sources convert their input to a PyArrow Table and use SourceStreamBuilder for enrichment (provenance columns, system tags, hashing).

class · orcapod.core.sources.arrow_table_source.ArrowTableSource

class ArrowTableSource(
table: 'pa.Table',
tag_columns: str | Collection[str] = (),
system_tag_columns: Collection[str] = (),
record_id_columns: str | Collection[str] | None = None,
infer_nullable: bool = False,
*,
tag_only_columns: TagOnlyColumns = (),
**kwargs: Any,
)

Bases: RootSource

A source backed by an in-memory PyArrow Table.

Uses SourceStreamBuilder to strip system columns, add per-row source-info provenance columns and a system tag column encoding the schema hash, then wraps the result in an ArrowTableStream.

Arrow represents "this field has no nulls" and "this field may have nulls" via a per-field nullable flag. By default all Arrow fields are nullable=True regardless of actual content, so a plain pa.table({...}) or a Polars .to_arrow() result will always produce T | None schema types unless the caller has explicitly constructed the schema.

ArrowTableSource therefore defaults to infer_nullable=False: the incoming table's schema is trusted as-is. This is safe when the caller has already set nullable flags deliberately (e.g. loaded from a typed Parquet file, or produced by an upstream pipeline stage).

If you are passing a raw Arrow table whose schema carries only Arrow's all-nullable default (nullable=True everywhere), set infer_nullable=True or call table.cast(arrow_utils.infer_schema_nullable(table)) yourself before constructing the source. infer_schema_nullable derives nullable = (null_count > 0) per column — i.e. only columns that actually contain nulls in the supplied data are marked nullable.

Note: other source classes (DictSource, ListSource, DataFrameSource, CsvSource, etc.) construct their Arrow tables internally from non-Arrow inputs and always apply infer_schema_nullable automatically, because the all-nullable Arrow default is never intentional in those paths. ArrowTableSource is unique in accepting a pre-existing pa.Table whose schema may already be deliberate.

Every row gets a record ID system tag derived from source_id and a per-row token. With record_id_columns the token is that column's value, so the ID follows the row. Without it the token is the row's position (row_0, row_1, ...), so a row supplied at a different position under the same source_id gets a different record ID. Downstream pipeline entries and SideEffectPod completion records are keyed on the system tags plus the hash of the row's data, so the moved row is treated as a new entry: it is recorded again (two entries with identical tag values but distinct record IDs) and delivered again by a SideEffectPod. No result is paired with the wrong row, because every key includes the row's own data hash. source_id must be stable too: when omitted it defaults to arrow_table:<algorithm label>:<hex> of the table hash, which changes with row order. Whenever row order can vary between runs, pass an explicit source_id and set record_id_columns to a column that is unique per row.

Data-tags (ITL-1023): every declared tag column is a data-tag by default, also a data column of the same name. tag_only_columns (keyword-only) names the tags kept plain (tag only, not data, as in v0.3); True keeps all of them plain.

method · orcapod.core.sources.arrow_table_source.ArrowTableSource.to_config

def to_config(db_registry=None) -> dict[str, Any]

Serialize metadata-only config (in-memory table is not serializable).

classmethod · orcapod.core.sources.arrow_table_source.ArrowTableSource.from_config

def from_config(config: dict[str, Any], db_registry=None) -> ArrowTableSource

Not supported — ArrowTableSource cannot be reconstructed from config.

Raises:

  • NotImplementedError: Always.

property · orcapod.core.sources.arrow_table_source.ArrowTableSource.table

Type: 'pa.Table'

Return the enriched table (with source-info and system tags).

class · orcapod.core.sources.dict_source.DictSource

class DictSource(
data: Collection[Mapping[str, DataValue]],
tag_columns: str | Collection[str] = (),
system_tag_columns: Collection[str] = (),
data_schema: SchemaLike | pa.Schema | None = None,
source_id: str | None = None,
*,
tag_only_columns: TagOnlyColumns = (),
record_id_columns: str | Collection[str] | None = None,
**kwargs: Any,
)

Bases: RootSource

A source backed by a collection of Python dictionaries.

Each dict becomes one (tag, data) pair in the stream. The dicts are converted to an Arrow table via the data-context type converter, then enriched by SourceStreamBuilder (source-info, schema-hash, system tags).

Data-tags (ITL-1023): every declared tag column is a data-tag by default, also a data column of the same name. tag_only_columns (keyword-only) names the tags kept plain (tag only, not data, as in v0.3); True keeps all of them plain.

method · orcapod.core.sources.dict_source.DictSource.to_config

def to_config(db_registry=None) -> dict[str, Any]

Serialize metadata-only config (data is not serializable).

classmethod · orcapod.core.sources.dict_source.DictSource.from_config

def from_config(config: dict[str, Any], db_registry=None) -> DictSource

Not supported — DictSource data cannot be reconstructed from config.

Raises:

  • NotImplementedError: Always.

class · orcapod.core.sources.list_source.ListSource

class ListSource(
name: str,
data: list[Any],
tag_function: Callable[[Any, int], dict[str, Any] | TagProtocol] | None = None,
expected_tag_keys: Collection[str] | None = None,
tag_function_hash_mode: Literal['content', 'signature', 'name'] = 'name',
data_schema: SchemaLike | None = None,
source_id: str | None = None,
record_id_columns: str | Collection[str] | None = None,
**kwargs: Any,
)

Bases: RootSource

A source backed by a Python list.

Each element in the list becomes one (tag, data) pair. The element is stored as the data under name; the tag is either the element's index (default) or the dict returned by tag_function(element, index).

method · orcapod.core.sources.list_source.ListSource.identity_structure

def identity_structure() -> Any

Return identity including class name, field name, elements, tag function hash and key columns (() for the position; ITL-1154).

method · orcapod.core.sources.list_source.ListSource.to_config

def to_config(db_registry=None) -> dict[str, Any]

Serialize metadata-only config (data is not serializable).

classmethod · orcapod.core.sources.list_source.ListSource.from_config

def from_config(config: dict[str, Any], db_registry=None) -> 'ListSource'

Not supported — ListSource data cannot be reconstructed from config.

Raises:

  • NotImplementedError: Always.

class · orcapod.core.sources.data_frame_source.DataFrameSource

class DataFrameSource(
data: FrameInitTypes,
tag_columns: str | Collection[str] = (),
system_tag_columns: Collection[str] = (),
source_id: str | None = None,
schema: pa.Schema | None = None,
*,
tag_only_columns: TagOnlyColumns = (),
record_id_columns: str | Collection[str] | None = None,
**kwargs: Any,
)

Bases: RootSource

A source backed by a Polars DataFrame (or any Polars-compatible data).

The DataFrame is converted to an Arrow table and enriched by SourceStreamBuilder (source-info, schema-hash, system tags).

Data-tags (ITL-1023): every declared tag column is a data-tag by default, also a data column of the same name. tag_only_columns (keyword-only) names the tags kept plain (tag only, not data, as in v0.3); True keeps all of them plain.

method · orcapod.core.sources.data_frame_source.DataFrameSource.to_config

def to_config(db_registry=None) -> dict[str, Any]

Serialize metadata-only config (DataFrame is not serializable).

classmethod · orcapod.core.sources.data_frame_source.DataFrameSource.from_config

def from_config(config: dict[str, Any], db_registry=None) -> DataFrameSource

Not supported — DataFrameSource cannot be reconstructed from config.

Raises:

  • NotImplementedError: Always.

class · orcapod.core.sources.csv_source.CSVSource

class CSVSource(
file_path: str,
tag_columns: str | Collection[str] = (),
system_tag_columns: Collection[str] = (),
record_id_columns: str | Collection[str] | None = None,
source_id: str | None = None,
schema: pa.Schema | None = None,
*,
tag_only_columns: TagOnlyColumns = (),
**kwargs: Any,
)

Bases: RootSource

A source backed by a CSV file.

The file is read once at construction time using PyArrow's CSV reader, converted to an Arrow table, and enriched by SourceStreamBuilder (source-info, schema-hash, system tags).

At the start of every run (check_for_new_data, ITL-1102) the source looks at the file's modification time and size. When either changed, it reads the file again and, if the rows changed (in order without a record_id_columns: row positions are record ids; in any order with one), holds the new rows and moves its modified time. Record ids keep their rule: the record_id_columns value, or the row's position.

Data-tags (ITL-1023): every declared tag column is a data-tag by default, also a data column of the same name. tag_only_columns (keyword-only) names the tags kept plain (tag only, not data, as in v0.3); True keeps all of them plain.

attribute · orcapod.core.sources.csv_source.CSVSource.SOURCE_KIND

Type: str

Config type key and default source-id kind of this source (one vocabulary).

method · orcapod.core.sources.csv_source.CSVSource.check_for_new_data

def check_for_new_data() -> bool

Take in the file's rows if the file changed and its rows changed.

Only the file's modification time and size are read when neither changed. A file written again with the same rows (in another order too, given a record_id_columns) leaves the held rows and the modified time as they are. The rows are compared as read from the file, every column included (a system_tag_columns column too), in order unless a record_id_columns gives the record ids (same_rows).

Returns:

  • bool: True if the source now holds new rows.

Raises:

  • ValueError: If the file's columns or their types changed: the source's schema is part of its identity; build a new source to read it.

method · orcapod.core.sources.csv_source.CSVSource.to_config

def to_config(db_registry=None) -> dict[str, Any]

Serialize this source's configuration to a JSON-compatible dict.

classmethod · orcapod.core.sources.csv_source.CSVSource.from_config

def from_config(config: dict[str, Any], db_registry=None) -> CSVSource

Reconstruct a CSVSource from a config dict.

class · orcapod.core.sources.delta_table_source.DeltaTableSource

class DeltaTableSource(
delta_table_path: PathLike,
tag_columns: str | Collection[str] = (),
system_tag_columns: Collection[str] = (),
record_id_columns: str | Collection[str] | None = None,
source_id: str | None = None,
*,
tag_only_columns: TagOnlyColumns = (),
**kwargs: Any,
)

Bases: RootSource

A source backed by a Delta Lake table.

The table is read once at construction time using deltalake's PyArrow integration. The resulting Arrow table is enriched by SourceStreamBuilder (source-info, schema-hash, system tags).

At the start of every run (check_for_new_data, ITL-1102) the source checks the table's Delta version. When the version moved, it reads the table again and, if the rows changed (in order without a record_id_columns, in any order with one: a version that only compacted files or changed metadata changes nothing), holds the new rows and moves its modified time. Record ids keep their rule: the record_id_columns value, or the row's position.

Data-tags (ITL-1023): every declared tag column is a data-tag by default, also a data column of the same name. tag_only_columns (keyword-only) names the tags kept plain (tag only, not data, as in v0.3); True keeps all of them plain.

attribute · orcapod.core.sources.delta_table_source.DeltaTableSource.SOURCE_KIND

Type: str

Config type key and default source-id kind of this source (one vocabulary).

method · orcapod.core.sources.delta_table_source.DeltaTableSource.check_for_new_data

def check_for_new_data() -> bool

Take in the table's rows if its Delta version moved and its rows changed.

Only the Delta log is read when the version did not move. A version that moved without changing the rows (OPTIMIZE, a metadata change) leaves the held rows and the modified time as they are. The rows are compared as read from the table, every column included (a system_tag_columns column too), in any order (same_rows).

Returns:

  • bool: True if the source now holds new rows.

Raises:

  • ValueError: If the table's columns or their types changed: the source's schema is part of its identity; build a new source to read it.

method · orcapod.core.sources.delta_table_source.DeltaTableSource.to_config

def to_config(db_registry=None) -> dict[str, Any]

Serialize this source's configuration to a JSON-compatible dict.

classmethod · orcapod.core.sources.delta_table_source.DeltaTableSource.from_config

def from_config(config: dict[str, Any], db_registry=None) -> 'DeltaTableSource'

Reconstruct a DeltaTableSource from a config dict.

class · orcapod.core.sources.sqlite_table_source.SQLiteTableSource

class SQLiteTableSource(
db_path: str | os.PathLike,
table_name: str,
tag_columns: str | Collection[str] | None = None,
system_tag_columns: Collection[str] = (),
record_id_columns: str | Collection[str] | None = None,
source_id: str | None = None,
label: str | None = None,
data_context: str | contexts.DataContext | None = None,
config: 'OrcapodConfig | None' = None,
*,
tag_only_columns: TagOnlyColumns = (),
)

Bases: DBTableSource

A read-only Source backed by a table in a SQLite database.

At construction time the source:

  1. Opens a SQLiteConnector for db_path.
  2. Validates the table exists.
  3. Resolves tag columns:
    • If tag_columns is provided, uses them as-is.
    • Otherwise uses the table's primary-key columns.
    • If the table has no explicit PK (ROWID-only), falls back to the implicit rowid integer column; such a table also needs record_id_columns ("rowid" or ()), or it is refused.
  4. Determines the fetch query: injects SELECT rowid, * when "rowid" is a resolved tag column and not a normal table column (handles both auto-detection and from_config reconstruction).
  5. Delegates to DBTableSource.__init__ for fetching and stream building.
  6. Closes the connector — all data is eagerly loaded into memory, so the connection is released immediately to avoid holding file locks.

Parameters:

  • db_path (str | os.PathLike): Path to the SQLite database file, or ":memory:" for an in-process in-memory database.
  • table_name (str): Name of the table to expose as a source.
  • tag_columns (str | Collection[str] | None) default None: Columns to use as tag columns. A bare string is accepted as a single column name. If None (default), the table's primary-key columns are used; ROWID-only tables fall back to ["rowid"].
  • system_tag_columns (Collection[str]) default (): Additional system-level tag columns.
  • record_id_columns (str | Collection[str] | None) default None: The key columns that identify a row; None: the primary key (a table without one is refused); (): the position.
  • source_id (str | None) default None: Canonical source name. Defaults to sqlite_table:<db path as passed>:main.<table>.
  • label (str | None) default None: Human-readable label for this source node.
  • data_context (str | contexts.DataContext | None) default None: Data context governing type conversion and hashing.
  • config ('OrcapodConfig | None') default None: OrcaPod configuration.
  • tag_only_columns (TagOnlyColumns) default (): Tag columns kept as tags only, not data, exactly as in v0.3; True for all of them. Keyword-only. Every other tag column is a data-tag: also a data column of the same name (ITL-1023).

Raises:

  • ValueError: If the table is not found or is empty.
  • sqlite3.OperationalError: If db_path cannot be opened.

attribute · orcapod.core.sources.sqlite_table_source.SQLiteTableSource.SOURCE_KIND

Type: str

Config type key and default source-id kind of this source (one vocabulary).

method · orcapod.core.sources.sqlite_table_source.SQLiteTableSource.to_config

def to_config(db_registry=None) -> dict[str, Any]

Serialize source configuration to a JSON-compatible dict.

classmethod · orcapod.core.sources.sqlite_table_source.SQLiteTableSource.from_config

def from_config(config: dict[str, Any], db_registry=None) -> 'SQLiteTableSource'

Reconstruct a SQLiteTableSource from a config dict.

Parameters:

  • config (dict[str, Any]): Dict as produced by to_config().

Returns:

  • 'SQLiteTableSource': A new SQLiteTableSource instance.

class · orcapod.core.sources.postgresql_table_source.PostgreSQLTableSource

class PostgreSQLTableSource(
dsn: str,
table_name: str,
tag_columns: str | Collection[str] | None = None,
system_tag_columns: Collection[str] = (),
record_id_columns: str | Collection[str] | None = None,
source_id: str | None = None,
label: str | None = None,
data_context: str | contexts.DataContext | None = None,
config: OrcapodConfig | None = None,
*,
tag_only_columns: TagOnlyColumns = (),
)

Bases: DBTableSource

A read-only Source backed by a table in a PostgreSQL database.

At construction time the source:

  1. Stores the DSN for serialisation.
  2. Opens a PostgreSQLConnector for dsn.
  3. Delegates to DBTableSource.__init__, which validates the table, resolves tag columns (defaults to PK columns), fetches all rows as Arrow batches, and builds the stream.
  4. Closes the connector — all data is eagerly loaded into memory, so the connection is released immediately.

PostgreSQL PK columns are always NOT NULL, so NULL tag values can only arise when tag_columns is overridden to point at a nullable column. Such NULLs are passed through as-is (Arrow supports nulls).

Parameters:

  • dsn (str): libpq connection string. URI form: "postgresql://user:pass@host:5432/dbname" Keyword form: "host=localhost dbname=mydb user=alice"
  • table_name (str): Name of the table to expose as a source.
  • tag_columns (str | Collection[str] | None) default None: Columns to use as tag columns. A bare string is accepted as a single column name. If None (default), the table's primary-key columns are used. Raises ValueError if the table has no primary key and no explicit columns are given.
  • system_tag_columns (Collection[str]) default (): Additional system-level tag columns.
  • record_id_columns (str | Collection[str] | None) default None: The key columns that identify a row; None: the primary key (a table without one is refused); (): the position.
  • source_id (str | None) default None: Canonical source name. Defaults to postgresql_table:<database>.<schema>.<table>.
  • label (str | None) default None: Human-readable label for this source node.
  • data_context (str | contexts.DataContext | None) default None: Data context governing type conversion and hashing.
  • config (OrcapodConfig | None) default None: OrcaPod configuration.
  • tag_only_columns (TagOnlyColumns) default (): Tag columns kept as tags only, not data, exactly as in v0.3; True for all of them. Keyword-only. Every other tag column is a data-tag: also a data column of the same name (ITL-1023).

Raises:

  • ValueError: If the table is not found, is empty, or has no PK and no tag_columns are given.
  • psycopg.OperationalError: If the DSN is invalid or connection fails.

attribute · orcapod.core.sources.postgresql_table_source.PostgreSQLTableSource.SOURCE_KIND

Type: str

Config type key and default source-id kind of this source (one vocabulary).

method · orcapod.core.sources.postgresql_table_source.PostgreSQLTableSource.to_config

def to_config(db_registry=None) -> dict[str, Any]

Serialize source configuration to a JSON-compatible dict.

classmethod · orcapod.core.sources.postgresql_table_source.PostgreSQLTableSource.from_config

def from_config(config: dict[str, Any], db_registry=None) -> PostgreSQLTableSource

Reconstruct a PostgreSQLTableSource from a config dict.

Parameters:

  • config (dict[str, Any]): Dict as produced by to_config().

Returns:

class · orcapod.core.sources.polling_source.PollingSource

class PollingSource(
impl: DynamicSourceProtocol[T],
tag_columns: str | Collection[str],
polling_config: PollingConfig = PollingConfig(),
source_id: str | None = None,
label: str | None = None,
data_context: str | Any | None = None,
config: Any | None = None,
record_id_columns: str | Collection[str] | None = None,
)

Bases: RootSource, Generic[T]

A root source that continuously emits data via a polling loop.

Wraps a DynamicSourceProtocol implementation. Under async execution (async_iter_data), the framework polls the impl on a fixed interval and yields new rows as they arrive. Under sync execution (iter_data), a single poll+fetch cycle is performed on each access and results are served from an accumulated in-memory cache.

The default PollingConfig() polls every 1 second, runs indefinitely (duration=0 means run until cancelled), allows up to 5 consecutive overrun intervals before terminating (an overrun interval is one that was consumed while the previous poll iteration was still executing — i.e. the loop fell behind its scheduled tick), retries up to 3 consecutive poll()/fetch() errors with 1-second exponential backoff, and resets error and overrun counters on every clean tick.

Parameters:

  • impl (DynamicSourceProtocol[T]): User-supplied DynamicSourceProtocol implementation that provides identity, to_config, from_config, poll, fetch, close, and schema methods.
  • tag_columns (str | Collection[str]): Column name(s) that form the tag (join key) for each row. All other columns become data columns.
  • polling_config (PollingConfig) default PollingConfig(): Scheduling and error-handling configuration. See PollingConfig for field semantics and defaults. All fields are validated at construction — ValueError is raised for out-of-range values.
  • source_id (str | None) default None: Optional stable string identifier for provenance tracking. Defaults to polling_source:<label>:<hex> of the semantic hash of impl.identity().
  • label (str | None) default None: Optional human-readable label shown in pipeline diagrams.
  • data_context (str | Any | None) default None: Optional data context key or instance for type conversion and hashing.
  • config (Any | None) default None: Optional Orcapod framework config.
  • record_id_columns (str | Collection[str] | None) default None: Required. The key columns whose values identify a row (the record id), in key order. A key is polled once: a batch that repeats a key of its own or of an earlier batch is refused with RowIdentifierError and not committed.

Raises:

  • ValueError: If record_id_columns is not given, or impl.schema() returns a non-None schema that does not include all columns named in tag_columns.

Example::

class MyDBSource:
def __init__(self, db_url):
self._db_url = db_url
def identity(self):
return ("MyDBSource", self._db_url)
def to_config(self):
return {"url": self._db_url}
@classmethod
def from_config(cls, config):
return cls(config["url"])
def schema(self):
return Schema({"row_id": int, "value": float})
async def poll(self, cursor=None):
return await self._check_has_new_rows(since=cursor)
async def fetch(self, cursor=None):
rows, new_cursor = await self._fetch_rows(since=cursor)
return Cursor.now(new_cursor), rows
async def close(self):
await self._conn.close()
# Poll every 5 s, run for 60 s, raise after 3 consecutive errors
src = PollingSource(
MyDBSource("postgresql://..."),
tag_columns="row_id",
polling_config=PollingConfig(interval=5.0, duration=60.0),
record_id_columns="row_id",
)

attribute · orcapod.core.sources.polling_source.PollingSource.SOURCE_KIND

Type: str

Config type key and default source-id kind of this source (one vocabulary).

method · orcapod.core.sources.polling_source.PollingSource.identity_structure

def identity_structure() -> Any

Identity derived from the impl's own identity, tag columns and key columns.

Delegates to impl.identity() so that the implementer fully controls what makes two PollingSource instances distinct; the key columns (record_id_columns) follow, as for every root source (ITL-1154).

method · orcapod.core.sources.polling_source.PollingSource.output_schema

def output_schema(
*,
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> tuple[Schema, Schema]

Return the output schema.

Uses three levels of resolution to avoid triggering a poll+fetch cycle:

  1. Declared-schema fast path — when impl.schema() returned a non-None schema at construction (populating _tag_schema and _data_schema), answers directly from those schemas. System-tag column names and types (str for source_id, bytes for record_id) are derived from _declared_schema_hash without any fetch. Applies to all ColumnConfig flags computable from declared schemas (system_tags); flags requiring actual persisted data (meta, source, context, content_hash) fall through to the next level.
  2. Cached-stream path — if _batches is already non-empty (at least one batch fetched via sync or async), delegates to the first batch without triggering a new poll+fetch cycle.
  3. Fallback — calls _sync_poll_and_commit() then _get_combined_stream(), running a synchronous poll+fetch via _run_sync. Only reached when the impl declared no schema and no batch has been fetched yet.

method · orcapod.core.sources.polling_source.PollingSource.keys

def keys(
*,
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> tuple[tuple[str, ...], tuple[str, ...]]

Return tag and data column keys.

Uses the same three-level resolution as output_schema: impl-declared schemas → cached _batches[0] → _sync_poll_and_commit() fallback. When the impl declared a schema, system-tag column names are derived from _declared_schema_hash without triggering a fetch.

method · orcapod.core.sources.polling_source.PollingSource.head

def head(
n: int = 10,
*,
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> StreamView

Return the first rows already polled; never polls.

Before the first poll there are no rows. The columns are the declared schema's, or none when the source declared no schema (finding it would mean polling).

method · orcapod.core.sources.polling_source.PollingSource.iter_data

def iter_data()

Iterate over (tag, data) pairs from the current snapshot.

method · orcapod.core.sources.polling_source.PollingSource.as_table

def as_table(
*,
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> pa.Table

Return the accumulated rows as a PyArrow table.

method · orcapod.core.sources.polling_source.PollingSource.to_config

def to_config(db_registry: Any = None) -> dict[str, Any]

Serialize this source to a JSON-compatible config dict.

Always stores impl_module and impl_class so that from_config can import and reconstruct the impl class — analogous to how DataFunction serializes its callable. The impl_config key carries the result of impl.to_config():

  • Non-None: from_config calls impl_class.from_config(impl_config).
  • None: from_config falls back to impl_class() (no-arg constructor).

Parameters:

  • db_registry (Any) default None: Unused; present for protocol compatibility.

Returns:

  • dict[str, Any]: A dict suitable for passing to from_config.

classmethod · orcapod.core.sources.polling_source.PollingSource.from_config

def from_config(config: dict[str, Any], db_registry: Any = None) -> PollingSource

Reconstruct a PollingSource by importing the impl class.

When config["impl_config"] is non-None, the impl is reconstructed via impl_class.from_config(impl_config). Otherwise the impl class is instantiated with no arguments.

Parameters:

  • config (dict[str, Any]): A dict as produced by to_config.
  • db_registry (Any) default None: Unused; present for protocol compatibility.

Returns:

Raises:

  • KeyError: If config is missing a required key.
  • ImportError: If the impl module cannot be imported.
  • AttributeError: If the impl class does not exist in the module.
  • TypeError: If the impl class cannot be instantiated.

method · orcapod.core.sources.polling_source.PollingSource.check_for_new_data

def check_for_new_data() -> bool

Poll once; take in what the poll finds and move the modified time if it has rows.

The start-of-run check (ITL-1102): one poll, and a fetch when the poll reports new data, as one synchronous read does (the first check fetches unconditionally). A poll or fetch that raises propagates: the run warns and goes on with the rows already polled, and the cursor, rows and modified time stay as they were.

Returns:

  • bool: True if a batch with at least one row was committed.

method · orcapod.core.sources.polling_source.PollingSource.async_iter_data

async def async_iter_data()

Async generator that continuously emits (tag, data) pairs.

Uses a per-iterator local_batch_idx to track the next position in the append-only _batches list. A drain step at the top of each loop iteration yields any newly committed batches (including those committed by concurrent sync callers) before sleeping.

The optimistic lock protocol — snapshot cursor (brief lock) → poll and fetch with no lock held → commit only if cursor unchanged (brief lock) — prevents TOCTOU races without ever holding _state_lock across an await.

impl.close() is always awaited before returning or raising.

class · orcapod.core.sources.derived_source.DerivedSource

class DerivedSource(
origin: 'FunctionJobNode | OperatorJobNode | RootSource',
source_id: str | None = None,
tags_as_data: Mapping[str, str] | None = None,
record_id_columns: str | Collection[str] | None = None,
**kwargs: Any,
)

Bases: RootSource

A source whose rows are derived from another source or from a job node.

There are two forms.

Plain (tags_as_data not given), created by FunctionNode.as_source() or OperatorNode.as_source(): a static stream backed by the computed records of a DB-backed stream node. It presents the node's stored results with the node's own tag and data columns. The origin must implement get_all_records(), output_schema(), keys() and content_hash().

With tags_as_data: a new root source over any source or job node that exposes chosen tag columns of the origin also as data columns, under the given names. A function pod never sees tags, and an operator never puts new content into the data group; a source defines its own tag and data columns, so this is where a key (for example a session start time) can also become an input to computation. The tags are the origin's tags. The data columns are the origin's data columns followed by the copies, in mapping order. Each copy has the tag column's exact Arrow type, nullability and field metadata. The copy equals the tag when the source is read; downstream it is plain data.

Being a source, this form starts provenance afresh by the rule every source follows: its own system-tag columns, record IDs derived from its source_id and each row's position (or record_id_columns), and a source-info token for every data column, the copies included. It stores nothing itself; like any source it can be wrapped with cached(database).

Identity (tags_as_data form): - source_id defaults to tags_as_data:<algorithm label>:<hex digest>, where label and digest are the derivation hash: the content hash an operator node doing this derivation over the origin would have, built from the origin's content hash, the sorted tags_as_data items and the record_id_columns when one is set. It changes when the origin's identity or one of these parameters changes. An explicit source_id replaces it. - content_hash follows the rule of every root source (class name, output schema and source_id) plus the sorted tags_as_data items, since two mappings can give one schema and one explicit source_id. - pipeline_hash is schema-only, as for every root source.

Identity (plain form): - content_hash is tied to the origin node's content hash. - pipeline_hash is schema-only, so plain derived sources with identical schemas share the same pipeline DB table.

If the origin is a job node that has not run yet, the source presents zero rows with the correct columns. In the tags_as_data form the rows appear once the node has run.

Data-tags (ITL-1019): the source carries the bindings of what it reads, in both forms. Over a job node they are the ones the node's stored table states (the marker of data_tags), which must be the node's own (a name in both halves of its keys); a table that disagrees is refused. Over a source they are the source's own. A tags_as_data copy of a data-tag is a new, plain data column. Nothing is added to the identity for the binding, because it is already there: the plain form's content_hash is the origin's, which carries it, and every pipeline_hash and the tags_as_data form's content_hash hash the output schema, in which a data-tag is in both halves.

Parameters:

  • origin ('FunctionJobNode | OperatorJobNode | RootSource'): The source or DB-backed job node to derive from.
  • source_id (str | None) default None: Canonical source name. Defaults as described above.
  • tags_as_data (Mapping[str, str] | None) default None: Maps a tag column of the origin to the name of the data column that receives a copy of it.
  • record_id_columns (str | Collection[str] | None) default None: Key columns whose values identify a row in record IDs and provenance tokens, instead of the row position. Only with tags_as_data.
  • **kwargs (Any) default {}: label, data_context (defaults to the origin's) and config, passed to RootSource.

Raises:

  • InputValidationError: If tags_as_data names a column that is not a tag of the origin, or a new name that is already a column, is repeated, or is reserved.
  • ValueError: If record_id_columns is given without tags_as_data.
  • TypeError: If the origin is a source and tags_as_data is not given.

Example::

experiment = DerivedSource(
SpiralDBTableSource(project, "experiment"),
tags_as_data={"session_start_datetime": "session_start"},
)

method · orcapod.core.sources.derived_source.DerivedSource.identity_structure

def identity_structure() -> Any

Return what the content hash is built from.

The plain form is tied to the origin node's content hash. The tags_as_data form is the root source rule (class name, output schema, source_id, key columns) followed by the sorted mapping items: two mappings can give the same schema (two tag columns of one type copied to one name) while the copy holds different values, so with an explicit source_id the mapping has to tell the sources apart (ITL-1000).

Returns:

  • Any: The identity structure.

method · orcapod.core.sources.derived_source.DerivedSource.to_config

def to_config(db_registry: Any = None) -> dict[str, Any]

Serialize this source to a JSON-compatible dict.

The origin's own config is embedded when the origin is a source, which makes the derived source reloadable whenever its origin is. A default source_id is not written: it is recomputed from the reloaded origin.

Parameters:

  • db_registry (Any) default None: Optional database registry, forwarded to the origin.

Returns:

  • dict[str, Any]: The config dict, with source_type "derived".

classmethod · orcapod.core.sources.derived_source.DerivedSource.from_config

def from_config(config: dict[str, Any], db_registry: Any = None) -> 'DerivedSource'

Rebuild a derived source by first rebuilding its origin.

Parameters:

  • config (dict[str, Any]): Dict as produced by to_config().
  • db_registry (Any) default None: Optional database registry, forwarded to the origin.

Returns:

  • 'DerivedSource': A new DerivedSource over the rebuilt origin.

Raises:

  • NotImplementedError: If the config holds no origin: a derived source over a job node cannot be rebuilt from a config.

method · orcapod.core.sources.derived_source.DerivedSource.check_for_new_data

def check_for_new_data() -> bool

Take in the origin's rows if they changed since this source read them.

Developer, November 14, 2026 (ITL-1102). Over a source, the origin is checked first (check_for_new_data). Then the origin's rows are read again only when its modified time moved since this source last read them: a source origin that took in new data, by this check or by a check made through another source over it, or a job node that ran again. The rows read are compared with the rows held (in any order only when a record_id_columns gives the record ids); when they differ the source holds the new ones and moves its modified time. A read or comparison that raises leaves everything as it was, so the next check reads again. A source that holds nothing (it has not read its origin yet, or read none of its rows: a job node that had not run) reads the origin at its next read; when the origin's time moved, its own time moves too, so a node that read no rows from it runs again.

Returns:

  • bool: True if the source's modified time moved: it holds new rows, or it holds
  • bool: none and its origin changed.

method · orcapod.core.sources.derived_source.DerivedSource.head

def head(
n: int = 10,
*,
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> StreamView

Return the first rows of the origin's records, without computing.

The records are the ones this source already read. Otherwise, an origin source gives its own head(n) (so a polling or caching origin is never polled or written), and an origin job node gives its stored rows, read now and not kept. An origin job node that has not run gives no rows. With sort_by_tags (or all_info), the rows are sorted by tag as as_table sorts them: all the stored rows of an origin job node, but only the rows an origin source reads for its head (see StreamBase.head).

class · orcapod.core.sources.cached_source.CachedSource

class CachedSource(
source: SourceProtocol,
cache_database: ArrowDatabaseProtocol,
cache_path_prefix: tuple[str, ...] = (),
cache_path: tuple[str, ...] | None = None,
source_id: str | None = None,
label: str | None = None,
data_context: str | contexts.DataContext | None = None,
config: OrcapodConfig | None = None,
)

Bases: RootSource

DB-backed wrapper around a source that caches every data.

Accepts any SourceProtocol implementation as the inner source. Implements StreamProtocol transparently so downstream consumers are unaware of caching. Cache table is scoped to the source's content_hash() — each unique source gets its own table.

Example::

source = ArrowTableSource(table, tag_columns=["id"])
cached = CachedSource(source, cache_database=db)
# or equivalently:
cached = source.cached(cache_database=db)

attribute · orcapod.core.sources.cached_source.CachedSource.SOURCE_KIND

Type: str

Config type key and default source-id kind of this source (one vocabulary).

method · orcapod.core.sources.cached_source.CachedSource.to_config

def to_config(db_registry: DatabaseRegistryProtocol | None = None) -> dict[str, Any]

Serialize this CachedSource configuration to a JSON-compatible dict.

Parameters:

  • db_registry (DatabaseRegistryProtocol | None) default None: Optional registry. When provided, the cache database config is registered (deduplicated) and replaced by its key string. When None, the full database config is embedded inline.

Returns:

  • dict[str, Any]: Dict containing the inner source config, cache database config,
  • dict[str, Any]: cache path prefix, and resolved cache path (for cache-only loading).

classmethod · orcapod.core.sources.cached_source.CachedSource.from_config

def from_config(
config: dict[str, Any],
db_registry: DatabaseRegistryProtocol | None = None,
) -> CachedSource

Reconstruct a CachedSource from a config dict.

If the inner source cannot be resolved (e.g. it requires live data that is unavailable), resolve_source_from_config returns a SourceProxy preserving the original source's identity. The CachedSource can still serve data from its cache database.

Parameters:

  • config (dict[str, Any]): Dict as produced by to_config.
  • db_registry (DatabaseRegistryProtocol | None) default None: Optional registry. When cache_database is a string key, it is resolved via db_registry.resolve(). When None and the cache_database value is a string key, a ValueError is raised.

Returns:

  • CachedSource: A new CachedSource constructed from the config.

property · orcapod.core.sources.cached_source.CachedSource.cache_path

Type: tuple[str, ...]

Cache table path: scoped to the source's content hash, then the layout version.

The layout version (sdb_v1) is always the last component, also when an explicit cache_path was given.

method · orcapod.core.sources.cached_source.CachedSource.check_for_new_data

def check_for_new_data() -> bool

Check the wrapped source; move only if the next read shows new rows (ITL-1102).

The wrapped source is checked first. When its modified time moved since the cache last took in its rows (by this check, or by a check made through another source that wraps it), its rows are hashed and compared with the rows this source presents (recorded when it built its merged rows; before a build, the rows already cached: a read of the cache; nothing is written). The cache keeps every row it ever held, so rows that were removed from the wrapped source change nothing. When this source would present rows it has not (another source sharing the cache may already have written them), the merged rows held in memory are dropped (the next read writes the new rows to the cache and merges them) and the modified time moves. A comparison that raises leaves everything as it was, so the next check compares again.

The invariant this keeps: the modified time moves exactly when the next read would present a different set of rows. That read presents the rows cached (every row any source sharing the cache wrote, including rows since deleted from the wrapped source) and the live rows, so a row another source wrote to a shared cache moves this source too, and a row deleted from the wrapped source moves nothing.

The check guarantees nothing for two cached sources that share one cache (the same database and cache_path, or the same table): give each cached source its own cache. A change missed there means a run does not compute the new rows; nothing wrong is stored (ITL-1143).

Returns:

  • bool: True if this source will present new rows at the next read.

property · orcapod.core.sources.cached_source.CachedSource.is_stale

Type: bool

True if the wrapped source has been modified since the last build.

Overrides StreamBase.is_stale because CachedSource is a RootSource (no upstreams/producer) yet still depends on the wrapped _source.

method · orcapod.core.sources.cached_source.CachedSource.clear_cache

def clear_cache() -> None

Discard in-memory cached stream (forces rebuild on next access).

method · orcapod.core.sources.cached_source.CachedSource.head

def head(
n: int = 10,
*,
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> StreamView

Return the first rows without reading or writing the cache.

Once the source has been used, its merged stream is in memory and this is a slice of it. Before that its rows are not known: they are the cache's earlier rows merged with the wrapped source's, and reading them writes the cache. The head then has the wrapped source's columns, no rows and NOT_COMPUTED (or the wrapped source's own reason when it has none to give).

method · orcapod.core.sources.cached_source.CachedSource.get_all_records

def get_all_records(
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> 'pa.Table | None'

Retrieve the cached records that the current Arrow hasher wrote.

class · orcapod.core.sources.source_proxy.SourceProxy

class SourceProxy(
source_id: str,
content_hash_str: str,
pipeline_hash_str: str,
tag_schema: Schema,
data_schema: Schema,
expected_class_name: str | None = None,
source_config: dict[str, Any] | None = None,
label: str | None = None,
)

Bases: RootSource

A proxy source that preserves identity and optionally delegates.

When created without a bound source, SourceProxy returns stored hashes and schemas but raises on data access. A live source can be substituted in later via bind — if the source's identity matches, all data methods delegate to it.

Parameters:

  • source_id (str): The original source's canonical ID.
  • content_hash_str (str): The original source's content hash string.
  • pipeline_hash_str (str): The original source's pipeline hash string.
  • tag_schema (Schema): The original source's tag schema.
  • data_schema (Schema): The original source's data schema.
  • expected_class_name (str | None) default None: Class name of the original source (e.g. "ArrowTableSource"). Informational and used for validation in bind.
  • source_config (dict[str, Any] | None) default None: The original source's serialized config (for to_config).
  • label (str | None) default None: Optional label.

property · orcapod.core.sources.source_proxy.SourceProxy.expected_class_name

Type: str | None

The class name of the original source this proxy stands in for.

property · orcapod.core.sources.source_proxy.SourceProxy.delegate

Type: SourceProtocol | None

The bound live source, or None if no source has been bound.

property · orcapod.core.sources.source_proxy.SourceProxy.is_bound

Type: bool

True if a live source has been bound via bind.

method · orcapod.core.sources.source_proxy.SourceProxy.bind

def bind(source: SourceProtocol) -> None

Bind a live source to this proxy, enabling data access.

The source must match this proxy's identity — same source_id, content_hash, pipeline_hash, tag schema keys, and data schema keys. If expected_class_name is set, the source's class name must also match.

Parameters:

  • source (SourceProtocol): The live source to bind.

Raises:

  • ValueError: If the source's identity does not match.

method · orcapod.core.sources.source_proxy.SourceProxy.unbind

def unbind() -> SourceProtocol | None

Remove and return the bound source, reverting to proxy behavior.

Returns:

  • SourceProtocol | None: The previously bound source, or None if none was bound.

property · orcapod.core.sources.source_proxy.SourceProxy.last_modified

Type: datetime | None

When the proxy's data last changed: its binding or its bound source.

The later of the time the proxy was last bound or unbound and the bound source's own last_modified; a bound source that does not know its time makes the proxy's unknown too (None). So a bound source that takes in new data at a run's check moves the proxy with it (ITL-1102).

method · orcapod.core.sources.source_proxy.SourceProxy.check_for_new_data

def check_for_new_data() -> bool

Check the bound source for new data; an unbound proxy has none (ITL-1102).

Returns:

  • bool: True if the bound source took in new data.

method · orcapod.core.sources.source_proxy.SourceProxy.head

def head(
n: int = 10,
*,
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> StreamView

Return the bound source's head, or no rows when no live source is bound.

method · orcapod.core.sources.source_proxy.SourceProxy.to_config

def to_config(db_registry=None) -> dict[str, Any]

Return the original source's config (preserves source_type).

classmethod · orcapod.core.sources.source_proxy.SourceProxy.from_config

def from_config(config: dict[str, Any], db_registry=None) -> SourceProxy

Not supported — SourceProxy is created by the deserialization pipeline.

class · orcapod.core.sources.source_registry.SourceRegistry

class SourceRegistry()

Registry mapping canonical source IDs to live RootSource objects.

A source ID is a stable, human-readable name (e.g. "delta_table:sales") that is independent of physical location. The registry lets downstream code resolve a source_id token embedded in a provenance string back to the concrete source object that produced it, without requiring a direct reference to the source object.

  • Registering the same object under the same ID is idempotent.
  • Registering a different object under an already-taken ID logs a warning and skips (rather than raising), so that sources constructed in different contexts don't crash each other via the global singleton.
  • Use replace when you explicitly want to overwrite an entry.

The module-level GLOBAL_SOURCE_REGISTRY is the default registry used when no explicit registry is provided.

method · orcapod.core.sources.source_registry.SourceRegistry.register

def register(source_id: str, source: 'RootSource') -> None

Register source under source_id.

If source_id is already taken by the same object, the call is a no-op. If it is taken by a different object, a warning is emitted and the existing entry is left unchanged.

method · orcapod.core.sources.source_registry.SourceRegistry.replace

def replace(source_id: str, source: 'RootSource') -> 'RootSource | None'

Register source under source_id, unconditionally replacing any existing entry. Returns the previous source if one existed.

method · orcapod.core.sources.source_registry.SourceRegistry.unregister

def unregister(source_id: str) -> 'RootSource'

Remove and return the source registered under source_id.

method · orcapod.core.sources.source_registry.SourceRegistry.get

def get(source_id: str) -> 'RootSource'

Return the source for source_id, raising KeyError if absent.

method · orcapod.core.sources.source_registry.SourceRegistry.get_optional

def get_optional(source_id: str) -> 'RootSource | None'

Return the source for source_id, or None if not registered.

class · orcapod.core.sources.base.RootSource

class RootSource(
source_id: str | None = None,
label: str | None = None,
data_context: str | contexts.DataContext | None = None,
config: OrcapodConfig | None = None,
)

Bases: StreamBase

Abstract base class for all root sources in Orcapod.

A RootSource is a pure stream — the root of a computational graph, producing data from an external source (file, database, in-memory data, etc.) with no upstream dependencies.

As a StreamProtocol:

  • source returns None (no upstream source pod)
  • upstreams is always empty
  • keys, output_schema, iter_data, as_table delegate to self._stream by default; concrete subclasses may override them.

As a PipelineElementProtocol:

  • pipeline_identity_structure() returns (tag_schema, data_schema) — schema-only, no data content — forming the base case of the pipeline identity Merkle chain.

Concrete subclasses must set self._stream in their __init__ to get the default stream delegation behavior. They may also override identity_structure().

property · orcapod.core.sources.base.RootSource.source_id

Type: str

Canonical name for this source used in the registry and provenance strings. If not set, raises ValueError.

method · orcapod.core.sources.base.RootSource.identity_structure

def identity_structure() -> Any

Identity from the class name, output schema, source_id and key columns.

The key columns are the resolved record_id_columns (() for the position): two sources that differ only in how they name their rows are two sources, so their records never share a cache (developer, November 16, 2026, ITL-1154). The pipeline hash stays schema-only.

method · orcapod.core.sources.base.RootSource.pipeline_identity_structure

def pipeline_identity_structure() -> Any

Return (tag_schema, data_schema) as the pipeline identity for this source. Schema-only: no data content is included, so sources with identical schemas share the same pipeline hash and therefore the same pipeline database table.

property · orcapod.core.sources.base.RootSource.producer

Type: None

Root sources have no upstream source pod.

property · orcapod.core.sources.base.RootSource.upstreams

Type: tuple[StreamProtocol, ...]

Sources have no upstream dependencies.

method · orcapod.core.sources.base.RootSource.output_schema

def output_schema(
*,
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> tuple[Schema, Schema]

Delegate to the underlying stream's output_schema.

method · orcapod.core.sources.base.RootSource.keys

def keys(
*,
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> tuple[tuple[str, ...], tuple[str, ...]]

Delegate to the underlying stream's keys.

method · orcapod.core.sources.base.RootSource.iter_data

def iter_data()

Delegate to the underlying stream's iter_data.

method · orcapod.core.sources.base.RootSource.as_table

def as_table(
*,
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> 'pa.Table'

Delegate to the underlying stream's as_table.

method · orcapod.core.sources.base.RootSource.check_for_new_data

def check_for_new_data() -> bool

Check whether the data behind the source changed; if so, take it in.

Every run (job.run(), run_async(), a job node's run(), a load-only run too) calls this once on each source feeding it, before deciding which nodes to run (developer, November 14, 2026, ITL-1102). A source that finds new data takes it in and moves its modified time to that moment, so every job node below it is stale and runs. A source that finds nothing changes nothing, so a current job stays current. head() and the notebook preview never call it.

A source built over data held in memory (ArrowTableSource, DictSource, ListSource, DataFrameSource) cannot change: this default returns False. A source that raises here does not fail the run: the run issues a SourceCheckFailedWarning and proceeds with the data the source holds, so a source must leave its data and modified time as they were when it raises. A source whose reads refresh it (PollingSource: a read polls) is refreshed again by a run that reads it, and a refresh that fails then raises (ITL-1139).

Returns:

  • bool: True if the source found new data and took it in.

method · orcapod.core.sources.base.RootSource.head

def head(
n: int = 10,
*,
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> StreamView

Return the first n rows of the table the source holds, without computing.

A source that reads its table when it is built holds it in memory, so this is a slice of it. A source that reads its table later overrides this with a bounded read. See StreamBase.head.

method · orcapod.core.sources.base.RootSource.cached

def cached(
cache_database: ArrowDatabaseProtocol,
cache_path_prefix: tuple[str, ...] = (),
**kwargs: Any,
) -> CachedSource

Return a CachedSource wrapping this source.

Parameters:

  • cache_database (ArrowDatabaseProtocol): Database to store cached records in.
  • cache_path_prefix (tuple[str, ...]) default (): Path prefix for the cache table.
  • **kwargs (Any) default {}: Additional keyword arguments passed to CachedSource.

Returns:

  • CachedSource: A CachedSource that caches this source's output.