Sources
Source classes provide the entry point for external data into Orcapod pipelines.
All sources convert their input to a PyArrow Table and use SourceStreamBuilder for
enrichment (provenance columns, system tags, hashing).
ArrowTableSource
Section titled “ArrowTableSource”class · orcapod.core.sources.arrow_table_source.ArrowTableSource
class ArrowTableSource( table: 'pa.Table', tag_columns: str | Collection[str] = (), system_tag_columns: Collection[str] = (), record_id_columns: str | Collection[str] | None = None, infer_nullable: bool = False, *, tag_only_columns: TagOnlyColumns = (), **kwargs: Any,)Bases: RootSource
A source backed by an in-memory PyArrow Table.
Uses SourceStreamBuilder to strip system columns, add per-row
source-info provenance columns and a system tag column encoding the
schema hash, then wraps the result in an ArrowTableStream.
Nullable handling
Section titled “Nullable handling”Arrow represents "this field has no nulls" and "this field may have nulls"
via a per-field nullable flag. By default all Arrow fields are
nullable=True regardless of actual content, so a plain pa.table({...})
or a Polars .to_arrow() result will always produce T | None schema
types unless the caller has explicitly constructed the schema.
ArrowTableSource therefore defaults to infer_nullable=False: the
incoming table's schema is trusted as-is. This is safe when the caller
has already set nullable flags deliberately (e.g. loaded from a typed
Parquet file, or produced by an upstream pipeline stage).
If you are passing a raw Arrow table whose schema carries only Arrow's
all-nullable default (nullable=True everywhere), set
infer_nullable=True or call
table.cast(arrow_utils.infer_schema_nullable(table)) yourself before
constructing the source. infer_schema_nullable derives
nullable = (null_count > 0) per column — i.e. only columns that
actually contain nulls in the supplied data are marked nullable.
Note: other source classes (DictSource, ListSource,
DataFrameSource, CsvSource, etc.) construct their Arrow tables
internally from non-Arrow inputs and always apply infer_schema_nullable
automatically, because the all-nullable Arrow default is never intentional
in those paths. ArrowTableSource is unique in accepting a pre-existing
pa.Table whose schema may already be deliberate.
Record identity
Section titled “Record identity”Every row gets a record ID system tag derived from source_id and a
per-row token. With record_id_columns the token is that column's value,
so the ID follows the row. Without it the token is the row's position
(row_0, row_1, ...), so a row supplied at a different position
under the same source_id gets a different record ID. Downstream
pipeline entries and SideEffectPod completion records are keyed on
the system tags plus the hash of the row's data, so the moved row is
treated as a new entry: it is recorded again (two entries with identical
tag values but distinct record IDs) and delivered again by a
SideEffectPod. No result is paired with the wrong row, because every
key includes the row's own data hash. source_id must be stable too:
when omitted it defaults to arrow_table:<algorithm label>:<hex> of the
table hash, which changes with row order.
Whenever row order can vary between runs, pass an explicit source_id
and set record_id_columns to a column that is unique per row.
Data-tags (ITL-1023): every declared tag column is a data-tag by default,
also a data column of the same name. tag_only_columns (keyword-only) names the
tags kept plain (tag only, not data, as in v0.3); True keeps all of them plain.
to_config
Section titled “to_config”method · orcapod.core.sources.arrow_table_source.ArrowTableSource.to_config
def to_config(db_registry=None) -> dict[str, Any]Serialize metadata-only config (in-memory table is not serializable).
from_config
Section titled “from_config”classmethod · orcapod.core.sources.arrow_table_source.ArrowTableSource.from_config
def from_config(config: dict[str, Any], db_registry=None) -> ArrowTableSourceNot supported — ArrowTableSource cannot be reconstructed from config.
Raises:
NotImplementedError: Always.
property · orcapod.core.sources.arrow_table_source.ArrowTableSource.table
Type: 'pa.Table'
Return the enriched table (with source-info and system tags).
DictSource
Section titled “DictSource”class · orcapod.core.sources.dict_source.DictSource
class DictSource( data: Collection[Mapping[str, DataValue]], tag_columns: str | Collection[str] = (), system_tag_columns: Collection[str] = (), data_schema: SchemaLike | pa.Schema | None = None, source_id: str | None = None, *, tag_only_columns: TagOnlyColumns = (), record_id_columns: str | Collection[str] | None = None, **kwargs: Any,)Bases: RootSource
A source backed by a collection of Python dictionaries.
Each dict becomes one (tag, data) pair in the stream. The dicts are
converted to an Arrow table via the data-context type converter, then
enriched by SourceStreamBuilder (source-info, schema-hash, system tags).
Data-tags (ITL-1023): every declared tag column is a data-tag by default,
also a data column of the same name. tag_only_columns (keyword-only) names the
tags kept plain (tag only, not data, as in v0.3); True keeps all of them plain.
to_config
Section titled “to_config”method · orcapod.core.sources.dict_source.DictSource.to_config
def to_config(db_registry=None) -> dict[str, Any]Serialize metadata-only config (data is not serializable).
from_config
Section titled “from_config”classmethod · orcapod.core.sources.dict_source.DictSource.from_config
def from_config(config: dict[str, Any], db_registry=None) -> DictSourceNot supported — DictSource data cannot be reconstructed from config.
Raises:
NotImplementedError: Always.
ListSource
Section titled “ListSource”class · orcapod.core.sources.list_source.ListSource
class ListSource( name: str, data: list[Any], tag_function: Callable[[Any, int], dict[str, Any] | TagProtocol] | None = None, expected_tag_keys: Collection[str] | None = None, tag_function_hash_mode: Literal['content', 'signature', 'name'] = 'name', data_schema: SchemaLike | None = None, source_id: str | None = None, record_id_columns: str | Collection[str] | None = None, **kwargs: Any,)Bases: RootSource
A source backed by a Python list.
Each element in the list becomes one (tag, data) pair. The element is
stored as the data under name; the tag is either the element's index
(default) or the dict returned by tag_function(element, index).
identity_structure
Section titled “identity_structure”method · orcapod.core.sources.list_source.ListSource.identity_structure
def identity_structure() -> AnyReturn identity including class name, field name, elements, tag
function hash and key columns (() for the position; ITL-1154).
to_config
Section titled “to_config”method · orcapod.core.sources.list_source.ListSource.to_config
def to_config(db_registry=None) -> dict[str, Any]Serialize metadata-only config (data is not serializable).
from_config
Section titled “from_config”classmethod · orcapod.core.sources.list_source.ListSource.from_config
def from_config(config: dict[str, Any], db_registry=None) -> 'ListSource'Not supported — ListSource data cannot be reconstructed from config.
Raises:
NotImplementedError: Always.
DataFrameSource
Section titled “DataFrameSource”class · orcapod.core.sources.data_frame_source.DataFrameSource
class DataFrameSource( data: FrameInitTypes, tag_columns: str | Collection[str] = (), system_tag_columns: Collection[str] = (), source_id: str | None = None, schema: pa.Schema | None = None, *, tag_only_columns: TagOnlyColumns = (), record_id_columns: str | Collection[str] | None = None, **kwargs: Any,)Bases: RootSource
A source backed by a Polars DataFrame (or any Polars-compatible data).
The DataFrame is converted to an Arrow table and enriched by
SourceStreamBuilder (source-info, schema-hash, system tags).
Data-tags (ITL-1023): every declared tag column is a data-tag by default,
also a data column of the same name. tag_only_columns (keyword-only) names the
tags kept plain (tag only, not data, as in v0.3); True keeps all of them plain.
to_config
Section titled “to_config”method · orcapod.core.sources.data_frame_source.DataFrameSource.to_config
def to_config(db_registry=None) -> dict[str, Any]Serialize metadata-only config (DataFrame is not serializable).
from_config
Section titled “from_config”classmethod · orcapod.core.sources.data_frame_source.DataFrameSource.from_config
def from_config(config: dict[str, Any], db_registry=None) -> DataFrameSourceNot supported — DataFrameSource cannot be reconstructed from config.
Raises:
NotImplementedError: Always.
CSVSource
Section titled “CSVSource”class · orcapod.core.sources.csv_source.CSVSource
class CSVSource( file_path: str, tag_columns: str | Collection[str] = (), system_tag_columns: Collection[str] = (), record_id_columns: str | Collection[str] | None = None, source_id: str | None = None, schema: pa.Schema | None = None, *, tag_only_columns: TagOnlyColumns = (), **kwargs: Any,)Bases: RootSource
A source backed by a CSV file.
The file is read once at construction time using PyArrow's CSV reader,
converted to an Arrow table, and enriched by SourceStreamBuilder
(source-info, schema-hash, system tags).
At the start of every run (check_for_new_data, ITL-1102) the source looks at
the file's modification time and size. When either changed, it reads the file
again and, if the rows changed (in order without a record_id_columns: row
positions are record ids; in any order with one), holds the new rows and
moves its modified time. Record ids keep their rule: the record_id_columns
value, or the row's position.
Data-tags (ITL-1023): every declared tag column is a data-tag by default,
also a data column of the same name. tag_only_columns (keyword-only) names the
tags kept plain (tag only, not data, as in v0.3); True keeps all of them plain.
SOURCE_KIND
Section titled “SOURCE_KIND”attribute · orcapod.core.sources.csv_source.CSVSource.SOURCE_KIND
Type: str
Config type key and default source-id kind of this source (one vocabulary).
check_for_new_data
Section titled “check_for_new_data”method · orcapod.core.sources.csv_source.CSVSource.check_for_new_data
def check_for_new_data() -> boolTake in the file's rows if the file changed and its rows changed.
Only the file's modification time and size are read when neither changed. A
file written again with the same rows (in another order too, given a
record_id_columns) leaves the held rows and
the modified time as they are. The rows are compared as read from the file,
every column included (a system_tag_columns column too), in order unless a
record_id_columns gives the record ids
(same_rows).
Returns:
bool:Trueif the source now holds new rows.
Raises:
ValueError: If the file's columns or their types changed: the source's schema is part of its identity; build a new source to read it.
to_config
Section titled “to_config”method · orcapod.core.sources.csv_source.CSVSource.to_config
def to_config(db_registry=None) -> dict[str, Any]Serialize this source's configuration to a JSON-compatible dict.
from_config
Section titled “from_config”classmethod · orcapod.core.sources.csv_source.CSVSource.from_config
def from_config(config: dict[str, Any], db_registry=None) -> CSVSourceReconstruct a CSVSource from a config dict.
DeltaTableSource
Section titled “DeltaTableSource”class · orcapod.core.sources.delta_table_source.DeltaTableSource
class DeltaTableSource( delta_table_path: PathLike, tag_columns: str | Collection[str] = (), system_tag_columns: Collection[str] = (), record_id_columns: str | Collection[str] | None = None, source_id: str | None = None, *, tag_only_columns: TagOnlyColumns = (), **kwargs: Any,)Bases: RootSource
A source backed by a Delta Lake table.
The table is read once at construction time using deltalake's
PyArrow integration. The resulting Arrow table is enriched by
SourceStreamBuilder (source-info, schema-hash, system tags).
At the start of every run (check_for_new_data, ITL-1102) the source checks the
table's Delta version. When the version moved, it reads the table again and, if
the rows changed (in order without a record_id_columns, in any order with one: a
version that only compacted files or
changed metadata changes nothing), holds the new rows and moves its modified time.
Record ids keep their rule: the record_id_columns value, or the row's position.
Data-tags (ITL-1023): every declared tag column is a data-tag by default,
also a data column of the same name. tag_only_columns (keyword-only) names the
tags kept plain (tag only, not data, as in v0.3); True keeps all of them plain.
SOURCE_KIND
Section titled “SOURCE_KIND”attribute · orcapod.core.sources.delta_table_source.DeltaTableSource.SOURCE_KIND
Type: str
Config type key and default source-id kind of this source (one vocabulary).
check_for_new_data
Section titled “check_for_new_data”method · orcapod.core.sources.delta_table_source.DeltaTableSource.check_for_new_data
def check_for_new_data() -> boolTake in the table's rows if its Delta version moved and its rows changed.
Only the Delta log is read when the version did not move. A version that moved
without changing the rows (OPTIMIZE, a metadata change) leaves the held rows
and the modified time as they are. The rows are compared as read from the
table, every column included (a system_tag_columns column too), in any
order (same_rows).
Returns:
bool:Trueif the source now holds new rows.
Raises:
ValueError: If the table's columns or their types changed: the source's schema is part of its identity; build a new source to read it.
to_config
Section titled “to_config”method · orcapod.core.sources.delta_table_source.DeltaTableSource.to_config
def to_config(db_registry=None) -> dict[str, Any]Serialize this source's configuration to a JSON-compatible dict.
from_config
Section titled “from_config”classmethod · orcapod.core.sources.delta_table_source.DeltaTableSource.from_config
def from_config(config: dict[str, Any], db_registry=None) -> 'DeltaTableSource'Reconstruct a DeltaTableSource from a config dict.
SQLiteTableSource
Section titled “SQLiteTableSource”class · orcapod.core.sources.sqlite_table_source.SQLiteTableSource
class SQLiteTableSource( db_path: str | os.PathLike, table_name: str, tag_columns: str | Collection[str] | None = None, system_tag_columns: Collection[str] = (), record_id_columns: str | Collection[str] | None = None, source_id: str | None = None, label: str | None = None, data_context: str | contexts.DataContext | None = None, config: 'OrcapodConfig | None' = None, *, tag_only_columns: TagOnlyColumns = (),)Bases: DBTableSource
A read-only Source backed by a table in a SQLite database.
At construction time the source:
- Opens a
SQLiteConnectorfor db_path. - Validates the table exists.
- Resolves tag columns:
- If tag_columns is provided, uses them as-is.
- Otherwise uses the table's primary-key columns.
- If the table has no explicit PK (ROWID-only), falls back to the
implicit
rowidinteger column; such a table also needsrecord_id_columns("rowid"or()), or it is refused.
- Determines the fetch query: injects
SELECT rowid, *when"rowid"is a resolved tag column and not a normal table column (handles both auto-detection andfrom_configreconstruction). - Delegates to
DBTableSource.__init__for fetching and stream building. - Closes the connector — all data is eagerly loaded into memory, so the connection is released immediately to avoid holding file locks.
Parameters:
- db_path (
str | os.PathLike): Path to the SQLite database file, or":memory:"for an in-process in-memory database. - table_name (
str): Name of the table to expose as a source. - tag_columns (
str | Collection[str] | None) defaultNone: Columns to use as tag columns. A bare string is accepted as a single column name. IfNone(default), the table's primary-key columns are used; ROWID-only tables fall back to["rowid"]. - system_tag_columns (
Collection[str]) default(): Additional system-level tag columns. - record_id_columns (
str | Collection[str] | None) defaultNone: The key columns that identify a row;None: the primary key (a table without one is refused);(): the position. - source_id (
str | None) defaultNone: Canonical source name. Defaults tosqlite_table:<db path as passed>:main.<table>. - label (
str | None) defaultNone: Human-readable label for this source node. - data_context (
str | contexts.DataContext | None) defaultNone: Data context governing type conversion and hashing. - config (
'OrcapodConfig | None') defaultNone: OrcaPod configuration. - tag_only_columns (
TagOnlyColumns) default(): Tag columns kept as tags only, not data, exactly as in v0.3;Truefor all of them. Keyword-only. Every other tag column is a data-tag: also a data column of the same name (ITL-1023).
Raises:
ValueError: If the table is not found or is empty.sqlite3.OperationalError: If db_path cannot be opened.
SOURCE_KIND
Section titled “SOURCE_KIND”attribute · orcapod.core.sources.sqlite_table_source.SQLiteTableSource.SOURCE_KIND
Type: str
Config type key and default source-id kind of this source (one vocabulary).
to_config
Section titled “to_config”method · orcapod.core.sources.sqlite_table_source.SQLiteTableSource.to_config
def to_config(db_registry=None) -> dict[str, Any]Serialize source configuration to a JSON-compatible dict.
from_config
Section titled “from_config”classmethod · orcapod.core.sources.sqlite_table_source.SQLiteTableSource.from_config
def from_config(config: dict[str, Any], db_registry=None) -> 'SQLiteTableSource'Reconstruct a SQLiteTableSource from a config dict.
Parameters:
- config (
dict[str, Any]): Dict as produced byto_config().
Returns:
'SQLiteTableSource': A newSQLiteTableSourceinstance.
PostgreSQLTableSource
Section titled “PostgreSQLTableSource”class · orcapod.core.sources.postgresql_table_source.PostgreSQLTableSource
class PostgreSQLTableSource( dsn: str, table_name: str, tag_columns: str | Collection[str] | None = None, system_tag_columns: Collection[str] = (), record_id_columns: str | Collection[str] | None = None, source_id: str | None = None, label: str | None = None, data_context: str | contexts.DataContext | None = None, config: OrcapodConfig | None = None, *, tag_only_columns: TagOnlyColumns = (),)Bases: DBTableSource
A read-only Source backed by a table in a PostgreSQL database.
At construction time the source:
- Stores the DSN for serialisation.
- Opens a
PostgreSQLConnectorfor dsn. - Delegates to
DBTableSource.__init__, which validates the table, resolves tag columns (defaults to PK columns), fetches all rows as Arrow batches, and builds the stream. - Closes the connector — all data is eagerly loaded into memory, so the connection is released immediately.
PostgreSQL PK columns are always NOT NULL, so NULL tag values can
only arise when tag_columns is overridden to point at a nullable
column. Such NULLs are passed through as-is (Arrow supports nulls).
Parameters:
- dsn (
str): libpq connection string. URI form:"postgresql://user:pass@host:5432/dbname"Keyword form:"host=localhost dbname=mydb user=alice" - table_name (
str): Name of the table to expose as a source. - tag_columns (
str | Collection[str] | None) defaultNone: Columns to use as tag columns. A bare string is accepted as a single column name. IfNone(default), the table's primary-key columns are used. RaisesValueErrorif the table has no primary key and no explicit columns are given. - system_tag_columns (
Collection[str]) default(): Additional system-level tag columns. - record_id_columns (
str | Collection[str] | None) defaultNone: The key columns that identify a row;None: the primary key (a table without one is refused);(): the position. - source_id (
str | None) defaultNone: Canonical source name. Defaults topostgresql_table:<database>.<schema>.<table>. - label (
str | None) defaultNone: Human-readable label for this source node. - data_context (
str | contexts.DataContext | None) defaultNone: Data context governing type conversion and hashing. - config (
OrcapodConfig | None) defaultNone: OrcaPod configuration. - tag_only_columns (
TagOnlyColumns) default(): Tag columns kept as tags only, not data, exactly as in v0.3;Truefor all of them. Keyword-only. Every other tag column is a data-tag: also a data column of the same name (ITL-1023).
Raises:
ValueError: If the table is not found, is empty, or has no PK and no tag_columns are given.psycopg.OperationalError: If the DSN is invalid or connection fails.
SOURCE_KIND
Section titled “SOURCE_KIND”attribute · orcapod.core.sources.postgresql_table_source.PostgreSQLTableSource.SOURCE_KIND
Type: str
Config type key and default source-id kind of this source (one vocabulary).
to_config
Section titled “to_config”method · orcapod.core.sources.postgresql_table_source.PostgreSQLTableSource.to_config
def to_config(db_registry=None) -> dict[str, Any]Serialize source configuration to a JSON-compatible dict.
from_config
Section titled “from_config”classmethod · orcapod.core.sources.postgresql_table_source.PostgreSQLTableSource.from_config
def from_config(config: dict[str, Any], db_registry=None) -> PostgreSQLTableSourceReconstruct a PostgreSQLTableSource from a config dict.
Parameters:
- config (
dict[str, Any]): Dict as produced byto_config().
Returns:
PostgreSQLTableSource: A newPostgreSQLTableSourceinstance.
PollingSource
Section titled “PollingSource”class · orcapod.core.sources.polling_source.PollingSource
class PollingSource( impl: DynamicSourceProtocol[T], tag_columns: str | Collection[str], polling_config: PollingConfig = PollingConfig(), source_id: str | None = None, label: str | None = None, data_context: str | Any | None = None, config: Any | None = None, record_id_columns: str | Collection[str] | None = None,)Bases: RootSource, Generic[T]
A root source that continuously emits data via a polling loop.
Wraps a DynamicSourceProtocol implementation. Under async execution
(async_iter_data), the framework polls the impl on a fixed interval
and yields new rows as they arrive. Under sync execution (iter_data),
a single poll+fetch cycle is performed on each access and results are
served from an accumulated in-memory cache.
The default PollingConfig() polls every 1 second, runs indefinitely
(duration=0 means run until cancelled), allows up to 5 consecutive
overrun intervals before terminating (an overrun interval is one that was
consumed while the previous poll iteration was still executing — i.e. the
loop fell behind its scheduled tick), retries up to 3 consecutive
poll()/fetch() errors with 1-second exponential backoff, and
resets error and overrun counters on every clean tick.
Parameters:
- impl (
DynamicSourceProtocol[T]): User-suppliedDynamicSourceProtocolimplementation that providesidentity,to_config,from_config,poll,fetch,close, andschemamethods. - tag_columns (
str | Collection[str]): Column name(s) that form the tag (join key) for each row. All other columns become data columns. - polling_config (
PollingConfig) defaultPollingConfig(): Scheduling and error-handling configuration. SeePollingConfigfor field semantics and defaults. All fields are validated at construction —ValueErroris raised for out-of-range values. - source_id (
str | None) defaultNone: Optional stable string identifier for provenance tracking. Defaults topolling_source:<label>:<hex>of the semantic hash ofimpl.identity(). - label (
str | None) defaultNone: Optional human-readable label shown in pipeline diagrams. - data_context (
str | Any | None) defaultNone: Optional data context key or instance for type conversion and hashing. - config (
Any | None) defaultNone: Optional Orcapod framework config. - record_id_columns (
str | Collection[str] | None) defaultNone: Required. The key columns whose values identify a row (the record id), in key order. A key is polled once: a batch that repeats a key of its own or of an earlier batch is refused withRowIdentifierErrorand not committed.
Raises:
ValueError: Ifrecord_id_columnsis not given, orimpl.schema()returns a non-Noneschema that does not include all columns named intag_columns.
Example::
class MyDBSource: def __init__(self, db_url): self._db_url = db_url
def identity(self): return ("MyDBSource", self._db_url)
def to_config(self): return {"url": self._db_url}
@classmethod def from_config(cls, config): return cls(config["url"])
def schema(self): return Schema({"row_id": int, "value": float})
async def poll(self, cursor=None): return await self._check_has_new_rows(since=cursor)
async def fetch(self, cursor=None): rows, new_cursor = await self._fetch_rows(since=cursor) return Cursor.now(new_cursor), rows
async def close(self): await self._conn.close()
# Poll every 5 s, run for 60 s, raise after 3 consecutive errorssrc = PollingSource( MyDBSource("postgresql://..."), tag_columns="row_id", polling_config=PollingConfig(interval=5.0, duration=60.0), record_id_columns="row_id",)SOURCE_KIND
Section titled “SOURCE_KIND”attribute · orcapod.core.sources.polling_source.PollingSource.SOURCE_KIND
Type: str
Config type key and default source-id kind of this source (one vocabulary).
identity_structure
Section titled “identity_structure”method · orcapod.core.sources.polling_source.PollingSource.identity_structure
def identity_structure() -> AnyIdentity derived from the impl's own identity, tag columns and key columns.
Delegates to impl.identity() so that the implementer fully
controls what makes two PollingSource instances distinct; the key
columns (record_id_columns) follow, as for every root source (ITL-1154).
output_schema
Section titled “output_schema”method · orcapod.core.sources.polling_source.PollingSource.output_schema
def output_schema( *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False,) -> tuple[Schema, Schema]Return the output schema.
Uses three levels of resolution to avoid triggering a poll+fetch cycle:
- Declared-schema fast path — when
impl.schema()returned a non-Noneschema at construction (populating_tag_schemaand_data_schema), answers directly from those schemas. System-tag column names and types (strforsource_id,bytesforrecord_id) are derived from_declared_schema_hashwithout any fetch. Applies to allColumnConfigflags computable from declared schemas (system_tags); flags requiring actual persisted data (meta,source,context,content_hash) fall through to the next level. - Cached-stream path — if
_batchesis already non-empty (at least one batch fetched via sync or async), delegates to the first batch without triggering a new poll+fetch cycle. - Fallback — calls
_sync_poll_and_commit()then_get_combined_stream(), running a synchronous poll+fetch via_run_sync. Only reached when the impl declared no schema and no batch has been fetched yet.
method · orcapod.core.sources.polling_source.PollingSource.keys
def keys( *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False,) -> tuple[tuple[str, ...], tuple[str, ...]]Return tag and data column keys.
Uses the same three-level resolution as output_schema: impl-declared
schemas → cached _batches[0] → _sync_poll_and_commit() fallback.
When the impl declared a schema, system-tag column names are derived
from _declared_schema_hash without triggering a fetch.
method · orcapod.core.sources.polling_source.PollingSource.head
def head( n: int = 10, *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False,) -> StreamViewReturn the first rows already polled; never polls.
Before the first poll there are no rows. The columns are the declared schema's, or none when the source declared no schema (finding it would mean polling).
iter_data
Section titled “iter_data”method · orcapod.core.sources.polling_source.PollingSource.iter_data
def iter_data()Iterate over (tag, data) pairs from the current snapshot.
as_table
Section titled “as_table”method · orcapod.core.sources.polling_source.PollingSource.as_table
def as_table( *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False,) -> pa.TableReturn the accumulated rows as a PyArrow table.
to_config
Section titled “to_config”method · orcapod.core.sources.polling_source.PollingSource.to_config
def to_config(db_registry: Any = None) -> dict[str, Any]Serialize this source to a JSON-compatible config dict.
Always stores impl_module and impl_class so that
from_config can import and reconstruct the impl class —
analogous to how DataFunction serializes its callable. The
impl_config key carries the result of impl.to_config():
- Non-
None:from_configcallsimpl_class.from_config(impl_config). None:from_configfalls back toimpl_class()(no-arg constructor).
Parameters:
- db_registry (
Any) defaultNone: Unused; present for protocol compatibility.
Returns:
dict[str, Any]: A dict suitable for passing tofrom_config.
from_config
Section titled “from_config”classmethod · orcapod.core.sources.polling_source.PollingSource.from_config
def from_config(config: dict[str, Any], db_registry: Any = None) -> PollingSourceReconstruct a PollingSource by importing the impl class.
When config["impl_config"] is non-None, the impl is
reconstructed via impl_class.from_config(impl_config). Otherwise
the impl class is instantiated with no arguments.
Parameters:
- config (
dict[str, Any]): A dict as produced byto_config. - db_registry (
Any) defaultNone: Unused; present for protocol compatibility.
Returns:
PollingSource: A newPollingSourcewrapping the imported impl.
Raises:
KeyError: Ifconfigis missing a required key.ImportError: If the impl module cannot be imported.AttributeError: If the impl class does not exist in the module.TypeError: If the impl class cannot be instantiated.
check_for_new_data
Section titled “check_for_new_data”method · orcapod.core.sources.polling_source.PollingSource.check_for_new_data
def check_for_new_data() -> boolPoll once; take in what the poll finds and move the modified time if it has rows.
The start-of-run check (ITL-1102): one poll, and a fetch when the poll reports new data, as one synchronous read does (the first check fetches unconditionally). A poll or fetch that raises propagates: the run warns and goes on with the rows already polled, and the cursor, rows and modified time stay as they were.
Returns:
bool:Trueif a batch with at least one row was committed.
async_iter_data
Section titled “async_iter_data”method · orcapod.core.sources.polling_source.PollingSource.async_iter_data
async def async_iter_data()Async generator that continuously emits (tag, data) pairs.
Uses a per-iterator local_batch_idx to track the next position in
the append-only _batches list. A drain step at the top of each
loop iteration yields any newly committed batches (including those
committed by concurrent sync callers) before sleeping.
The optimistic lock protocol — snapshot cursor (brief lock) → poll and
fetch with no lock held → commit only if cursor unchanged (brief lock)
— prevents TOCTOU races without ever holding _state_lock across an
await.
impl.close() is always awaited before returning or raising.
DerivedSource
Section titled “DerivedSource”class · orcapod.core.sources.derived_source.DerivedSource
class DerivedSource( origin: 'FunctionJobNode | OperatorJobNode | RootSource', source_id: str | None = None, tags_as_data: Mapping[str, str] | None = None, record_id_columns: str | Collection[str] | None = None, **kwargs: Any,)Bases: RootSource
A source whose rows are derived from another source or from a job node.
There are two forms.
Plain (tags_as_data not given), created by FunctionNode.as_source()
or OperatorNode.as_source(): a static stream backed by the computed records
of a DB-backed stream node. It presents the node's stored results with the
node's own tag and data columns. The origin must implement
get_all_records(), output_schema(), keys() and content_hash().
With tags_as_data: a new root source over any source or job node that
exposes chosen tag columns of the origin also as data columns, under the
given names. A function pod never sees tags, and an operator never puts new
content into the data group; a source defines its own tag and data columns, so
this is where a key (for example a session start time) can also become an
input to computation. The tags are the origin's tags. The data columns are the
origin's data columns followed by the copies, in mapping order. Each copy has
the tag column's exact Arrow type, nullability and field metadata. The copy
equals the tag when the source is read; downstream it is plain data.
Being a source, this form starts provenance afresh by the rule every source
follows: its own system-tag columns, record IDs derived from its source_id
and each row's position (or record_id_columns), and a source-info token
for every data column, the copies included. It stores nothing itself; like
any source it can be wrapped with cached(database).
Identity (tags_as_data form):
- source_id defaults to tags_as_data:<algorithm label>:<hex digest>,
where label and digest are the derivation hash: the content hash an
operator node doing this derivation over the origin would have, built
from the origin's content hash, the sorted tags_as_data items and
the record_id_columns when one is set. It changes when the origin's
identity or one of these parameters changes. An explicit source_id
replaces it.
- content_hash follows the rule of every root source (class name,
output schema and source_id) plus the sorted tags_as_data items,
since two mappings can give one schema and one explicit source_id.
- pipeline_hash is schema-only, as for every root source.
Identity (plain form):
- content_hash is tied to the origin node's content hash.
- pipeline_hash is schema-only, so plain derived sources with
identical schemas share the same pipeline DB table.
If the origin is a job node that has not run yet, the source presents zero
rows with the correct columns. In the tags_as_data form the rows appear
once the node has run.
Data-tags (ITL-1019): the source carries the bindings of what it reads,
in both forms. Over a job node they are the ones the node's stored table states
(the marker of data_tags), which must be the node's own (a name in
both halves of its keys); a table that disagrees is refused. Over a source they
are the source's own. A tags_as_data copy of a data-tag is a new,
plain data column. Nothing is added to the identity for the binding, because it
is already there: the plain form's content_hash is the origin's, which carries
it, and every pipeline_hash and the tags_as_data form's content_hash
hash the output schema, in which a data-tag is in both halves.
Parameters:
- origin (
'FunctionJobNode | OperatorJobNode | RootSource'): The source or DB-backed job node to derive from. - source_id (
str | None) defaultNone: Canonical source name. Defaults as described above. - tags_as_data (
Mapping[str, str] | None) defaultNone: Maps a tag column of the origin to the name of the data column that receives a copy of it. - record_id_columns (
str | Collection[str] | None) defaultNone: Key columns whose values identify a row in record IDs and provenance tokens, instead of the row position. Only withtags_as_data. - **kwargs (
Any) default{}:label,data_context(defaults to the origin's) andconfig, passed toRootSource.
Raises:
InputValidationError: Iftags_as_datanames a column that is not a tag of the origin, or a new name that is already a column, is repeated, or is reserved.ValueError: Ifrecord_id_columnsis given withouttags_as_data.TypeError: If the origin is a source andtags_as_datais not given.
Example::
experiment = DerivedSource( SpiralDBTableSource(project, "experiment"), tags_as_data={"session_start_datetime": "session_start"},)identity_structure
Section titled “identity_structure”method · orcapod.core.sources.derived_source.DerivedSource.identity_structure
def identity_structure() -> AnyReturn what the content hash is built from.
The plain form is tied to the origin node's content hash. The
tags_as_data form is the root source rule (class name, output schema,
source_id, key columns) followed by the sorted mapping items: two mappings can give
the same schema (two tag columns of one type copied to one name) while the
copy holds different values, so with an explicit source_id the mapping
has to tell the sources apart (ITL-1000).
Returns:
Any: The identity structure.
to_config
Section titled “to_config”method · orcapod.core.sources.derived_source.DerivedSource.to_config
def to_config(db_registry: Any = None) -> dict[str, Any]Serialize this source to a JSON-compatible dict.
The origin's own config is embedded when the origin is a source, which
makes the derived source reloadable whenever its origin is. A default
source_id is not written: it is recomputed from the reloaded origin.
Parameters:
- db_registry (
Any) defaultNone: Optional database registry, forwarded to the origin.
Returns:
dict[str, Any]: The config dict, withsource_type"derived".
from_config
Section titled “from_config”classmethod · orcapod.core.sources.derived_source.DerivedSource.from_config
def from_config(config: dict[str, Any], db_registry: Any = None) -> 'DerivedSource'Rebuild a derived source by first rebuilding its origin.
Parameters:
- config (
dict[str, Any]): Dict as produced byto_config(). - db_registry (
Any) defaultNone: Optional database registry, forwarded to the origin.
Returns:
'DerivedSource': A newDerivedSourceover the rebuilt origin.
Raises:
NotImplementedError: If the config holds no origin: a derived source over a job node cannot be rebuilt from a config.
check_for_new_data
Section titled “check_for_new_data”method · orcapod.core.sources.derived_source.DerivedSource.check_for_new_data
def check_for_new_data() -> boolTake in the origin's rows if they changed since this source read them.
Developer, November 14, 2026 (ITL-1102). Over a source, the origin is checked
first (check_for_new_data). Then the origin's rows are read again only when
its modified time moved since this source last read them: a source origin that
took in new data, by this check or by a check made through another source over
it, or a job node that ran again. The rows read are compared with the rows
held (in any order only when a record_id_columns gives the record ids); when
they differ the source holds the new ones and moves its
modified time. A read or comparison that raises leaves everything as it was, so
the next check reads again. A source that holds nothing (it has not read its
origin yet, or read none of its rows: a job node that had not run) reads the
origin at its next read; when the origin's time moved, its own time moves too,
so a node that read no rows from it runs again.
Returns:
bool:Trueif the source's modified time moved: it holds new rows, or it holdsbool: none and its origin changed.
method · orcapod.core.sources.derived_source.DerivedSource.head
def head( n: int = 10, *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False,) -> StreamViewReturn the first rows of the origin's records, without computing.
The records are the ones this source already read. Otherwise, an origin source
gives its own head(n) (so a polling or caching origin is never polled or
written), and an origin job node gives its stored rows, read now and not kept.
An origin job node that has not run gives no rows. With sort_by_tags (or
all_info), the rows are sorted by tag as as_table sorts them: all the
stored rows of an origin job node, but only the rows an origin source reads for
its head (see StreamBase.head).
CachedSource
Section titled “CachedSource”class · orcapod.core.sources.cached_source.CachedSource
class CachedSource( source: SourceProtocol, cache_database: ArrowDatabaseProtocol, cache_path_prefix: tuple[str, ...] = (), cache_path: tuple[str, ...] | None = None, source_id: str | None = None, label: str | None = None, data_context: str | contexts.DataContext | None = None, config: OrcapodConfig | None = None,)Bases: RootSource
DB-backed wrapper around a source that caches every data.
Accepts any SourceProtocol implementation as the inner source.
Implements StreamProtocol transparently so downstream consumers
are unaware of caching. Cache table is scoped to the source's
content_hash() — each unique source gets its own table.
Example::
source = ArrowTableSource(table, tag_columns=["id"])cached = CachedSource(source, cache_database=db)# or equivalently:cached = source.cached(cache_database=db)SOURCE_KIND
Section titled “SOURCE_KIND”attribute · orcapod.core.sources.cached_source.CachedSource.SOURCE_KIND
Type: str
Config type key and default source-id kind of this source (one vocabulary).
to_config
Section titled “to_config”method · orcapod.core.sources.cached_source.CachedSource.to_config
def to_config(db_registry: DatabaseRegistryProtocol | None = None) -> dict[str, Any]Serialize this CachedSource configuration to a JSON-compatible dict.
Parameters:
- db_registry (
DatabaseRegistryProtocol | None) defaultNone: Optional registry. When provided, the cache database config is registered (deduplicated) and replaced by its key string. When None, the full database config is embedded inline.
Returns:
dict[str, Any]: Dict containing the inner source config, cache database config,dict[str, Any]: cache path prefix, and resolved cache path (for cache-only loading).
from_config
Section titled “from_config”classmethod · orcapod.core.sources.cached_source.CachedSource.from_config
def from_config( config: dict[str, Any], db_registry: DatabaseRegistryProtocol | None = None,) -> CachedSourceReconstruct a CachedSource from a config dict.
If the inner source cannot be resolved (e.g. it requires live data
that is unavailable), resolve_source_from_config returns a
SourceProxy preserving the original source's identity. The
CachedSource can still serve data from its cache database.
Parameters:
- config (
dict[str, Any]): Dict as produced byto_config. - db_registry (
DatabaseRegistryProtocol | None) defaultNone: Optional registry. Whencache_databaseis a string key, it is resolved viadb_registry.resolve(). When None and the cache_database value is a string key, a ValueError is raised.
Returns:
CachedSource: A new CachedSource constructed from the config.
cache_path
Section titled “cache_path”property · orcapod.core.sources.cached_source.CachedSource.cache_path
Type: tuple[str, ...]
Cache table path: scoped to the source's content hash, then the layout version.
The layout version (sdb_v1) is always the last component, also when an
explicit cache_path was given.
check_for_new_data
Section titled “check_for_new_data”method · orcapod.core.sources.cached_source.CachedSource.check_for_new_data
def check_for_new_data() -> boolCheck the wrapped source; move only if the next read shows new rows (ITL-1102).
The wrapped source is checked first. When its modified time moved since the cache last took in its rows (by this check, or by a check made through another source that wraps it), its rows are hashed and compared with the rows this source presents (recorded when it built its merged rows; before a build, the rows already cached: a read of the cache; nothing is written). The cache keeps every row it ever held, so rows that were removed from the wrapped source change nothing. When this source would present rows it has not (another source sharing the cache may already have written them), the merged rows held in memory are dropped (the next read writes the new rows to the cache and merges them) and the modified time moves. A comparison that raises leaves everything as it was, so the next check compares again.
The invariant this keeps: the modified time moves exactly when the next read would present a different set of rows. That read presents the rows cached (every row any source sharing the cache wrote, including rows since deleted from the wrapped source) and the live rows, so a row another source wrote to a shared cache moves this source too, and a row deleted from the wrapped source moves nothing.
The check guarantees nothing for two cached sources that share one cache (the
same database and cache_path, or the same table): give each cached source
its own cache. A change missed there means a run does not compute the new rows;
nothing wrong is stored (ITL-1143).
Returns:
bool:Trueif this source will present new rows at the next read.
is_stale
Section titled “is_stale”property · orcapod.core.sources.cached_source.CachedSource.is_stale
Type: bool
True if the wrapped source has been modified since the last build.
Overrides StreamBase.is_stale because CachedSource is a RootSource
(no upstreams/producer) yet still depends on the wrapped _source.
clear_cache
Section titled “clear_cache”method · orcapod.core.sources.cached_source.CachedSource.clear_cache
def clear_cache() -> NoneDiscard in-memory cached stream (forces rebuild on next access).
method · orcapod.core.sources.cached_source.CachedSource.head
def head( n: int = 10, *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False,) -> StreamViewReturn the first rows without reading or writing the cache.
Once the source has been used, its merged stream is in memory and this is a slice
of it. Before that its rows are not known: they are the cache's earlier rows
merged with the wrapped source's, and reading them writes the cache. The head
then has the wrapped source's columns, no rows and NOT_COMPUTED (or the
wrapped source's own reason when it has none to give).
get_all_records
Section titled “get_all_records”method · orcapod.core.sources.cached_source.CachedSource.get_all_records
def get_all_records( columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False,) -> 'pa.Table | None'Retrieve the cached records that the current Arrow hasher wrote.
SourceProxy
Section titled “SourceProxy”class · orcapod.core.sources.source_proxy.SourceProxy
class SourceProxy( source_id: str, content_hash_str: str, pipeline_hash_str: str, tag_schema: Schema, data_schema: Schema, expected_class_name: str | None = None, source_config: dict[str, Any] | None = None, label: str | None = None,)Bases: RootSource
A proxy source that preserves identity and optionally delegates.
When created without a bound source, SourceProxy returns stored hashes
and schemas but raises on data access. A live source can be substituted
in later via bind — if the source's identity matches, all data
methods delegate to it.
Parameters:
- source_id (
str): The original source's canonical ID. - content_hash_str (
str): The original source's content hash string. - pipeline_hash_str (
str): The original source's pipeline hash string. - tag_schema (
Schema): The original source's tag schema. - data_schema (
Schema): The original source's data schema. - expected_class_name (
str | None) defaultNone: Class name of the original source (e.g."ArrowTableSource"). Informational and used for validation inbind. - source_config (
dict[str, Any] | None) defaultNone: The original source's serialized config (for to_config). - label (
str | None) defaultNone: Optional label.
expected_class_name
Section titled “expected_class_name”property · orcapod.core.sources.source_proxy.SourceProxy.expected_class_name
Type: str | None
The class name of the original source this proxy stands in for.
delegate
Section titled “delegate”property · orcapod.core.sources.source_proxy.SourceProxy.delegate
Type: SourceProtocol | None
The bound live source, or None if no source has been bound.
is_bound
Section titled “is_bound”property · orcapod.core.sources.source_proxy.SourceProxy.is_bound
Type: bool
True if a live source has been bound via bind.
method · orcapod.core.sources.source_proxy.SourceProxy.bind
def bind(source: SourceProtocol) -> NoneBind a live source to this proxy, enabling data access.
The source must match this proxy's identity — same source_id,
content_hash, pipeline_hash, tag schema keys, and data
schema keys. If expected_class_name is set, the source's class
name must also match.
Parameters:
- source (
SourceProtocol): The live source to bind.
Raises:
ValueError: If the source's identity does not match.
unbind
Section titled “unbind”method · orcapod.core.sources.source_proxy.SourceProxy.unbind
def unbind() -> SourceProtocol | NoneRemove and return the bound source, reverting to proxy behavior.
Returns:
SourceProtocol | None: The previously bound source, orNoneif none was bound.
last_modified
Section titled “last_modified”property · orcapod.core.sources.source_proxy.SourceProxy.last_modified
Type: datetime | None
When the proxy's data last changed: its binding or its bound source.
The later of the time the proxy was last bound or unbound and the bound
source's own last_modified; a bound source that does not know its time
makes the proxy's unknown too (None). So a bound source that takes in new
data at a run's check moves the proxy with it (ITL-1102).
check_for_new_data
Section titled “check_for_new_data”method · orcapod.core.sources.source_proxy.SourceProxy.check_for_new_data
def check_for_new_data() -> boolCheck the bound source for new data; an unbound proxy has none (ITL-1102).
Returns:
bool:Trueif the bound source took in new data.
method · orcapod.core.sources.source_proxy.SourceProxy.head
def head( n: int = 10, *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False,) -> StreamViewReturn the bound source's head, or no rows when no live source is bound.
to_config
Section titled “to_config”method · orcapod.core.sources.source_proxy.SourceProxy.to_config
def to_config(db_registry=None) -> dict[str, Any]Return the original source's config (preserves source_type).
from_config
Section titled “from_config”classmethod · orcapod.core.sources.source_proxy.SourceProxy.from_config
def from_config(config: dict[str, Any], db_registry=None) -> SourceProxyNot supported — SourceProxy is created by the deserialization pipeline.
SourceRegistry
Section titled “SourceRegistry”class · orcapod.core.sources.source_registry.SourceRegistry
class SourceRegistry()Registry mapping canonical source IDs to live RootSource objects.
A source ID is a stable, human-readable name (e.g. "delta_table:sales")
that is independent of physical location. The registry lets downstream
code resolve a source_id token embedded in a provenance string back to
the concrete source object that produced it, without requiring a direct
reference to the source object.
Registration behaviour
Section titled “Registration behaviour”- Registering the same object under the same ID is idempotent.
- Registering a different object under an already-taken ID logs a warning and skips (rather than raising), so that sources constructed in different contexts don't crash each other via the global singleton.
- Use
replacewhen you explicitly want to overwrite an entry.
The module-level GLOBAL_SOURCE_REGISTRY is the default registry used
when no explicit registry is provided.
register
Section titled “register”method · orcapod.core.sources.source_registry.SourceRegistry.register
def register(source_id: str, source: 'RootSource') -> NoneRegister source under source_id.
If source_id is already taken by the same object, the call is a no-op. If it is taken by a different object, a warning is emitted and the existing entry is left unchanged.
replace
Section titled “replace”method · orcapod.core.sources.source_registry.SourceRegistry.replace
def replace(source_id: str, source: 'RootSource') -> 'RootSource | None'Register source under source_id, unconditionally replacing any existing entry. Returns the previous source if one existed.
unregister
Section titled “unregister”method · orcapod.core.sources.source_registry.SourceRegistry.unregister
def unregister(source_id: str) -> 'RootSource'Remove and return the source registered under source_id.
method · orcapod.core.sources.source_registry.SourceRegistry.get
def get(source_id: str) -> 'RootSource'Return the source for source_id, raising KeyError if absent.
get_optional
Section titled “get_optional”method · orcapod.core.sources.source_registry.SourceRegistry.get_optional
def get_optional(source_id: str) -> 'RootSource | None'Return the source for source_id, or None if not registered.
RootSource
Section titled “RootSource”class · orcapod.core.sources.base.RootSource
class RootSource( source_id: str | None = None, label: str | None = None, data_context: str | contexts.DataContext | None = None, config: OrcapodConfig | None = None,)Bases: StreamBase
Abstract base class for all root sources in Orcapod.
A RootSource is a pure stream — the root of a computational graph, producing data from an external source (file, database, in-memory data, etc.) with no upstream dependencies.
As a StreamProtocol:
sourcereturnsNone(no upstream source pod)upstreamsis always emptykeys,output_schema,iter_data,as_tabledelegate toself._streamby default; concrete subclasses may override them.
As a PipelineElementProtocol:
pipeline_identity_structure()returns(tag_schema, data_schema)— schema-only, no data content — forming the base case of the pipeline identity Merkle chain.
Concrete subclasses must set self._stream in their __init__ to get
the default stream delegation behavior. They may also override
identity_structure().
source_id
Section titled “source_id”property · orcapod.core.sources.base.RootSource.source_id
Type: str
Canonical name for this source used in the registry and provenance
strings. If not set, raises ValueError.
identity_structure
Section titled “identity_structure”method · orcapod.core.sources.base.RootSource.identity_structure
def identity_structure() -> AnyIdentity from the class name, output schema, source_id and key columns.
The key columns are the resolved record_id_columns (() for the
position): two sources that differ only in how they name their rows are two
sources, so their records never share a cache (developer, November 16, 2026,
ITL-1154). The pipeline hash stays schema-only.
pipeline_identity_structure
Section titled “pipeline_identity_structure”method · orcapod.core.sources.base.RootSource.pipeline_identity_structure
def pipeline_identity_structure() -> AnyReturn (tag_schema, data_schema) as the pipeline identity for this source. Schema-only: no data content is included, so sources with identical schemas share the same pipeline hash and therefore the same pipeline database table.
producer
Section titled “producer”property · orcapod.core.sources.base.RootSource.producer
Type: None
Root sources have no upstream source pod.
upstreams
Section titled “upstreams”property · orcapod.core.sources.base.RootSource.upstreams
Type: tuple[StreamProtocol, ...]
Sources have no upstream dependencies.
output_schema
Section titled “output_schema”method · orcapod.core.sources.base.RootSource.output_schema
def output_schema( *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False,) -> tuple[Schema, Schema]Delegate to the underlying stream's output_schema.
method · orcapod.core.sources.base.RootSource.keys
def keys( *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False,) -> tuple[tuple[str, ...], tuple[str, ...]]Delegate to the underlying stream's keys.
iter_data
Section titled “iter_data”method · orcapod.core.sources.base.RootSource.iter_data
def iter_data()Delegate to the underlying stream's iter_data.
as_table
Section titled “as_table”method · orcapod.core.sources.base.RootSource.as_table
def as_table( *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False,) -> 'pa.Table'Delegate to the underlying stream's as_table.
check_for_new_data
Section titled “check_for_new_data”method · orcapod.core.sources.base.RootSource.check_for_new_data
def check_for_new_data() -> boolCheck whether the data behind the source changed; if so, take it in.
Every run (job.run(), run_async(), a job node's run(), a load-only
run too) calls this once on each source feeding it, before deciding which
nodes to run (developer, November 14, 2026, ITL-1102). A source that finds new
data takes it in and moves its modified time to that moment, so every job node
below it is stale and runs. A source that finds nothing changes nothing, so a
current job stays current. head() and the notebook preview never call it.
A source built over data held in memory (ArrowTableSource, DictSource,
ListSource, DataFrameSource) cannot change: this default returns
False. A source that raises here does not fail the run: the run issues a
SourceCheckFailedWarning and proceeds with the data the source holds, so a
source must leave its data and modified time as they were when it raises. A
source whose reads refresh it (PollingSource: a read polls) is refreshed
again by a run that reads it, and a refresh that fails then raises (ITL-1139).
Returns:
bool:Trueif the source found new data and took it in.
method · orcapod.core.sources.base.RootSource.head
def head( n: int = 10, *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False,) -> StreamViewReturn the first n rows of the table the source holds, without computing.
A source that reads its table when it is built holds it in memory, so this is a
slice of it. A source that reads its table later overrides this with a bounded
read. See StreamBase.head.
cached
Section titled “cached”method · orcapod.core.sources.base.RootSource.cached
def cached( cache_database: ArrowDatabaseProtocol, cache_path_prefix: tuple[str, ...] = (), **kwargs: Any,) -> CachedSourceReturn a CachedSource wrapping this source.
Parameters:
- cache_database (
ArrowDatabaseProtocol): Database to store cached records in. - cache_path_prefix (
tuple[str, ...]) default(): Path prefix for the cache table. - **kwargs (
Any) default{}: Additional keyword arguments passed toCachedSource.
Returns:
CachedSource: ACachedSourcethat caches this source's output.