Skip to content
This site documents the development version of Orcapod (the main branch), which may differ from the latest release. For the releases, see PyPI and the changelog.

Nodes

Nodes are the elements of a compiled pipeline graph. The plain node classes describe the graph of a Pipeline blueprint; the job node classes below add database-backed execution.

class · orcapod.core.nodes.function_node.FunctionNode

class FunctionNode(
function_pod: FunctionPodProtocol,
input_stream: StreamProtocol,
tracker_manager: TrackerManagerProtocol | None = None,
label: str | None = None,
config: OrcapodConfig | None = None,
table_scope: Literal['pipeline_hash', 'content_hash'] = 'pipeline_hash',
)

Bases: FunctionNodeBase

Thin blueprint descriptor for a function pod invocation.

Carries no database references. Calling iter_data() raises PipelineJobRequiredError — wrap the containing Pipeline in a PipelineJob to obtain an executable FunctionJobNode.

This is the node type recorded inside a Pipeline context manager and serialized to disk via Pipeline.save().

method · orcapod.core.nodes.function_node.FunctionNode.set_ephemeral_database

def set_ephemeral_database(database: 'ArrowDatabaseProtocol | None') -> None

No-op — blueprint nodes carry no database references.

classmethod · orcapod.core.nodes.function_node.FunctionNode.from_descriptor

def from_descriptor(
descriptor: dict[str, Any],
function_pod: FunctionPodProtocol | None,
input_stream: StreamProtocol | None,
databases: dict[str, Any],
) -> 'FunctionNode'

Construct a FunctionNode from a serialized descriptor.

When function_pod and input_stream are both provided the node operates in full mode -- constructed normally via __init__. When function_pod is None the node is created in read-only mode with metadata from the descriptor; computation methods will raise PipelineJobRequiredError.

Parameters:

  • descriptor (dict[str, Any]): The serialized node descriptor dict.
  • function_pod (FunctionPodProtocol | None): An optional live function pod. None for read-only mode.
  • input_stream (StreamProtocol | None): An optional live input stream. None for read-only mode.
  • databases (dict[str, Any]): Mapping of database role names ("pipeline", "result") to database instances.

Returns:

  • 'FunctionNode': A new FunctionNode instance.

method · orcapod.core.nodes.function_node.FunctionNode.iter_data

def iter_data() -> Iterator[tuple[TagProtocol, DataProtocol]]

Raise PipelineJobRequiredError — blueprint nodes cannot produce data.

Raises:

  • PipelineJobRequiredError: Always.

method · orcapod.core.nodes.function_node.FunctionNode.as_node

def as_node() -> FunctionNode

Return the lightweight blueprint equivalent of this node.

Returns a fresh FunctionNode with the same function pod, input stream, label, table scope, and tracker manager. Its content_hash() / pipeline_hash() are identical to those of this node.

Returns:

Raises:

  • RuntimeError: If this node is in UNAVAILABLE state (no live function pod). UNAVAILABLE nodes cannot be cloned into a usable blueprint — callers must handle this status before invoking as_node().

class · orcapod.core.nodes.operator_node.OperatorNode

class OperatorNode(
operator: OperatorPodProtocol,
input_streams: Collection[StreamProtocol],
tracker_manager: TrackerManagerProtocol | None = None,
label: str | None = None,
config: OrcapodConfig | None = None,
table_scope: Literal['pipeline_hash', 'content_hash'] = 'pipeline_hash',
)

Bases: OperatorNodeBase

Thin blueprint descriptor for an operator pod invocation.

Carries no database references. Calling iter_data() raises PipelineJobRequiredError — wrap the containing Pipeline in a PipelineJob to obtain an executable OperatorJobNode.

This is the node type recorded inside a Pipeline context manager and serialized to disk via Pipeline.save().

method · orcapod.core.nodes.operator_node.OperatorNode.iter_data

def iter_data() -> Iterator[tuple[TagProtocol, DataProtocol]]

Raise PipelineJobRequiredError — blueprint nodes cannot produce data.

Raises:

  • PipelineJobRequiredError: Always.

method · orcapod.core.nodes.operator_node.OperatorNode.head

def head(
n: int = 10,
*,
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> StreamView

Return no rows: a blueprint node holds no data.

method · orcapod.core.nodes.operator_node.OperatorNode.as_table

def as_table(
*,
columns: ColumnConfig | dict[Any, Any] | None = None,
all_info: bool = False,
) -> pa.Table

Raise PipelineJobRequiredError — blueprint nodes cannot produce data.

Raises:

  • PipelineJobRequiredError: Always.

classmethod · orcapod.core.nodes.operator_node.OperatorNode.from_descriptor

def from_descriptor(
descriptor: dict[str, Any],
operator: OperatorPodProtocol | None,
input_streams: tuple[StreamProtocol, ...] | list[StreamProtocol],
databases: dict[str, Any],
) -> 'OperatorNode'

Construct an OperatorNode from a serialized descriptor.

When operator and input_streams are provided the node operates in full mode — constructed normally via __init__. When operator is None the node is created in read-only mode with metadata from the descriptor; computation methods will raise PipelineJobRequiredError.

Parameters:

  • descriptor (dict[str, Any]): The serialized node descriptor dict.
  • operator (OperatorPodProtocol | None): An optional live operator instance. None for read-only mode.
  • input_streams (tuple[StreamProtocol, ...] | list[StreamProtocol]): Input streams for the operator. Empty tuple for read-only mode.
  • databases (dict[str, Any]): Unused — kept for API parity with OperatorJobNode.from_descriptor.

Returns:

  • 'OperatorNode': A new OperatorNode instance.

Raises:

  • ValueError: If table_scope is missing or invalid.

method · orcapod.core.nodes.operator_node.OperatorNode.set_ephemeral_database

def set_ephemeral_database(database: 'ArrowDatabaseProtocol | None') -> None

No-op — blueprint operator nodes carry no database references.

method · orcapod.core.nodes.operator_node.OperatorNode.as_node

def as_node() -> OperatorNode

Return the lightweight blueprint equivalent of this node.

Returns a fresh OperatorNode with the same operator, input streams, label, table scope, and tracker manager. Its content_hash() / pipeline_hash() are identical to those of this node.

Returns:

Raises:

  • RuntimeError: If this node is in UNAVAILABLE state (no live operator). UNAVAILABLE nodes cannot be cloned into a usable blueprint — callers must handle this status before invoking as_node().

class · orcapod.core.nodes.source_node.SourceNode

class SourceNode(
name: str,
tag_schema: Schema,
data_schema: Schema,
data_context: str | contexts.DataContext | None = None,
label: str | None = None,
config: OrcapodConfig | None = None,
)

Bases: SourceNodeBase

Schema-only input-slot declaration for Pipeline recording.

Replaces SourceSpec as the user-facing way to declare typed pipeline inputs. Pass a SourceNode inside a with pipeline: block as the upstream for any pod invocation.

Example::

slot = SourceNode(name="data", tag_schema={"id": int}, data_schema={"v": float})
with pipeline:
result = my_pod(slot)
job = PipelineJob.from_pipeline(pipeline, database=db, sources={"data": my_source})
job.run()

method · orcapod.core.nodes.source_node.SourceNode.iter_data

def iter_data() -> Iterator[tuple[TagProtocol, DataProtocol]]

Raise UnboundSourceError — SourceNode carries no data.

Raises:

  • UnboundSourceError: Always.

classmethod · orcapod.core.nodes.source_node.SourceNode.from_stream

def from_stream(stream: StreamProtocol, name: str | None = None) -> SourceNode

Wrap stream in a SourceNode, or return it unchanged if already one.

Derives tag and data schemas from stream.output_schema(). The slot name is resolved in priority order:

  1. name parameter (explicit override — highest priority).
  2. stream.source_id when stream is a RootSource — the canonical source identity, stable across pipeline serialisation / deserialisation.
  3. "{stream.label}:{hash_prefix}" for any other concrete stream type, combining the cosmetic label with a short content-hash suffix to ensure uniqueness.

Parameters:

  • stream (StreamProtocol): The upstream stream to wrap.
  • name (str | None) default None: Optional explicit slot name.

Returns:

  • SourceNode: The original stream if it is already a SourceNode; otherwise
  • SourceNode: a new SourceNode with schemas derived from stream.

The database-backed nodes a PipelineJob builds; these are what job.nodes[label] returns.

class · orcapod.core.nodes.function_node.FunctionJobNode

class FunctionJobNode(
function_pod: FunctionPodProtocol,
input_stream: StreamProtocol,
tracker_manager: TrackerManagerProtocol | None = None,
label: str | None = None,
config: OrcapodConfig | None = None,
pipeline_database: ArrowDatabaseProtocol | None = None,
result_database: ArrowDatabaseProtocol | None = None,
ephemeral_database: ArrowDatabaseProtocol | None = None,
table_scope: Literal['pipeline_hash', 'content_hash'] = 'pipeline_hash',
)

Bases: RunHoldingNode, FunctionNodeBase

DB-backed execution node for function pod invocations.

Created by PipelineJob at run time; never recorded inside a plain Pipeline. Carries all persistence logic: CachedFunctionPod wrapping, pipeline records, and two-phase iter_data() / async execution.

property · orcapod.core.nodes.function_node.FunctionJobNode.node_config

Type: NodeConfig

Per-node pipeline execution configuration.

method · orcapod.core.nodes.function_node.FunctionJobNode.require_storable_tables

def require_storable_tables(
pipeline_database: ArrowDatabaseProtocol,
result_database: ArrowDatabaseProtocol | None = None,
) -> None

Refuse databases that cannot store this node's tables (ITL-1162).

Checks the result table (CachedFunctionPod.require_storable_results) against result_database and the pipeline records' own columns against pipeline_database. Nothing is computed and nothing changes. A node loaded without its function stores nothing and is not checked.

Parameters:

  • pipeline_database (ArrowDatabaseProtocol): The database for pipeline records.
  • result_database (ArrowDatabaseProtocol | None) default None: The database for results; defaults to pipeline_database.at("_result"), as in attach_databases.

Raises:

  • InputValidationError: If the node passes data-tags through and pipeline_database cannot keep their marker (ITL-1019).
  • UnstorableColumnError: If a database cannot store a column, naming this node.

method · orcapod.core.nodes.function_node.FunctionJobNode.attach_databases

def attach_databases(
pipeline_database: ArrowDatabaseProtocol | None = None,
result_database: ArrowDatabaseProtocol | None = None,
ephemeral_database: ArrowDatabaseProtocol | None = None,
) -> None

Attach databases for persistent caching and pipeline records.

For live nodes (_function_pod is set), creates a CachedFunctionPod wrapping the original function pod for result caching. For read-only stubs (_function_pod is None, e.g. nodes loaded from a saved job), creates a _ResultDatabaseReader using the record path stored in the descriptor so that get_all_records() and _load_cached_entries() still work.

When pipeline_database is None, the persistent-DB portion of setup is skipped entirely — no CachedFunctionPod is created and _pipeline_database is left as None. Only ephemeral_database wiring proceeds in that case.

The databases are expected to be pre-scoped by the pipeline (via db.at(*pipeline_name).at("_result") etc.) so no additional path prefix is needed here.

Parameters:

  • pipeline_database (ArrowDatabaseProtocol | None) default None: Database for pipeline records. Pass None to skip persistent-DB setup (ephemeral-only or deferred wiring).
  • result_database (ArrowDatabaseProtocol | None) default None: Database for cached results. Defaults to pipeline_database.at("_result") when pipeline_database is not None.
  • ephemeral_database (ArrowDatabaseProtocol | None) default None: Optional ephemeral database to attach immediately via set_ephemeral_database(). Equivalent to calling set_ephemeral_database(ephemeral_database) after attach_databases().

classmethod · orcapod.core.nodes.function_node.FunctionJobNode.from_descriptor

def from_descriptor(
descriptor: dict[str, Any],
input_stream: StreamProtocol | None = None,
databases: dict[str, Any] | None = None,
job_content_hash: str | None = None,
expected_row_count: int | None = None,
) -> 'FunctionJobNode'

Create a read-only FunctionJobNode stub from a serialized descriptor.

Used by PipelineJob.load() to populate _persistent_node_map with database-backed stubs for function nodes. The stubs support get_all_records() and _load_cached_entries() on previously computed results without needing a live function_pod.

Computation-path methods (execute(), iter_data()) will fail for stub nodes because _function_pod is None.

Parameters:

  • descriptor (dict[str, Any]): Serialized node descriptor dict (as produced by Pipeline.save()).
  • input_stream (StreamProtocol | None) default None: Optional upstream stream. May be None for read-only stubs where execution is not needed.
  • databases (dict[str, Any] | None) default None: Optional mapping {"pipeline": db, "result": db}. When "pipeline" is present, attach_databases() is called immediately so the stub is ready for DB queries.
  • job_content_hash (str | None) default None: Optional live (data-inclusive) content hash from the run that produced the records being loaded. When provided, overrides the blueprint hash stored in descriptor so that content_hash() matches the _node_content_hash column written to the DB at execution time. Required for get_all_records() to correctly filter DB rows when table_scope="pipeline_hash" (the default).
  • expected_row_count (int | None) default None: How many pipeline records of this node (with its run's content hash) the database held when the job was saved, from the job file's optional run.expected_rows; None when not recorded. Advisory: a read always returns what the database holds and only warns (OrcapodDatabaseContentWarning) when the database holds another number.

Returns:

  • 'FunctionJobNode': A FunctionJobNode in read-only stub mode
  • 'FunctionJobNode': (load_status=UNAVAILABLE).

method · orcapod.core.nodes.function_node.FunctionJobNode.clear_cache

def clear_cache() -> None

Clear in-memory output caches, content hash cache, and node identity path cache.

The node then holds no run: it is stale and reads empty, as before its first run in this process (ITL-1095). Its stored results stay stored (as_source()).

property · orcapod.core.nodes.function_node.FunctionJobNode.last_run_not_stored

Type: int

Inputs the node's last completed run skipped because nothing was stored.

Non-zero only after a load-only run (run(load_only=True)); 0 before a run.

method · orcapod.core.nodes.function_node.FunctionJobNode.set_ephemeral_database

def set_ephemeral_database(database: 'ArrowDatabaseProtocol | None') -> None

Assign or remove the ephemeral result database.

When database is not None, creates a CachedFunctionPod backed by database so that ephemeral writes use the same format as persistent writes. When database is None, clears both the database and the ephemeral pod.

Parameters:

  • database ('ArrowDatabaseProtocol | None'): The ArrowDatabaseProtocol to use for ephemeral result storage, or None to detach and revert to persistent-only writes.

method · orcapod.core.nodes.function_node.FunctionJobNode.as_node

def as_node() -> FunctionNode

Return the lightweight FunctionNode equivalent of this job node.

Returns:

  • FunctionNode: A new FunctionNode with the same function pod, input stream,
  • FunctionNode: label, table scope, and tracker manager. Its content_hash() /
  • FunctionNode: pipeline_hash() are identical to those of this FunctionJobNode.

method · orcapod.core.nodes.function_node.FunctionJobNode.execute_data

def execute_data(
tag: TagProtocol,
data: DataProtocol,
) -> tuple[TagProtocol, DataProtocol | None]

Execute a single data: compute, persist, and cache.

Internal method for orchestrators. The caller must guarantee that the tag and data conform to the expected input schema (matching self._input_stream). No validation is performed.

Parameters:

  • tag (TagProtocol): The tag associated with the data.
  • data (DataProtocol): The input data to process.

Returns:

  • tuple[TagProtocol, DataProtocol | None]: A (tag, output_data) tuple.

method · orcapod.core.nodes.function_node.FunctionJobNode.execute

def execute(
input_stream: StreamProtocol,
*,
observer: ExecutionObserverProtocol | None = None,
error_policy: Literal['continue', 'fail_fast'] = 'continue',
run_id: str | None = None,
load_only: bool = False,
) -> list[tuple[TagProtocol, DataProtocol]]

Execute all data from a stream: compute, persist, and cache.

For each data: fire on_data_start, check the in-memory cache (populated from DB if needed), compute if missing, fire on_data_end or on_data_crash.

Afterwards the node's rows (iter_data, as_table, head) are those of this run's inputs. If the run raises, it defines no rows: the node drops its in-memory rows and reads as a node with no run in this process; what was stored stays stored.

Parameters:

  • input_stream (StreamProtocol): The input stream to process.
  • observer (ExecutionObserverProtocol | None) default None: Optional execution observer for hooks.
  • error_policy (Literal['continue', 'fail_fast']) default 'continue': "continue" skips failed data; "fail_fast" re-raises on the first failure.
  • run_id (str | None) default None: Optional pipeline run identifier threaded through to InvocationContext.pipeline_run_id.
  • load_only (bool) default False: Fill the node with the results already stored for the stream's rows and compute nothing (ITL-1095): no function call, no write; inputs without a stored result are skipped and counted (last_run_not_stored). A loaded node (no function pod) can run this way.

Returns:

  • list[tuple[TagProtocol, DataProtocol]]: Materialized list of (tag, output_data) pairs, excluding
  • list[tuple[TagProtocol, DataProtocol]]: None outputs and failed data.

Raises:

  • RuntimeError: If the node was loaded by PipelineJob.load and so has no function pod to compute with.
  • InputValidationError: If the input stream is of another data context than this node; raised before any row is processed, whatever the error policy.

method · orcapod.core.nodes.function_node.FunctionJobNode.get_cached_results

def get_cached_results(
base_entry_ids: list[bytes],
) -> dict[bytes, tuple[TagProtocol, DataProtocol]]

Public cache façade: return already-computed results for the given base entry IDs.

Serves hits directly from the in-memory cache (_cached_output_datas). For IDs not yet cached, delegates to _load_cached_entries which calls _fetch_joined_records to load from the pipeline and result databases. Add-only semantics: existing in-memory entries are never cleared or overwritten.

Does NOT apply user-facing column filtering — see get_all_records for that.

Parameters:

  • base_entry_ids (list[bytes]): Stable base entry IDs (from compute_base_entry_id) to look up.

Returns:

  • dict[bytes, tuple[TagProtocol, DataProtocol]]: Mapping from base_entry_id to (tag, output_data) for found entries.
  • dict[bytes, tuple[TagProtocol, DataProtocol]]: Empty dict if no DB is attached, base_entry_ids is empty, or no
  • dict[bytes, tuple[TagProtocol, DataProtocol]]: matches are found.

method · orcapod.core.nodes.function_node.FunctionJobNode.compute_base_entry_id

def compute_base_entry_id(tag: TagProtocol, input_data: DataProtocol) -> bytes

Computes the stable (recomputation-index-free) entry ID for a (tag, data) pair.

Hashes the tag's system-tag columns plus INPUT_DATA_HASH_COL (binary). Because it excludes the recomputation index it is stable across all recomputation attempts for the same logical input.

The base entry ID is stored in _PIPELINE_BASE_ENTRY_ID_COL and will be used as the in-memory cache key and Phase 1 filter in subsequent tasks.

Parameters:

  • tag (TagProtocol): The tag datagram for the input row.
  • input_data (DataProtocol): The data datagram for the input row.

Returns:

  • bytes: A bytes value in b"{method}:{digest}" format.

method · orcapod.core.nodes.function_node.FunctionJobNode.compute_pipeline_entry_id

def compute_pipeline_entry_id(
tag: TagProtocol,
input_data: DataProtocol,
recomputation_index: int = 0,
) -> bytes

Compute a versioned pipeline entry ID from tag + system tags + input data hash + index.

Extends the base preimage (see _build_entry_id_preimage) with a _PIPELINE_RECOMPUTATION_INDEX_COL column (value: recomputation_index, type pa.int32()) so that each recomputation attempt receives a distinct DB primary key.

At recomputation_index=0 this produces a hash that differs from the pre-ITL-508 implementation (the index column is now part of the preimage). Existing pipeline DB records are implicitly invalidated — acceptable because the project is pre-v0.1.0.

The preimage is system_tags + INPUT_DATA_HASH_COL + recomputation_index.

Parameters:

  • tag (TagProtocol): The tag datagram for the input row.
  • input_data (DataProtocol): The data datagram for the input row.
  • recomputation_index (int) default 0: Position in the recomputation chain. 0 for the first computation, N+1 for each miss-triggered recompute.

Returns:

  • bytes: Method-prefixed raw bytes (b"{method}:{digest}") uniquely
  • bytes: identifying this (tag, input_data, node run, recomputation attempt).
  • bytes: Suitable for storage in a pa.large_binary() column.

method · orcapod.core.nodes.function_node.FunctionJobNode.add_pipeline_record

def add_pipeline_record(
tag: TagProtocol,
input_data: DataProtocol,
data_record_id: uuid.UUID,
computed: bool,
is_ephemeral: bool = False,
output_data_hash: ContentHash | None = None,
) -> None

Add a pipeline record to the database for a processed data.

Computes the next recomputation index by querying existing rows in the pipeline DB that share the same base_entry_id, then writes at max_index + 1. The write uses skip_duplicates=True so that concurrent asyncio coroutines competing for the same versioned entry ID are safely serialised: the first writer lands, subsequent writers whose compute happened to produce the same hash are silently no-oped.

The pipeline record stores:

  • Tag columns (including system tags)
  • All source columns of the input data (provenance, not data values)
  • Output data record ID (for joining with result records)
  • Base entry ID (_PIPELINE_BASE_ENTRY_ID_COL)
  • Recomputation index (_PIPELINE_RECOMPUTATION_INDEX_COL)
  • Whether the result is stored in the ephemeral database
  • Input data context key
  • Whether the result was freshly computed or cached
  • Output data content hash (OUTPUT_DATA_HASH_COL) — used by downstream EmptyData flow-through to look up the downstream result cache.

Parameters:

  • tag (TagProtocol): The tag associated with the input data.
  • input_data (DataProtocol): The input data that was processed.
  • data_record_id (uuid.UUID): UUID of the result record in the result database.
  • computed (bool): Whether the result was freshly computed (True) or served from a cache (False).
  • is_ephemeral (bool) default False: Whether the result is stored in the ephemeral database.
  • output_data_hash (ContentHash | None) default None: Content hash of the output produced by processing input_data. When provided, stored as OUTPUT_DATA_HASH_COL so that downstream EmptyData flow-through can locate its cached result keyed by the same hash (= the downstream's INPUT_DATA_HASH_COL).

method · orcapod.core.nodes.function_node.FunctionJobNode.get_all_records

def get_all_records(
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> pa.Table | None

Public table view: return all computed results joined with their pipeline records.

Calls _fetch_joined_records to obtain the raw joined table, then applies ColumnConfig-driven column dropping to produce a user-facing result. NODE_CONTENT_HASH_COL, _PIPELINE_ENTRY_ID_COL, _PIPELINE_BASE_ENTRY_ID_COL, and _PIPELINE_RECOMPUTATION_INDEX_COL are always dropped — they are internal discriminator columns, not user-facing data. These columns are listed explicitly to ensure they are dropped even when all_info=True (which skips the meta-prefix sweep).

Does NOT populate the in-memory cache — see get_cached_results for that.

Parameters:

  • columns (ColumnConfig | dict[str, Any] | None) default None: Column configuration controlling which groups are included. Accepts a ColumnConfig instance or a dict shorthand (e.g. {"meta": True}).
  • all_info (bool) default False: If True, equivalent to enabling all column groups.

Returns:

  • pa.Table | None: A pa.Table of joined results, or None if no database is
  • pa.Table | None: attached, either DB fetch returns None, or the join produces
  • pa.Table | None: no rows.

Raises:

  • RuntimeError: If the node was loaded by PipelineJob.load and no database is attached.

Warns:

  • OrcapodDatabaseContentWarning: Once per database, if the node was loaded from a job file that records another number of pipeline records for its run than the database holds. The database is the source of truth.

method · orcapod.core.nodes.function_node.FunctionJobNode.as_source

def as_source(tags_as_data: 'Mapping[str, str] | None' = None)

Return a DerivedSource backed by the DB records of this node.

Its source_id is the DerivedSource default: without tags_as_data, derived:<algorithm label>:<hex> of this node's content hash.

Parameters:

  • tags_as_data ('Mapping[str, str] | None') default None: Maps a tag column of this node to the name of a data column that receives a copy of it. When given, the result is a new root source with its own provenance and a source_id derived from this node's content hash and the mapping; see DerivedSource.

Raises:

  • RuntimeError: If no database is attached.
  • InputValidationError: If tags_as_data is not valid for this node's columns.

method · orcapod.core.nodes.function_node.FunctionJobNode.head

def head(
n: int = 10,
*,
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> StreamView

Return the first rows of the node's current run; never runs the function.

After a run (or execute) the rows are the node's results in memory for that run's inputs, the first of what as_table returns. A node that has not run in this process, or whose input changed since (stale), shows no rows and says so, pointing to run() and to as_source() for its stored results (ITL-1095); the database is not read. The node has all of its rows, so with sort_by_tags (or all_info) the head is exactly as_table(...).slice(0, n), whatever order the rows were computed in. See StreamBase.head.

method · orcapod.core.nodes.function_node.FunctionJobNode.as_table

def as_table(
*,
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> 'pa.Table'

Return the rows of this node's current run as a PyArrow table.

Read-only: never computes and never reads the database. Before a run in this process, or once the node is stale, the table has zero rows and the node's columns (ITL-1095); as_source() reads what the database holds.

Raises:

  • RuntimeError: If the node was loaded and no database is attached.

method · orcapod.core.nodes.function_node.FunctionJobNode.iter_data

def iter_data() -> Iterator[tuple[TagProtocol, DataProtocol]]

Yield the (tag, data) rows of this node's current run.

Strictly read-only: never computes and never reads the database. Callers must call run() (or run the job) first.

After a completed run (execute, async_execute, run() or a job run) the rows are exactly those of that run's inputs, on either orchestrator, even though results of other inputs stay cached and stored (ITL-1093). A node that has not run in this process, whose run raised, or that is stale (its input changed since its run) yields nothing (ITL-1095); as_source() reads every row stored for it.

Raises:

  • RuntimeError: If load_status is UNAVAILABLE.

method · orcapod.core.nodes.function_node.FunctionJobNode.run

def run(*, load_only: bool = False) -> StreamView

Run the node as a job run would, and return a StreamView of its rows.

State-driven (developer, November 13, 2026; ITL-1095): a node that holds the rows of a completed run and is not stale is not run again; the view shows what it holds, and nothing is computed, read or written. Otherwise the node runs, after every job node it depends on that holds no current run (each once, in dependency order, as job.run() would run them; last_run reports the counts), and the run's writes are committed. A node built over streams outside a job reads its input as it is.

Parameters:

  • load_only (bool) default False: Fill the node, and the nodes it runs first, with what is already stored and compute nothing (as job.run(load_only=True)). A node filled this way shows status loaded_only and is not current for a later full run(), which computes what it skipped.

Returns:

  • StreamView: A StreamView of every row as_table() then returns: status rows,
  • StreamView: or loaded_only after a load-only run.

Raises:

  • RuntimeError: If no database is attached (UNAVAILABLE); if the node cannot compute (loaded without a function pod, or a placeholder pod) and load_only is false; or if its input is unavailable (CACHE_ONLY): read its stored rows with as_source().

method · orcapod.core.nodes.function_node.FunctionJobNode.async_execute

async def async_execute(
input_channel: ReadableChannel[tuple[TagProtocol, DataProtocol]],
output: WritableChannel[tuple[TagProtocol, DataProtocol]],
*,
observer: ExecutionObserverProtocol | None = None,
error_policy: Literal['continue', 'fail_fast'] = 'continue',
run_id: str | None = None,
load_only: bool = False,
) -> None

Streaming async execution for FunctionJobNode.

When a database is attached, uses two-phase execution: replay cached results first, then compute missing data concurrently. Otherwise, routes each data through async_process_data directly.

In CACHE_ONLY mode the upstream is unavailable; all cached results are served directly from persistent storage without touching the input channel.

Afterwards the node's rows are those of this run's inputs, as after execute, although every stored result was loaded into its cache. If the run raises or is cancelled, it defines no rows: the node drops its in-memory rows and reads as a node with no run in this process.

Parameters:

  • input_channel (ReadableChannel[tuple[TagProtocol, DataProtocol]]): Single readable channel of (tag, data) pairs.
  • output (WritableChannel[tuple[TagProtocol, DataProtocol]]): Writable channel for output (tag, data) pairs.
  • observer (ExecutionObserverProtocol | None) default None: Optional execution observer for hooks.
  • error_policy (Literal['continue', 'fail_fast']) default 'continue': "continue" skips rows whose function raised; "fail_fast" raises the exception of the first row whose function raised, as execute does, and cancels the rows still in flight. Rows run concurrently, so that is the first row to fail, not necessarily the first in input order.
  • run_id (str | None) default None: Optional pipeline run identifier threaded through to InvocationContext.pipeline_run_id.
  • load_only (bool) default False: As for execute: fill the node with the stored results of its input rows and compute nothing (ITL-1095).

Raises:

  • Exception: Under "fail_fast", the exception of the first row whose function raised, inside the ExceptionGroup of the node's task group. output is closed first.
  • RuntimeError: If the node was loaded by PipelineJob.load: without a store it has nothing to serve, and it has no function pod to compute with. output is closed first.

class · orcapod.core.nodes.operator_node.OperatorJobNode

class OperatorJobNode(
operator: OperatorPodProtocol,
input_streams: tuple[StreamProtocol, ...] | list[StreamProtocol],
tracker_manager: TrackerManagerProtocol | None = None,
label: str | None = None,
config: OrcapodConfig | None = None,
pipeline_database: ArrowDatabaseProtocol | None = None,
cache_mode: CacheMode = CacheMode.OFF,
table_scope: Literal['pipeline_hash', 'content_hash'] = 'pipeline_hash',
)

Bases: RunHoldingNode, OperatorNodeBase

DB-backed execution node for operator pod invocations.

Stream node representing an operator invocation with optional DB persistence.

When constructed without database parameters, provides the core stream interface (identity, schema, iteration) without any persistence. When databases are provided (either at construction or via attach_databases), adds pipeline record storage with per-row deduplication, get_all_records() for retrieving stored results, as_source() for creating a DerivedSource from DB records, and three-tier cache mode (OFF / LOG / REPLAY).

Node identity path structure::

operator.uri / schema:{pipeline_hash} / instance:{content_hash}

Where pipeline_hash encodes the pipeline structure (operator + upstream topology) and instance:{content_hash} is the data-inclusive hash that encodes upstream source identities, ensuring each unique source combination gets its own cache table.

method · orcapod.core.nodes.operator_node.OperatorJobNode.as_node

def as_node() -> OperatorNode

Return the lightweight OperatorNode equivalent of this job node.

Returns:

  • OperatorNode: A new OperatorNode with the same operator, input streams,
  • OperatorNode: label, table scope, and tracker manager. Its content_hash() /
  • OperatorNode: pipeline_hash() are identical to those of this OperatorJobNode.

method · orcapod.core.nodes.operator_node.OperatorJobNode.set_ephemeral_database

def set_ephemeral_database(database: 'ArrowDatabaseProtocol | None') -> None

No-op — full ephemeral support for operators is deferred to ITL-509.

method · orcapod.core.nodes.operator_node.OperatorJobNode.clear_cache

def clear_cache() -> None

Clear output caches and node identity path cache.

The node then holds no run: it is stale and reads empty, as before its first run in this process (ITL-1095). Rows it stored stay stored (as_source()).

method · orcapod.core.nodes.operator_node.OperatorJobNode.require_storable_tables

def require_storable_tables(
pipeline_database: ArrowDatabaseProtocol,
cache_mode: CacheMode = CacheMode.OFF,
) -> None

Refuse a database that cannot store this node's cache (ITL-1162).

With caching on, checks the cache's own columns (__node_content_hash, __data_context); the operator's output columns are checked when they are written. Nothing is computed and nothing changes.

Parameters:

  • pipeline_database (ArrowDatabaseProtocol): The database for the cache.
  • cache_mode (CacheMode) default CacheMode.OFF: The caching behaviour; OFF stores nothing.

Raises:

  • UnstorableColumnError: If the database cannot store a column, naming this node.

method · orcapod.core.nodes.operator_node.OperatorJobNode.attach_databases

def attach_databases(
pipeline_database: ArrowDatabaseProtocol,
cache_mode: CacheMode = CacheMode.OFF,
) -> None

Attach a database for persistent caching and pipeline records.

Parameters:

  • pipeline_database (ArrowDatabaseProtocol): Database for pipeline records.
  • cache_mode (CacheMode) default CacheMode.OFF: Caching behaviour (OFF, LOG, or REPLAY).

Raises:

  • InputValidationError: If the node's output has data-tags, the cache mode stores or replays them (LOG or REPLAY), and the database cannot keep the marker that states them (ITL-1019). Nothing is attached or written.

classmethod · orcapod.core.nodes.operator_node.OperatorJobNode.from_descriptor

def from_descriptor(
descriptor: dict[str, Any],
operator: OperatorPodProtocol | None,
input_streams: tuple[StreamProtocol, ...] | list[StreamProtocol],
databases: dict[str, Any],
job_content_hash: str | None = None,
expected_row_count: int | None = None,
) -> OperatorJobNode

Construct an OperatorJobNode from a serialized descriptor.

When operator and input_streams are provided the node operates in full mode — constructed normally via __init__. When operator is None the node is created in read-only mode with metadata from the descriptor; computation methods will raise RuntimeError.

Parameters:

  • descriptor (dict[str, Any]): The serialized node descriptor dict.
  • operator (OperatorPodProtocol | None): An optional live operator instance. None for read-only mode.
  • input_streams (tuple[StreamProtocol, ...] | list[StreamProtocol]): Input streams for the operator. Empty tuple for read-only mode.
  • databases (dict[str, Any]): Mapping of database role names ("pipeline") to database instances.
  • job_content_hash (str | None) default None: The content hash of the node in the saved job's run, which its stored rows carry; None when the file does not record it.
  • expected_row_count (int | None) default None: How many rows the database held under the run's key when the job was saved, from the job file's optional run.expected_rows; None when not recorded. Advisory: a read returns what the database holds and only warns (OrcapodDatabaseContentWarning) when it differs.

Returns:

method · orcapod.core.nodes.operator_node.OperatorJobNode.declared_data_tags

def declared_data_tags() -> tuple[str, ...]

Return the data-tags of this node's output, as the node declares them.

A node built in this process declares its operator's predicted output: the names in both halves of keys(). A node loaded by PipelineJob.load declares its saved descriptor's output schema, the saved job being an independent point of verification of the stored table (developer, November 16, 2026). These are the tags the node's stored table must mark.

Returns:

  • tuple[str, ...]: The data-tags, in tag order; empty for a plain output.

method · orcapod.core.nodes.operator_node.OperatorJobNode.get_cached_output

def get_cached_output() -> StreamProtocol | None

Return cached output stream in REPLAY mode, else None.

Returns:

  • StreamProtocol | None: The cached stream if REPLAY mode and DB records exist,
  • StreamProtocol | None: otherwise None.

method · orcapod.core.nodes.operator_node.OperatorJobNode.execute

def execute(
*input_streams: StreamProtocol,
observer: ExecutionObserverProtocol | None = None,
load_only: bool = False,
) -> list[tuple[TagProtocol, DataProtocol]]

Execute input streams: compute, persist, and cache.

Parameters:

  • *input_streams (StreamProtocol) default (): Input streams to execute.
  • observer (ExecutionObserverProtocol | None) default None: Optional execution observer for hooks.
  • load_only (bool) default False: A load-only run (ITL-1095): the operator still runs on the rows that reach it (it is not a stored computation), but nothing is written: a LOG node does not store its output. REPLAY replays as in any run.

Returns:

  • list[tuple[TagProtocol, DataProtocol]]: Materialized list of (tag, data) pairs.

method · orcapod.core.nodes.operator_node.OperatorJobNode.run

def run(*, load_only: bool = False) -> StreamView

Run the node as a job run would, and return a StreamView of its rows.

State-driven (developer, November 13, 2026; ITL-1095): a node that holds the output of a completed run and is not stale is not run again; the view shows it. Otherwise the node runs, after every job node it depends on that holds no current run (each once, in dependency order, as job.run() would run them; last_run reports the counts), and the run's writes are committed. A node built over streams outside a job runs its operator on them directly.

Parameters:

  • load_only (bool) default False: As job.run(load_only=True): the operator runs on the rows that reach it, a LOG node writes nothing, and the nodes run first load their stored results only.

Returns:

  • StreamView: A StreamView of every row as_table() then returns, naming its rows
  • StreamView: as head() then names them.

Raises:

  • RuntimeError: If the node was loaded by PipelineJob.load and load_only is false.

method · orcapod.core.nodes.operator_node.OperatorJobNode.iter_data

def iter_data() -> Iterator[tuple[TagProtocol, DataProtocol]]

Return an iterator over the (tag, data) rows of the node's current run.

Read-only: never computes and never reads the database. Yields the rows of the latest run(), execute() or async_execute(); nothing before a run in this process, after clear_cache() or once the node is stale, in every cache mode (ITL-1095). as_source() reads what the node stored (LOG).

method · orcapod.core.nodes.operator_node.OperatorJobNode.as_table

def as_table(
*,
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> 'pa.Table'

Return the output of the node's current run as a PyArrow table.

Read-only: never computes and never reads the database. Returns the rows of the latest run(), execute() or async_execute(); before a run in this process, after clear_cache() or once the node is stale, a zero-row table with the node's columns, in every cache mode (ITL-1095).

That zero-row table has the columns, Arrow types and nullability of the node's rows, system tags and source info included: it is the operator's result over its inputs' zero-row tables, got without computing or reading a row (ITL-912). Known limitation: when an input cannot say its Arrow types without rows (a PollingSource before its first poll, an unbound input slot or SourceProxy, a DerivedSource, or a loaded node with no live upstream), the table is built from the node's Python output schema: an int8 column reads as int64 and the system-tag columns are absent (ITL-1089, ITL-985).

method · orcapod.core.nodes.operator_node.OperatorJobNode.head

def head(
n: int = 10,
*,
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> StreamView

Return the first rows of the node's current run; never runs the operator.

After a run (or execute) the rows are the node's output in memory. Before a run in this process the node shows no rows and says so, pointing to run() and to as_source(), in every cache mode; once stale it says its inputs changed (ITL-1095). The database is never read. See StreamBase.head.

method · orcapod.core.nodes.operator_node.OperatorJobNode.get_all_records

def get_all_records(
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> 'pa.Table | None'

Retrieve all stored records from the pipeline database.

Returns the stored output table with column filtering applied per ColumnConfig conventions. Returns None when no database is attached or no matching records exist. Switching to OFF does not remove records stored by an earlier LOG run.

This is the raw records view: a data-tag column keeps the marker the stored table states it with (ITL-1019), checked against the node on read. as_source() reads the binding from it.

Raises:

  • InputValidationError: If the stored table and the node disagree on which tags are data-tags.

method · orcapod.core.nodes.operator_node.OperatorJobNode.as_source

def as_source(tags_as_data: 'Mapping[str, str] | None' = None)

Return a DerivedSource backed by the DB records of this node.

Its source_id is the DerivedSource default: without tags_as_data, derived:<algorithm label>:<hex> of this node's content hash.

Parameters:

  • tags_as_data ('Mapping[str, str] | None') default None: Maps a tag column of this node to the name of a data column that receives a copy of it. When given, the result is a new root source with its own provenance and a source_id derived from this node's content hash and the mapping; see DerivedSource.

Raises:

  • RuntimeError: If no database is attached.
  • InputValidationError: If tags_as_data is not valid for this node's columns.

method · orcapod.core.nodes.operator_node.OperatorJobNode.async_execute

async def async_execute(
inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]],
output: WritableChannel[tuple[TagProtocol, DataProtocol]],
*,
observer: 'ExecutionObserverProtocol | None' = None,
load_only: bool = False,
) -> None

Async execution with cache mode handling when DB is attached.

Every mode except REPLAY runs the wrapped operator's async_execute, forwards its rows downstream and keeps them on the node, exactly as execute() does, so iter_data() and as_table() return them after the run whichever orchestrator ran the node.

Without a database, or with one in OFF mode, nothing is stored.

Parameters:

  • inputs (Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]]): Sequence of readable channels from upstream nodes.
  • output (WritableChannel[tuple[TagProtocol, DataProtocol]]): Writable channel for output (tag, data) pairs.
  • observer ('ExecutionObserverProtocol | None') default None: Optional execution observer for hooks.
  • load_only (bool) default False: As for execute: a LOG node stores nothing (ITL-1095).

class · orcapod.core.nodes.source_node.SourceJobNode

class SourceJobNode(
name: str,
tag_schema: Schema | None = None,
data_schema: Schema | None = None,
bound_source: StreamProtocol | None = None,
data_context: str | contexts.DataContext | None = None,
)

Bases: SourceNodeBase

Execution-ready source node wrapping an optional concrete stream.

Used inside PipelineJob._persistent_node_map. The _concrete field is mutable — PipelineJob.bind(sources={...}) updates it in-place so that downstream FunctionJobNode objects (which hold a reference to this same object) automatically see the new concrete source without cascading reference updates.

Hash behaviour:

  • content_hash() — delegates to the bound source's identity_structure() when bound (via identity_structure() override); falls back to schema-based identity when unbound.
  • pipeline_hash() — always schema-based (inherited); never data-inclusive. This invariant keeps DB paths stable across different data sources bound to the same slot.

Parameters:

  • name (str): Slot name.
  • tag_schema (Schema | None) default None: Tag schema. Optional when bound_source is provided; inferred from bound_source.output_schema() when omitted. Stored for future rebind validation.
  • data_schema (Schema | None) default None: Data schema. Optional when bound_source is provided; inferred from bound_source.output_schema() when omitted. Stored for future rebind validation.
  • bound_source (StreamProtocol | None) default None: Optional concrete stream. Can be set or replaced later via job_node.bound_source = source.
  • data_context (str | contexts.DataContext | None) default None: Optional data context override.

Raises:

  • ValueError: If both tag_schema and data_schema are omitted and bound_source is None.

method · orcapod.core.nodes.source_node.SourceJobNode.run

def run(*, load_only: bool = False) -> StreamView

Read the bound source's rows and return a StreamView of all of them.

As every job node's run(), it first checks the bound source for new data (check_sources, ITL-1102): a check that raises warns (SourceCheckFailedWarning) and the source's held rows are read. A source is read, never computed, so load_only changes nothing, except that a load-only run refuses a bound stream that computes or delivers when read (ITL-1121); a job node's run() takes it (ITL-1095).

Parameters:

  • load_only (bool) default False: Accepted for every job node; a source reads its rows either way.

Raises:

  • UnboundSourceError: When no concrete source is bound.
  • ValueError: If load_only and the bound stream computes or delivers when read (node_run.refuse_unloadable_inputs).

property · orcapod.core.nodes.source_node.SourceJobNode.bound_source

Type: StreamProtocol | None

The concrete stream currently bound to this slot, or None.

property · orcapod.core.nodes.source_node.SourceJobNode.last_modified

Type: datetime | None

When the slot's content last changed: its binding or its bound source.

The later of the time the slot was bound and the bound source's own last_modified (a CachedSource or PollingSource moves it when it reads new rows). A bound source that does not know its modification time makes the slot's unknown too (None), so every node below it is stale. Every run first checks the bound source for new data (check_for_new_data, ITL-1102), which moves its modified time when it found some. A source that cannot see its data change (one built over data held in memory) is not re-read: build a new source object and bind it, or clear the dependent nodes.

method · orcapod.core.nodes.source_node.SourceJobNode.identity_structure

def identity_structure() -> Any

Delegate to bound_source.identity_structure() when bound.

This is the correct extension point: content_hash() flows from identity_structure(), so overriding here avoids bypassing the caching and resolver logic in ContentIdentifiableBase.content_hash().

Returns:

  • Any: Bound source's identity structure when bound; schema-based
  • Any: identity (inherited from SourceNodeBase) when unbound.

method · orcapod.core.nodes.source_node.SourceJobNode.iter_data

def iter_data() -> Iterator[tuple[TagProtocol, DataProtocol]]

Delegate to concrete source, or raise if unbound.

Raises:

  • UnboundSourceError: When no concrete source is attached.

method · orcapod.core.nodes.source_node.SourceJobNode.head

def head(
n: int = 10,
*,
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> StreamView

Return the bound source's head, or no rows when no source is bound.

A narrower bound source's rows are shown as rows of the declared schema.

method · orcapod.core.nodes.source_node.SourceJobNode.async_iter_data

async def async_iter_data() -> AsyncIterator[tuple[TagProtocol, DataProtocol]]

Delegate to bound_source.async_iter_data() when bound.

Overrides SourceNodeBase.async_iter_data() to route through the bound source's own async generator instead of wrapping iter_data() synchronously. This ensures that dynamic sources such as PollingSource run their async polling loop rather than returning a static snapshot.

Raises:

  • UnboundSourceError: When no concrete source is attached.

method · orcapod.core.nodes.source_node.SourceJobNode.output_schema

def output_schema(
*,
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> tuple[Schema, Schema]

Return (tag_schema, data_schema), delegating to bound_source when set.

When bound_source has the declared schema, this is a transparent pass-through to bound_source.output_schema(columns=columns, all_info=all_info) so callers get the same result as querying the source directly. A narrower bound source (x: int for a declared x: int | None) is given as the declared schema (ITL-1112).

When unbound, delegates to SourceNodeBase.output_schema() which includes system-tag schema entries derived from the declared schemas.

Parameters:

  • columns (ColumnConfig | dict[str, Any] | None) default None: Column selection config.
  • all_info (bool) default False: If True, include all available column groups.

Returns:

  • tuple[Schema, Schema]: Tuple of (tag_schema, data_schema).

method · orcapod.core.nodes.source_node.SourceJobNode.as_table

def as_table(
*,
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> pa.Table

Materialize the concrete source as a PyArrow Table.

A narrower bound source's rows are given as rows of the declared schema (ITL-1112).

Parameters:

  • columns (ColumnConfig | dict[str, Any] | None) default None: Column selection config.
  • all_info (bool) default False: If True, include all metadata columns.

Raises:

  • UnboundSourceError: When no concrete source is attached.

classmethod · orcapod.core.nodes.source_node.SourceJobNode.from_stream

def from_stream(stream: StreamProtocol, name: str | None = None) -> 'SourceJobNode'

Create a SourceJobNode from stream using three-way logic.

The three cases are:

  1. stream is already a SourceJobNode — copy it, preserving the existing bound_source (the SJN itself is not used as the bound source).
  2. stream is a SourceNode (schema-only, no data) — create an unbound SourceJobNode with the same name and schemas. The resulting SJN has bound_source=None and the same content_hash() as stream (schema-based).
  3. Any other concrete stream (ArrowTableSource, etc.) — create a bound SourceJobNode whose content_hash() delegates to the concrete stream.

Used by Pipeline.compile for each leaf of the pipeline. The stream is bound as any source is, so a leaf without exactly one source-id and one record-id system-tag column (a raw ArrowTableStream) is refused (ITL-1126).

Parameters:

  • stream (StreamProtocol): The stream to wrap.
  • name (str | None) default None: Optional explicit slot name. In all three cases, when name is provided it takes precedence over the stream's own name. For concrete streams (Case 3), when name is None the slot name is resolved as: stream.source_id for RootSource instances, or "{stream.label}:{hash_prefix}" for any other concrete stream type.

Returns:

  • 'SourceJobNode': A SourceJobNode configured according to the case above.

method · orcapod.core.nodes.source_node.SourceJobNode.as_node

def as_node() -> SourceNode

Return the lightweight SourceNode equivalent of this job node.

Returns:

  • SourceNode: A new SourceNode with the same name and schemas.