Nodes
Nodes are the elements of a compiled pipeline graph. The plain node classes describe the
graph of a Pipeline blueprint; the job node classes below add database-backed execution.
FunctionNode
Section titled “FunctionNode”class · orcapod.core.nodes.function_node.FunctionNode
class FunctionNode( function_pod: FunctionPodProtocol, input_stream: StreamProtocol, tracker_manager: TrackerManagerProtocol | None = None, label: str | None = None, config: OrcapodConfig | None = None, table_scope: Literal['pipeline_hash', 'content_hash'] = 'pipeline_hash',)Bases: FunctionNodeBase
Thin blueprint descriptor for a function pod invocation.
Carries no database references. Calling iter_data() raises
PipelineJobRequiredError — wrap the containing Pipeline in a
PipelineJob to obtain an executable FunctionJobNode.
This is the node type recorded inside a Pipeline context manager
and serialized to disk via Pipeline.save().
set_ephemeral_database
Section titled “set_ephemeral_database”method · orcapod.core.nodes.function_node.FunctionNode.set_ephemeral_database
def set_ephemeral_database(database: 'ArrowDatabaseProtocol | None') -> NoneNo-op — blueprint nodes carry no database references.
from_descriptor
Section titled “from_descriptor”classmethod · orcapod.core.nodes.function_node.FunctionNode.from_descriptor
def from_descriptor( descriptor: dict[str, Any], function_pod: FunctionPodProtocol | None, input_stream: StreamProtocol | None, databases: dict[str, Any],) -> 'FunctionNode'Construct a FunctionNode from a serialized descriptor.
When function_pod and input_stream are both provided the node
operates in full mode -- constructed normally via __init__.
When function_pod is None the node is created in read-only
mode with metadata from the descriptor; computation methods will
raise PipelineJobRequiredError.
Parameters:
- descriptor (
dict[str, Any]): The serialized node descriptor dict. - function_pod (
FunctionPodProtocol | None): An optional live function pod.Nonefor read-only mode. - input_stream (
StreamProtocol | None): An optional live input stream.Nonefor read-only mode. - databases (
dict[str, Any]): Mapping of database role names ("pipeline","result") to database instances.
Returns:
'FunctionNode': A newFunctionNodeinstance.
iter_data
Section titled “iter_data”method · orcapod.core.nodes.function_node.FunctionNode.iter_data
def iter_data() -> Iterator[tuple[TagProtocol, DataProtocol]]Raise PipelineJobRequiredError — blueprint nodes cannot produce data.
Raises:
PipelineJobRequiredError: Always.
as_node
Section titled “as_node”method · orcapod.core.nodes.function_node.FunctionNode.as_node
def as_node() -> FunctionNodeReturn the lightweight blueprint equivalent of this node.
Returns a fresh FunctionNode with the same function pod, input
stream, label, table scope, and tracker manager. Its
content_hash() / pipeline_hash() are identical to those of
this node.
Returns:
FunctionNode: A new equivalentFunctionNode.
Raises:
RuntimeError: If this node is in UNAVAILABLE state (no live function pod). UNAVAILABLE nodes cannot be cloned into a usable blueprint — callers must handle this status before invokingas_node().
OperatorNode
Section titled “OperatorNode”class · orcapod.core.nodes.operator_node.OperatorNode
class OperatorNode( operator: OperatorPodProtocol, input_streams: Collection[StreamProtocol], tracker_manager: TrackerManagerProtocol | None = None, label: str | None = None, config: OrcapodConfig | None = None, table_scope: Literal['pipeline_hash', 'content_hash'] = 'pipeline_hash',)Bases: OperatorNodeBase
Thin blueprint descriptor for an operator pod invocation.
Carries no database references. Calling iter_data() raises
PipelineJobRequiredError — wrap the containing Pipeline in a
PipelineJob to obtain an executable OperatorJobNode.
This is the node type recorded inside a Pipeline context manager
and serialized to disk via Pipeline.save().
iter_data
Section titled “iter_data”method · orcapod.core.nodes.operator_node.OperatorNode.iter_data
def iter_data() -> Iterator[tuple[TagProtocol, DataProtocol]]Raise PipelineJobRequiredError — blueprint nodes cannot produce data.
Raises:
PipelineJobRequiredError: Always.
method · orcapod.core.nodes.operator_node.OperatorNode.head
def head( n: int = 10, *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False,) -> StreamViewReturn no rows: a blueprint node holds no data.
as_table
Section titled “as_table”method · orcapod.core.nodes.operator_node.OperatorNode.as_table
def as_table( *, columns: ColumnConfig | dict[Any, Any] | None = None, all_info: bool = False,) -> pa.TableRaise PipelineJobRequiredError — blueprint nodes cannot produce data.
Raises:
PipelineJobRequiredError: Always.
from_descriptor
Section titled “from_descriptor”classmethod · orcapod.core.nodes.operator_node.OperatorNode.from_descriptor
def from_descriptor( descriptor: dict[str, Any], operator: OperatorPodProtocol | None, input_streams: tuple[StreamProtocol, ...] | list[StreamProtocol], databases: dict[str, Any],) -> 'OperatorNode'Construct an OperatorNode from a serialized descriptor.
When operator and input_streams are provided the node operates
in full mode — constructed normally via __init__. When
operator is None the node is created in read-only mode with
metadata from the descriptor; computation methods will raise
PipelineJobRequiredError.
Parameters:
- descriptor (
dict[str, Any]): The serialized node descriptor dict. - operator (
OperatorPodProtocol | None): An optional live operator instance.Nonefor read-only mode. - input_streams (
tuple[StreamProtocol, ...] | list[StreamProtocol]): Input streams for the operator. Empty tuple for read-only mode. - databases (
dict[str, Any]): Unused — kept for API parity withOperatorJobNode.from_descriptor.
Returns:
'OperatorNode': A newOperatorNodeinstance.
Raises:
ValueError: Iftable_scopeis missing or invalid.
set_ephemeral_database
Section titled “set_ephemeral_database”method · orcapod.core.nodes.operator_node.OperatorNode.set_ephemeral_database
def set_ephemeral_database(database: 'ArrowDatabaseProtocol | None') -> NoneNo-op — blueprint operator nodes carry no database references.
as_node
Section titled “as_node”method · orcapod.core.nodes.operator_node.OperatorNode.as_node
def as_node() -> OperatorNodeReturn the lightweight blueprint equivalent of this node.
Returns a fresh OperatorNode with the same operator, input streams,
label, table scope, and tracker manager. Its content_hash() /
pipeline_hash() are identical to those of this node.
Returns:
OperatorNode: A new equivalentOperatorNode.
Raises:
RuntimeError: If this node is in UNAVAILABLE state (no live operator). UNAVAILABLE nodes cannot be cloned into a usable blueprint — callers must handle this status before invokingas_node().
SourceNode
Section titled “SourceNode”class · orcapod.core.nodes.source_node.SourceNode
class SourceNode( name: str, tag_schema: Schema, data_schema: Schema, data_context: str | contexts.DataContext | None = None, label: str | None = None, config: OrcapodConfig | None = None,)Bases: SourceNodeBase
Schema-only input-slot declaration for Pipeline recording.
Replaces SourceSpec as the user-facing way to declare typed pipeline
inputs. Pass a SourceNode inside a with pipeline: block as the
upstream for any pod invocation.
Example::
slot = SourceNode(name="data", tag_schema={"id": int}, data_schema={"v": float})with pipeline: result = my_pod(slot)
job = PipelineJob.from_pipeline(pipeline, database=db, sources={"data": my_source})job.run()iter_data
Section titled “iter_data”method · orcapod.core.nodes.source_node.SourceNode.iter_data
def iter_data() -> Iterator[tuple[TagProtocol, DataProtocol]]Raise UnboundSourceError — SourceNode carries no data.
Raises:
UnboundSourceError: Always.
from_stream
Section titled “from_stream”classmethod · orcapod.core.nodes.source_node.SourceNode.from_stream
def from_stream(stream: StreamProtocol, name: str | None = None) -> SourceNodeWrap stream in a SourceNode, or return it unchanged if already one.
Derives tag and data schemas from stream.output_schema(). The slot
name is resolved in priority order:
- name parameter (explicit override — highest priority).
stream.source_idwhen stream is aRootSource— the canonical source identity, stable across pipeline serialisation / deserialisation."{stream.label}:{hash_prefix}"for any other concrete stream type, combining the cosmetic label with a short content-hash suffix to ensure uniqueness.
Parameters:
- stream (
StreamProtocol): The upstream stream to wrap. - name (
str | None) defaultNone: Optional explicit slot name.
Returns:
SourceNode: The original stream if it is already aSourceNode; otherwiseSourceNode: a newSourceNodewith schemas derived from stream.
Job nodes
Section titled “Job nodes”The database-backed nodes a PipelineJob builds; these are what job.nodes[label] returns.
FunctionJobNode
Section titled “FunctionJobNode”class · orcapod.core.nodes.function_node.FunctionJobNode
class FunctionJobNode( function_pod: FunctionPodProtocol, input_stream: StreamProtocol, tracker_manager: TrackerManagerProtocol | None = None, label: str | None = None, config: OrcapodConfig | None = None, pipeline_database: ArrowDatabaseProtocol | None = None, result_database: ArrowDatabaseProtocol | None = None, ephemeral_database: ArrowDatabaseProtocol | None = None, table_scope: Literal['pipeline_hash', 'content_hash'] = 'pipeline_hash',)Bases: RunHoldingNode, FunctionNodeBase
DB-backed execution node for function pod invocations.
Created by PipelineJob at run time; never recorded inside a plain
Pipeline. Carries all persistence logic: CachedFunctionPod
wrapping, pipeline records, and two-phase iter_data() / async
execution.
node_config
Section titled “node_config”property · orcapod.core.nodes.function_node.FunctionJobNode.node_config
Type: NodeConfig
Per-node pipeline execution configuration.
require_storable_tables
Section titled “require_storable_tables”method · orcapod.core.nodes.function_node.FunctionJobNode.require_storable_tables
def require_storable_tables( pipeline_database: ArrowDatabaseProtocol, result_database: ArrowDatabaseProtocol | None = None,) -> NoneRefuse databases that cannot store this node's tables (ITL-1162).
Checks the result table (CachedFunctionPod.require_storable_results)
against result_database and the pipeline records' own columns against
pipeline_database. Nothing is computed and nothing changes. A node
loaded without its function stores nothing and is not checked.
Parameters:
- pipeline_database (
ArrowDatabaseProtocol): The database for pipeline records. - result_database (
ArrowDatabaseProtocol | None) defaultNone: The database for results; defaults topipeline_database.at("_result"), as inattach_databases.
Raises:
InputValidationError: If the node passes data-tags through andpipeline_databasecannot keep their marker (ITL-1019).UnstorableColumnError: If a database cannot store a column, naming this node.
attach_databases
Section titled “attach_databases”method · orcapod.core.nodes.function_node.FunctionJobNode.attach_databases
def attach_databases( pipeline_database: ArrowDatabaseProtocol | None = None, result_database: ArrowDatabaseProtocol | None = None, ephemeral_database: ArrowDatabaseProtocol | None = None,) -> NoneAttach databases for persistent caching and pipeline records.
For live nodes (_function_pod is set), creates a
CachedFunctionPod wrapping the original function pod for result
caching. For read-only stubs (_function_pod is None, e.g.
nodes loaded from a saved job), creates a _ResultDatabaseReader
using the record path stored in the descriptor so that
get_all_records() and _load_cached_entries() still work.
When pipeline_database is None, the persistent-DB portion of
setup is skipped entirely — no CachedFunctionPod is created and
_pipeline_database is left as None. Only
ephemeral_database wiring proceeds in that case.
The databases are expected to be pre-scoped by the pipeline (via
db.at(*pipeline_name).at("_result") etc.) so no additional path
prefix is needed here.
Parameters:
- pipeline_database (
ArrowDatabaseProtocol | None) defaultNone: Database for pipeline records. PassNoneto skip persistent-DB setup (ephemeral-only or deferred wiring). - result_database (
ArrowDatabaseProtocol | None) defaultNone: Database for cached results. Defaults topipeline_database.at("_result")whenpipeline_databaseis notNone. - ephemeral_database (
ArrowDatabaseProtocol | None) defaultNone: Optional ephemeral database to attach immediately viaset_ephemeral_database(). Equivalent to callingset_ephemeral_database(ephemeral_database)afterattach_databases().
from_descriptor
Section titled “from_descriptor”classmethod · orcapod.core.nodes.function_node.FunctionJobNode.from_descriptor
def from_descriptor( descriptor: dict[str, Any], input_stream: StreamProtocol | None = None, databases: dict[str, Any] | None = None, job_content_hash: str | None = None, expected_row_count: int | None = None,) -> 'FunctionJobNode'Create a read-only FunctionJobNode stub from a serialized descriptor.
Used by PipelineJob.load() to populate _persistent_node_map
with database-backed stubs for function nodes. The stubs support
get_all_records() and _load_cached_entries() on previously
computed results without needing a live function_pod.
Computation-path methods (execute(), iter_data()) will
fail for stub nodes because _function_pod is None.
Parameters:
- descriptor (
dict[str, Any]): Serialized node descriptor dict (as produced byPipeline.save()). - input_stream (
StreamProtocol | None) defaultNone: Optional upstream stream. May beNonefor read-only stubs where execution is not needed. - databases (
dict[str, Any] | None) defaultNone: Optional mapping{"pipeline": db, "result": db}. When"pipeline"is present,attach_databases()is called immediately so the stub is ready for DB queries. - job_content_hash (
str | None) defaultNone: Optional live (data-inclusive) content hash from the run that produced the records being loaded. When provided, overrides the blueprint hash stored in descriptor so thatcontent_hash()matches the_node_content_hashcolumn written to the DB at execution time. Required forget_all_records()to correctly filter DB rows whentable_scope="pipeline_hash"(the default). - expected_row_count (
int | None) defaultNone: How many pipeline records of this node (with its run's content hash) the database held when the job was saved, from the job file's optionalrun.expected_rows;Nonewhen not recorded. Advisory: a read always returns what the database holds and only warns (OrcapodDatabaseContentWarning) when the database holds another number.
Returns:
'FunctionJobNode': AFunctionJobNodein read-only stub mode'FunctionJobNode': (load_status=UNAVAILABLE).
clear_cache
Section titled “clear_cache”method · orcapod.core.nodes.function_node.FunctionJobNode.clear_cache
def clear_cache() -> NoneClear in-memory output caches, content hash cache, and node identity path cache.
The node then holds no run: it is stale and reads empty, as before its first run
in this process (ITL-1095). Its stored results stay stored (as_source()).
last_run_not_stored
Section titled “last_run_not_stored”property · orcapod.core.nodes.function_node.FunctionJobNode.last_run_not_stored
Type: int
Inputs the node's last completed run skipped because nothing was stored.
Non-zero only after a load-only run (run(load_only=True)); 0 before a run.
set_ephemeral_database
Section titled “set_ephemeral_database”method · orcapod.core.nodes.function_node.FunctionJobNode.set_ephemeral_database
def set_ephemeral_database(database: 'ArrowDatabaseProtocol | None') -> NoneAssign or remove the ephemeral result database.
When database is not None, creates a CachedFunctionPod backed by
database so that ephemeral writes use the same format as persistent writes.
When database is None, clears both the database and the ephemeral pod.
Parameters:
- database (
'ArrowDatabaseProtocol | None'): TheArrowDatabaseProtocolto use for ephemeral result storage, orNoneto detach and revert to persistent-only writes.
as_node
Section titled “as_node”method · orcapod.core.nodes.function_node.FunctionJobNode.as_node
def as_node() -> FunctionNodeReturn the lightweight FunctionNode equivalent of this job node.
Returns:
FunctionNode: A newFunctionNodewith the same function pod, input stream,FunctionNode: label, table scope, and tracker manager. Itscontent_hash()/FunctionNode:pipeline_hash()are identical to those of thisFunctionJobNode.
execute_data
Section titled “execute_data”method · orcapod.core.nodes.function_node.FunctionJobNode.execute_data
def execute_data( tag: TagProtocol, data: DataProtocol,) -> tuple[TagProtocol, DataProtocol | None]Execute a single data: compute, persist, and cache.
Internal method for orchestrators. The caller must guarantee that
the tag and data conform to the expected input schema (matching
self._input_stream). No validation is performed.
Parameters:
- tag (
TagProtocol): The tag associated with the data. - data (
DataProtocol): The input data to process.
Returns:
tuple[TagProtocol, DataProtocol | None]: A(tag, output_data)tuple.
execute
Section titled “execute”method · orcapod.core.nodes.function_node.FunctionJobNode.execute
def execute( input_stream: StreamProtocol, *, observer: ExecutionObserverProtocol | None = None, error_policy: Literal['continue', 'fail_fast'] = 'continue', run_id: str | None = None, load_only: bool = False,) -> list[tuple[TagProtocol, DataProtocol]]Execute all data from a stream: compute, persist, and cache.
For each data: fire on_data_start, check the in-memory cache
(populated from DB if needed), compute if missing, fire
on_data_end or on_data_crash.
Afterwards the node's rows (iter_data, as_table, head) are those
of this run's inputs. If the run raises, it defines no rows: the node drops
its in-memory rows and reads as a node with no run in this process; what was
stored stays stored.
Parameters:
- input_stream (
StreamProtocol): The input stream to process. - observer (
ExecutionObserverProtocol | None) defaultNone: Optional execution observer for hooks. - error_policy (
Literal['continue', 'fail_fast']) default'continue':"continue"skips failed data;"fail_fast"re-raises on the first failure. - run_id (
str | None) defaultNone: Optional pipeline run identifier threaded through toInvocationContext.pipeline_run_id. - load_only (
bool) defaultFalse: Fill the node with the results already stored for the stream's rows and compute nothing (ITL-1095): no function call, no write; inputs without a stored result are skipped and counted (last_run_not_stored). A loaded node (no function pod) can run this way.
Returns:
list[tuple[TagProtocol, DataProtocol]]: Materialized list of (tag, output_data) pairs, excludinglist[tuple[TagProtocol, DataProtocol]]:Noneoutputs and failed data.
Raises:
RuntimeError: If the node was loaded byPipelineJob.loadand so has no function pod to compute with.InputValidationError: If the input stream is of another data context than this node; raised before any row is processed, whatever the error policy.
get_cached_results
Section titled “get_cached_results”method · orcapod.core.nodes.function_node.FunctionJobNode.get_cached_results
def get_cached_results( base_entry_ids: list[bytes],) -> dict[bytes, tuple[TagProtocol, DataProtocol]]Public cache façade: return already-computed results for the given base entry IDs.
Serves hits directly from the in-memory cache (_cached_output_datas).
For IDs not yet cached, delegates to _load_cached_entries which calls
_fetch_joined_records to load from the pipeline and result databases.
Add-only semantics: existing in-memory entries are never cleared or
overwritten.
Does NOT apply user-facing column filtering — see get_all_records
for that.
Parameters:
- base_entry_ids (
list[bytes]): Stable base entry IDs (fromcompute_base_entry_id) to look up.
Returns:
dict[bytes, tuple[TagProtocol, DataProtocol]]: Mapping from base_entry_id to(tag, output_data)for found entries.dict[bytes, tuple[TagProtocol, DataProtocol]]: Empty dict if no DB is attached,base_entry_idsis empty, or nodict[bytes, tuple[TagProtocol, DataProtocol]]: matches are found.
compute_base_entry_id
Section titled “compute_base_entry_id”method · orcapod.core.nodes.function_node.FunctionJobNode.compute_base_entry_id
def compute_base_entry_id(tag: TagProtocol, input_data: DataProtocol) -> bytesComputes the stable (recomputation-index-free) entry ID for a (tag, data) pair.
Hashes the tag's system-tag columns plus INPUT_DATA_HASH_COL (binary).
Because it excludes the recomputation index it is stable across all
recomputation attempts for the same logical input.
The base entry ID is stored in _PIPELINE_BASE_ENTRY_ID_COL and will be
used as the in-memory cache key and Phase 1 filter in subsequent tasks.
Parameters:
- tag (
TagProtocol): The tag datagram for the input row. - input_data (
DataProtocol): The data datagram for the input row.
Returns:
bytes: A bytes value inb"{method}:{digest}"format.
compute_pipeline_entry_id
Section titled “compute_pipeline_entry_id”method · orcapod.core.nodes.function_node.FunctionJobNode.compute_pipeline_entry_id
def compute_pipeline_entry_id( tag: TagProtocol, input_data: DataProtocol, recomputation_index: int = 0,) -> bytesCompute a versioned pipeline entry ID from tag + system tags + input data hash + index.
Extends the base preimage (see _build_entry_id_preimage) with a
_PIPELINE_RECOMPUTATION_INDEX_COL column (value: recomputation_index,
type pa.int32()) so that each recomputation attempt receives a distinct
DB primary key.
At recomputation_index=0 this produces a hash that differs from the
pre-ITL-508 implementation (the index column is now part of the preimage).
Existing pipeline DB records are implicitly invalidated — acceptable
because the project is pre-v0.1.0.
The preimage is system_tags + INPUT_DATA_HASH_COL + recomputation_index.
Parameters:
- tag (
TagProtocol): The tag datagram for the input row. - input_data (
DataProtocol): The data datagram for the input row. - recomputation_index (
int) default0: Position in the recomputation chain.0for the first computation,N+1for each miss-triggered recompute.
Returns:
bytes: Method-prefixed raw bytes (b"{method}:{digest}") uniquelybytes: identifying this (tag, input_data, node run, recomputation attempt).bytes: Suitable for storage in apa.large_binary()column.
add_pipeline_record
Section titled “add_pipeline_record”method · orcapod.core.nodes.function_node.FunctionJobNode.add_pipeline_record
def add_pipeline_record( tag: TagProtocol, input_data: DataProtocol, data_record_id: uuid.UUID, computed: bool, is_ephemeral: bool = False, output_data_hash: ContentHash | None = None,) -> NoneAdd a pipeline record to the database for a processed data.
Computes the next recomputation index by querying existing rows in the
pipeline DB that share the same base_entry_id, then writes at
max_index + 1. The write uses skip_duplicates=True so that
concurrent asyncio coroutines competing for the same versioned entry ID
are safely serialised: the first writer lands, subsequent writers whose
compute happened to produce the same hash are silently no-oped.
The pipeline record stores:
- Tag columns (including system tags)
- All source columns of the input data (provenance, not data values)
- Output data record ID (for joining with result records)
- Base entry ID (
_PIPELINE_BASE_ENTRY_ID_COL) - Recomputation index (
_PIPELINE_RECOMPUTATION_INDEX_COL) - Whether the result is stored in the ephemeral database
- Input data context key
- Whether the result was freshly computed or cached
- Output data content hash (
OUTPUT_DATA_HASH_COL) — used by downstreamEmptyDataflow-through to look up the downstream result cache.
Parameters:
- tag (
TagProtocol): The tag associated with the input data. - input_data (
DataProtocol): The input data that was processed. - data_record_id (
uuid.UUID): UUID of the result record in the result database. - computed (
bool): Whether the result was freshly computed (True) or served from a cache (False). - is_ephemeral (
bool) defaultFalse: Whether the result is stored in the ephemeral database. - output_data_hash (
ContentHash| None) defaultNone: Content hash of the output produced by processinginput_data. When provided, stored asOUTPUT_DATA_HASH_COLso that downstreamEmptyDataflow-through can locate its cached result keyed by the same hash (= the downstream'sINPUT_DATA_HASH_COL).
get_all_records
Section titled “get_all_records”method · orcapod.core.nodes.function_node.FunctionJobNode.get_all_records
def get_all_records( columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False,) -> pa.Table | NonePublic table view: return all computed results joined with their pipeline records.
Calls _fetch_joined_records to obtain the raw joined table, then
applies ColumnConfig-driven column dropping to produce a
user-facing result. NODE_CONTENT_HASH_COL, _PIPELINE_ENTRY_ID_COL,
_PIPELINE_BASE_ENTRY_ID_COL, and _PIPELINE_RECOMPUTATION_INDEX_COL
are always dropped — they are internal discriminator columns, not
user-facing data. These columns are listed explicitly to ensure they are
dropped even when all_info=True (which skips the meta-prefix sweep).
Does NOT populate the in-memory cache — see get_cached_results
for that.
Parameters:
- columns (
ColumnConfig| dict[str, Any] | None) defaultNone: Column configuration controlling which groups are included. Accepts aColumnConfiginstance or a dict shorthand (e.g.{"meta": True}). - all_info (
bool) defaultFalse: IfTrue, equivalent to enabling all column groups.
Returns:
pa.Table | None: Apa.Tableof joined results, orNoneif no database ispa.Table | None: attached, either DB fetch returnsNone, or the join producespa.Table | None: no rows.
Raises:
RuntimeError: If the node was loaded byPipelineJob.loadand no database is attached.
Warns:
OrcapodDatabaseContentWarning: Once per database, if the node was loaded from a job file that records another number of pipeline records for its run than the database holds. The database is the source of truth.
as_source
Section titled “as_source”method · orcapod.core.nodes.function_node.FunctionJobNode.as_source
def as_source(tags_as_data: 'Mapping[str, str] | None' = None)Return a DerivedSource backed by the DB records of this node.
Its source_id is the DerivedSource default: without
tags_as_data, derived:<algorithm label>:<hex> of this node's
content hash.
Parameters:
- tags_as_data (
'Mapping[str, str] | None') defaultNone: Maps a tag column of this node to the name of a data column that receives a copy of it. When given, the result is a new root source with its own provenance and asource_idderived from this node's content hash and the mapping; seeDerivedSource.
Raises:
RuntimeError: If no database is attached.InputValidationError: Iftags_as_datais not valid for this node's columns.
method · orcapod.core.nodes.function_node.FunctionJobNode.head
def head( n: int = 10, *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False,) -> StreamViewReturn the first rows of the node's current run; never runs the function.
After a run (or execute) the rows are the node's results in memory for that
run's inputs, the first of what as_table returns. A node that has not run in
this process, or whose input changed since (stale), shows no rows and says so,
pointing to run() and to as_source() for its stored results (ITL-1095);
the database is not read. The node has all of its rows, so with sort_by_tags
(or all_info) the head is exactly as_table(...).slice(0, n), whatever
order the rows were computed in. See StreamBase.head.
as_table
Section titled “as_table”method · orcapod.core.nodes.function_node.FunctionJobNode.as_table
def as_table( *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False,) -> 'pa.Table'Return the rows of this node's current run as a PyArrow table.
Read-only: never computes and never reads the database. Before a run in this
process, or once the node is stale, the table has zero rows and the node's
columns (ITL-1095); as_source() reads what the database holds.
Raises:
RuntimeError: If the node was loaded and no database is attached.
iter_data
Section titled “iter_data”method · orcapod.core.nodes.function_node.FunctionJobNode.iter_data
def iter_data() -> Iterator[tuple[TagProtocol, DataProtocol]]Yield the (tag, data) rows of this node's current run.
Strictly read-only: never computes and never reads the database. Callers must
call run() (or run the job) first.
After a completed run (execute, async_execute, run() or a job run)
the rows are exactly those of that run's inputs, on either orchestrator, even
though results of other inputs stay cached and stored (ITL-1093). A node that
has not run in this process, whose run raised, or that is stale (its input
changed since its run) yields nothing (ITL-1095); as_source() reads every
row stored for it.
Raises:
RuntimeError: Ifload_statusis UNAVAILABLE.
method · orcapod.core.nodes.function_node.FunctionJobNode.run
def run(*, load_only: bool = False) -> StreamViewRun the node as a job run would, and return a StreamView of its rows.
State-driven (developer, November 13, 2026; ITL-1095): a node that holds the rows
of a completed run and is not stale is not run again; the view shows what it
holds, and nothing is computed, read or written. Otherwise the node runs, after
every job node it depends on that holds no current run (each once, in
dependency order, as job.run() would run them; last_run reports the
counts), and the run's writes are committed. A node built over streams outside a
job reads its input as it is.
Parameters:
- load_only (
bool) defaultFalse: Fill the node, and the nodes it runs first, with what is already stored and compute nothing (asjob.run(load_only=True)). A node filled this way shows statusloaded_onlyand is not current for a later fullrun(), which computes what it skipped.
Returns:
StreamView: AStreamViewof every rowas_table()then returns: statusrows,StreamView: orloaded_onlyafter a load-only run.
Raises:
RuntimeError: If no database is attached (UNAVAILABLE); if the node cannot compute (loaded without a function pod, or a placeholder pod) andload_onlyis false; or if its input is unavailable (CACHE_ONLY): read its stored rows withas_source().
async_execute
Section titled “async_execute”method · orcapod.core.nodes.function_node.FunctionJobNode.async_execute
async def async_execute( input_channel: ReadableChannel[tuple[TagProtocol, DataProtocol]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], *, observer: ExecutionObserverProtocol | None = None, error_policy: Literal['continue', 'fail_fast'] = 'continue', run_id: str | None = None, load_only: bool = False,) -> NoneStreaming async execution for FunctionJobNode.
When a database is attached, uses two-phase execution: replay cached
results first, then compute missing data concurrently. Otherwise,
routes each data through async_process_data directly.
In CACHE_ONLY mode the upstream is unavailable; all cached results
are served directly from persistent storage without touching the input
channel.
Afterwards the node's rows are those of this run's inputs, as after
execute, although every stored result was loaded into its cache. If the
run raises or is cancelled, it defines no rows: the node drops its in-memory
rows and reads as a node with no run in this process.
Parameters:
- input_channel (
ReadableChannel[tuple[TagProtocol, DataProtocol]]): Single readable channel of (tag, data) pairs. - output (
WritableChannel[tuple[TagProtocol, DataProtocol]]): Writable channel for output (tag, data) pairs. - observer (
ExecutionObserverProtocol | None) defaultNone: Optional execution observer for hooks. - error_policy (
Literal['continue', 'fail_fast']) default'continue':"continue"skips rows whose function raised;"fail_fast"raises the exception of the first row whose function raised, asexecutedoes, and cancels the rows still in flight. Rows run concurrently, so that is the first row to fail, not necessarily the first in input order. - run_id (
str | None) defaultNone: Optional pipeline run identifier threaded through toInvocationContext.pipeline_run_id. - load_only (
bool) defaultFalse: As forexecute: fill the node with the stored results of its input rows and compute nothing (ITL-1095).
Raises:
Exception: Under"fail_fast", the exception of the first row whose function raised, inside theExceptionGroupof the node's task group. output is closed first.RuntimeError: If the node was loaded byPipelineJob.load: without a store it has nothing to serve, and it has no function pod to compute with. output is closed first.
OperatorJobNode
Section titled “OperatorJobNode”class · orcapod.core.nodes.operator_node.OperatorJobNode
class OperatorJobNode( operator: OperatorPodProtocol, input_streams: tuple[StreamProtocol, ...] | list[StreamProtocol], tracker_manager: TrackerManagerProtocol | None = None, label: str | None = None, config: OrcapodConfig | None = None, pipeline_database: ArrowDatabaseProtocol | None = None, cache_mode: CacheMode = CacheMode.OFF, table_scope: Literal['pipeline_hash', 'content_hash'] = 'pipeline_hash',)Bases: RunHoldingNode, OperatorNodeBase
DB-backed execution node for operator pod invocations.
Stream node representing an operator invocation with optional DB persistence.
When constructed without database parameters, provides the core stream
interface (identity, schema, iteration) without any persistence. When
databases are provided (either at construction or via attach_databases),
adds pipeline record storage with per-row deduplication, get_all_records()
for retrieving stored results, as_source() for creating a
DerivedSource from DB records, and three-tier cache mode
(OFF / LOG / REPLAY).
Node identity path structure::
operator.uri / schema:{pipeline_hash} / instance:{content_hash}Where pipeline_hash encodes the pipeline structure (operator +
upstream topology) and instance:{content_hash} is the
data-inclusive hash that encodes upstream source identities, ensuring
each unique source combination gets its own cache table.
as_node
Section titled “as_node”method · orcapod.core.nodes.operator_node.OperatorJobNode.as_node
def as_node() -> OperatorNodeReturn the lightweight OperatorNode equivalent of this job node.
Returns:
OperatorNode: A newOperatorNodewith the same operator, input streams,OperatorNode: label, table scope, and tracker manager. Itscontent_hash()/OperatorNode:pipeline_hash()are identical to those of thisOperatorJobNode.
set_ephemeral_database
Section titled “set_ephemeral_database”method · orcapod.core.nodes.operator_node.OperatorJobNode.set_ephemeral_database
def set_ephemeral_database(database: 'ArrowDatabaseProtocol | None') -> NoneNo-op — full ephemeral support for operators is deferred to ITL-509.
clear_cache
Section titled “clear_cache”method · orcapod.core.nodes.operator_node.OperatorJobNode.clear_cache
def clear_cache() -> NoneClear output caches and node identity path cache.
The node then holds no run: it is stale and reads empty, as before its first run
in this process (ITL-1095). Rows it stored stay stored (as_source()).
require_storable_tables
Section titled “require_storable_tables”method · orcapod.core.nodes.operator_node.OperatorJobNode.require_storable_tables
def require_storable_tables( pipeline_database: ArrowDatabaseProtocol, cache_mode: CacheMode = CacheMode.OFF,) -> NoneRefuse a database that cannot store this node's cache (ITL-1162).
With caching on, checks the cache's own columns (__node_content_hash,
__data_context); the operator's output columns are checked when they
are written. Nothing is computed and nothing changes.
Parameters:
- pipeline_database (
ArrowDatabaseProtocol): The database for the cache. - cache_mode (
CacheMode) defaultCacheMode.OFF: The caching behaviour;OFFstores nothing.
Raises:
UnstorableColumnError: If the database cannot store a column, naming this node.
attach_databases
Section titled “attach_databases”method · orcapod.core.nodes.operator_node.OperatorJobNode.attach_databases
def attach_databases( pipeline_database: ArrowDatabaseProtocol, cache_mode: CacheMode = CacheMode.OFF,) -> NoneAttach a database for persistent caching and pipeline records.
Parameters:
- pipeline_database (
ArrowDatabaseProtocol): Database for pipeline records. - cache_mode (
CacheMode) defaultCacheMode.OFF: Caching behaviour (OFF, LOG, or REPLAY).
Raises:
InputValidationError: If the node's output has data-tags, the cache mode stores or replays them (LOGorREPLAY), and the database cannot keep the marker that states them (ITL-1019). Nothing is attached or written.
from_descriptor
Section titled “from_descriptor”classmethod · orcapod.core.nodes.operator_node.OperatorJobNode.from_descriptor
def from_descriptor( descriptor: dict[str, Any], operator: OperatorPodProtocol | None, input_streams: tuple[StreamProtocol, ...] | list[StreamProtocol], databases: dict[str, Any], job_content_hash: str | None = None, expected_row_count: int | None = None,) -> OperatorJobNodeConstruct an OperatorJobNode from a serialized descriptor.
When operator and input_streams are provided the node operates
in full mode — constructed normally via __init__. When
operator is None the node is created in read-only mode with
metadata from the descriptor; computation methods will raise
RuntimeError.
Parameters:
- descriptor (
dict[str, Any]): The serialized node descriptor dict. - operator (
OperatorPodProtocol | None): An optional live operator instance.Nonefor read-only mode. - input_streams (
tuple[StreamProtocol, ...] | list[StreamProtocol]): Input streams for the operator. Empty tuple for read-only mode. - databases (
dict[str, Any]): Mapping of database role names ("pipeline") to database instances. - job_content_hash (
str | None) defaultNone: The content hash of the node in the saved job's run, which its stored rows carry;Nonewhen the file does not record it. - expected_row_count (
int | None) defaultNone: How many rows the database held under the run's key when the job was saved, from the job file's optionalrun.expected_rows;Nonewhen not recorded. Advisory: a read returns what the database holds and only warns (OrcapodDatabaseContentWarning) when it differs.
Returns:
OperatorJobNode: A newOperatorJobNodeinstance.
declared_data_tags
Section titled “declared_data_tags”method · orcapod.core.nodes.operator_node.OperatorJobNode.declared_data_tags
def declared_data_tags() -> tuple[str, ...]Return the data-tags of this node's output, as the node declares them.
A node built in this process declares its operator's predicted output: the
names in both halves of keys(). A node loaded by PipelineJob.load
declares its saved descriptor's output schema, the saved job being an
independent point of verification of the stored table (developer, November 16,
2026). These are the tags the node's stored table must mark.
Returns:
tuple[str, ...]: The data-tags, in tag order; empty for a plain output.
get_cached_output
Section titled “get_cached_output”method · orcapod.core.nodes.operator_node.OperatorJobNode.get_cached_output
def get_cached_output() -> StreamProtocol | NoneReturn cached output stream in REPLAY mode, else None.
Returns:
StreamProtocol | None: The cached stream if REPLAY mode and DB records exist,StreamProtocol | None: otherwise None.
execute
Section titled “execute”method · orcapod.core.nodes.operator_node.OperatorJobNode.execute
def execute( *input_streams: StreamProtocol, observer: ExecutionObserverProtocol | None = None, load_only: bool = False,) -> list[tuple[TagProtocol, DataProtocol]]Execute input streams: compute, persist, and cache.
Parameters:
- *input_streams (
StreamProtocol) default(): Input streams to execute. - observer (
ExecutionObserverProtocol | None) defaultNone: Optional execution observer for hooks. - load_only (
bool) defaultFalse: A load-only run (ITL-1095): the operator still runs on the rows that reach it (it is not a stored computation), but nothing is written: aLOGnode does not store its output.REPLAYreplays as in any run.
Returns:
list[tuple[TagProtocol, DataProtocol]]: Materialized list of (tag, data) pairs.
method · orcapod.core.nodes.operator_node.OperatorJobNode.run
def run(*, load_only: bool = False) -> StreamViewRun the node as a job run would, and return a StreamView of its rows.
State-driven (developer, November 13, 2026; ITL-1095): a node that holds the
output of a completed run and is not stale is not run again; the view shows it.
Otherwise the node runs, after every job node it depends on that holds no
current run (each once, in dependency order, as job.run() would run them;
last_run reports the counts), and the run's writes are committed. A node
built over streams outside a job runs its operator on them directly.
Parameters:
- load_only (
bool) defaultFalse: Asjob.run(load_only=True): the operator runs on the rows that reach it, aLOGnode writes nothing, and the nodes run first load their stored results only.
Returns:
StreamView: AStreamViewof every rowas_table()then returns, naming its rowsStreamView: ashead()then names them.
Raises:
RuntimeError: If the node was loaded byPipelineJob.loadandload_onlyis false.
iter_data
Section titled “iter_data”method · orcapod.core.nodes.operator_node.OperatorJobNode.iter_data
def iter_data() -> Iterator[tuple[TagProtocol, DataProtocol]]Return an iterator over the (tag, data) rows of the node's current run.
Read-only: never computes and never reads the database. Yields the rows of the
latest run(), execute() or async_execute(); nothing before a run in
this process, after clear_cache() or once the node is stale, in every cache
mode (ITL-1095). as_source() reads what the node stored (LOG).
as_table
Section titled “as_table”method · orcapod.core.nodes.operator_node.OperatorJobNode.as_table
def as_table( *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False,) -> 'pa.Table'Return the output of the node's current run as a PyArrow table.
Read-only: never computes and never reads the database. Returns the rows of the
latest run(), execute() or async_execute(); before a run in this
process, after clear_cache() or once the node is stale, a zero-row table
with the node's columns, in every cache mode (ITL-1095).
That zero-row table has the columns, Arrow types and nullability of the
node's rows, system tags and source info included: it is the operator's
result over its inputs' zero-row tables, got without computing or reading
a row (ITL-912). Known limitation: when an input cannot say its Arrow
types without rows (a PollingSource before its first poll, an unbound
input slot or SourceProxy, a DerivedSource, or a loaded node with
no live upstream), the table is built from the node's Python output
schema: an int8 column reads as int64 and the system-tag columns
are absent (ITL-1089, ITL-985).
method · orcapod.core.nodes.operator_node.OperatorJobNode.head
def head( n: int = 10, *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False,) -> StreamViewReturn the first rows of the node's current run; never runs the operator.
After a run (or execute) the rows are the node's output in memory. Before a
run in this process the node shows no rows and says so, pointing to run()
and to as_source(), in every cache mode; once stale it says its inputs
changed (ITL-1095). The database is never read. See StreamBase.head.
get_all_records
Section titled “get_all_records”method · orcapod.core.nodes.operator_node.OperatorJobNode.get_all_records
def get_all_records( columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False,) -> 'pa.Table | None'Retrieve all stored records from the pipeline database.
Returns the stored output table with column filtering applied
per ColumnConfig conventions. Returns None when no
database is attached or no matching records exist. Switching to OFF
does not remove records stored by an earlier LOG run.
This is the raw records view: a data-tag column keeps the marker the
stored table states it with (ITL-1019), checked against the node on read.
as_source() reads the binding from it.
Raises:
InputValidationError: If the stored table and the node disagree on which tags are data-tags.
as_source
Section titled “as_source”method · orcapod.core.nodes.operator_node.OperatorJobNode.as_source
def as_source(tags_as_data: 'Mapping[str, str] | None' = None)Return a DerivedSource backed by the DB records of this node.
Its source_id is the DerivedSource default: without
tags_as_data, derived:<algorithm label>:<hex> of this node's
content hash.
Parameters:
- tags_as_data (
'Mapping[str, str] | None') defaultNone: Maps a tag column of this node to the name of a data column that receives a copy of it. When given, the result is a new root source with its own provenance and asource_idderived from this node's content hash and the mapping; seeDerivedSource.
Raises:
RuntimeError: If no database is attached.InputValidationError: Iftags_as_datais not valid for this node's columns.
async_execute
Section titled “async_execute”method · orcapod.core.nodes.operator_node.OperatorJobNode.async_execute
async def async_execute( inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], *, observer: 'ExecutionObserverProtocol | None' = None, load_only: bool = False,) -> NoneAsync execution with cache mode handling when DB is attached.
Every mode except REPLAY runs the wrapped operator's async_execute,
forwards its rows downstream and keeps them on the node, exactly as
execute() does, so iter_data() and as_table() return them
after the run whichever orchestrator ran the node.
Without a database, or with one in OFF mode, nothing is stored.
Parameters:
- inputs (
Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]]): Sequence of readable channels from upstream nodes. - output (
WritableChannel[tuple[TagProtocol, DataProtocol]]): Writable channel for output (tag, data) pairs. - observer (
'ExecutionObserverProtocol | None') defaultNone: Optional execution observer for hooks. - load_only (
bool) defaultFalse: As forexecute: aLOGnode stores nothing (ITL-1095).
SourceJobNode
Section titled “SourceJobNode”class · orcapod.core.nodes.source_node.SourceJobNode
class SourceJobNode( name: str, tag_schema: Schema | None = None, data_schema: Schema | None = None, bound_source: StreamProtocol | None = None, data_context: str | contexts.DataContext | None = None,)Bases: SourceNodeBase
Execution-ready source node wrapping an optional concrete stream.
Used inside PipelineJob._persistent_node_map. The _concrete
field is mutable — PipelineJob.bind(sources={...}) updates it
in-place so that downstream FunctionJobNode objects (which hold a
reference to this same object) automatically see the new concrete source
without cascading reference updates.
Hash behaviour:
content_hash()— delegates to the bound source'sidentity_structure()when bound (viaidentity_structure()override); falls back to schema-based identity when unbound.pipeline_hash()— always schema-based (inherited); never data-inclusive. This invariant keeps DB paths stable across different data sources bound to the same slot.
Parameters:
- name (
str): Slot name. - tag_schema (
Schema| None) defaultNone: Tag schema. Optional when bound_source is provided; inferred frombound_source.output_schema()when omitted. Stored for future rebind validation. - data_schema (
Schema| None) defaultNone: Data schema. Optional when bound_source is provided; inferred frombound_source.output_schema()when omitted. Stored for future rebind validation. - bound_source (
StreamProtocol | None) defaultNone: Optional concrete stream. Can be set or replaced later viajob_node.bound_source = source. - data_context (
str | contexts.DataContext | None) defaultNone: Optional data context override.
Raises:
ValueError: If both tag_schema and data_schema are omitted and bound_source isNone.
method · orcapod.core.nodes.source_node.SourceJobNode.run
def run(*, load_only: bool = False) -> StreamViewRead the bound source's rows and return a StreamView of all of them.
As every job node's run(), it first checks the bound source for new data
(check_sources, ITL-1102): a check that raises warns
(SourceCheckFailedWarning) and the source's held rows are read. A source is
read, never computed, so load_only changes nothing, except that a load-only
run refuses a bound stream that computes or delivers when read (ITL-1121); a job
node's run() takes it (ITL-1095).
Parameters:
- load_only (
bool) defaultFalse: Accepted for every job node; a source reads its rows either way.
Raises:
UnboundSourceError: When no concrete source is bound.ValueError: Ifload_onlyand the bound stream computes or delivers when read (node_run.refuse_unloadable_inputs).
bound_source
Section titled “bound_source”property · orcapod.core.nodes.source_node.SourceJobNode.bound_source
Type: StreamProtocol | None
The concrete stream currently bound to this slot, or None.
last_modified
Section titled “last_modified”property · orcapod.core.nodes.source_node.SourceJobNode.last_modified
Type: datetime | None
When the slot's content last changed: its binding or its bound source.
The later of the time the slot was bound and the bound source's own
last_modified (a CachedSource or PollingSource moves it when it
reads new rows). A bound source that does not know its modification time makes
the slot's unknown too (None), so every node below it is stale. Every run
first checks the bound source for new data (check_for_new_data, ITL-1102),
which moves its modified time when it found some. A source that cannot see its
data change (one built over data held in memory) is not re-read: build a new
source object and bind it, or clear the dependent nodes.
identity_structure
Section titled “identity_structure”method · orcapod.core.nodes.source_node.SourceJobNode.identity_structure
def identity_structure() -> AnyDelegate to bound_source.identity_structure() when bound.
This is the correct extension point: content_hash() flows from identity_structure(), so overriding here avoids bypassing the caching and resolver logic in ContentIdentifiableBase.content_hash().
Returns:
Any: Bound source's identity structure when bound; schema-basedAny: identity (inherited from SourceNodeBase) when unbound.
iter_data
Section titled “iter_data”method · orcapod.core.nodes.source_node.SourceJobNode.iter_data
def iter_data() -> Iterator[tuple[TagProtocol, DataProtocol]]Delegate to concrete source, or raise if unbound.
Raises:
UnboundSourceError: When no concrete source is attached.
method · orcapod.core.nodes.source_node.SourceJobNode.head
def head( n: int = 10, *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False,) -> StreamViewReturn the bound source's head, or no rows when no source is bound.
A narrower bound source's rows are shown as rows of the declared schema.
async_iter_data
Section titled “async_iter_data”method · orcapod.core.nodes.source_node.SourceJobNode.async_iter_data
async def async_iter_data() -> AsyncIterator[tuple[TagProtocol, DataProtocol]]Delegate to bound_source.async_iter_data() when bound.
Overrides SourceNodeBase.async_iter_data() to route through the
bound source's own async generator instead of wrapping iter_data()
synchronously. This ensures that dynamic sources such as
PollingSource run their async polling loop rather than returning
a static snapshot.
Raises:
UnboundSourceError: When no concrete source is attached.
output_schema
Section titled “output_schema”method · orcapod.core.nodes.source_node.SourceJobNode.output_schema
def output_schema( *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False,) -> tuple[Schema, Schema]Return (tag_schema, data_schema), delegating to bound_source when set.
When bound_source has the declared schema, this is a transparent
pass-through to bound_source.output_schema(columns=columns, all_info=all_info)
so callers get the same result as querying the source directly. A narrower
bound source (x: int for a declared x: int | None) is given as the
declared schema (ITL-1112).
When unbound, delegates to SourceNodeBase.output_schema() which includes
system-tag schema entries derived from the declared schemas.
Parameters:
- columns (
ColumnConfig| dict[str, Any] | None) defaultNone: Column selection config. - all_info (
bool) defaultFalse: IfTrue, include all available column groups.
Returns:
as_table
Section titled “as_table”method · orcapod.core.nodes.source_node.SourceJobNode.as_table
def as_table( *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False,) -> pa.TableMaterialize the concrete source as a PyArrow Table.
A narrower bound source's rows are given as rows of the declared schema (ITL-1112).
Parameters:
- columns (
ColumnConfig| dict[str, Any] | None) defaultNone: Column selection config. - all_info (
bool) defaultFalse: If True, include all metadata columns.
Raises:
UnboundSourceError: When no concrete source is attached.
from_stream
Section titled “from_stream”classmethod · orcapod.core.nodes.source_node.SourceJobNode.from_stream
def from_stream(stream: StreamProtocol, name: str | None = None) -> 'SourceJobNode'Create a SourceJobNode from stream using three-way logic.
The three cases are:
- stream is already a
SourceJobNode— copy it, preserving the existingbound_source(the SJN itself is not used as the bound source). - stream is a
SourceNode(schema-only, no data) — create an unboundSourceJobNodewith the same name and schemas. The resulting SJN hasbound_source=Noneand the samecontent_hash()as stream (schema-based). - Any other concrete stream (
ArrowTableSource, etc.) — create a boundSourceJobNodewhosecontent_hash()delegates to the concrete stream.
Used by Pipeline.compile for each leaf of the pipeline. The stream is
bound as any source is, so a leaf without exactly one source-id and one
record-id system-tag column (a raw ArrowTableStream) is refused
(ITL-1126).
Parameters:
- stream (
StreamProtocol): The stream to wrap. - name (
str | None) defaultNone: Optional explicit slot name. In all three cases, when name is provided it takes precedence over the stream's own name. For concrete streams (Case 3), when name isNonethe slot name is resolved as:stream.source_idforRootSourceinstances, or"{stream.label}:{hash_prefix}"for any other concrete stream type.
Returns:
'SourceJobNode': ASourceJobNodeconfigured according to the case above.
as_node
Section titled “as_node”method · orcapod.core.nodes.source_node.SourceJobNode.as_node
def as_node() -> SourceNodeReturn the lightweight SourceNode equivalent of this job node.
Returns:
SourceNode: A newSourceNodewith the same name and schemas.