Pipeline
Top-level objects for composing and executing Orcapod data pipelines. Pipeline is a pure
blueprint; PipelineJob binds a pipeline to sources and a database and runs it.
Pipeline
Section titled “Pipeline”class · orcapod.pipeline.graph.Pipeline
class Pipeline( name: str | tuple[str, ...], tracker_manager: cp.TrackerManagerProtocol | None = None, auto_compile: bool = True,)Bases: AbstractPipelineBase[GraphNode]
A pure computational blueprint recording operator and function pod invocations.
During the with block, operator and function pod invocations are
recorded into an internal graph via the unified _record_invocation()
path inherited from AbstractPipelineBase. On context exit,
compile() (also inherited) rewires the graph into a frozen DAG:
- Leaf streams not registered as invocations →
SourceNodedeclarations - Function pod invocations →
FunctionNode - Operator invocations →
OperatorNode
To run a Pipeline, use
PipelineJob.from_pipeline(pipeline, sources=..., database=...) to create
a PipelineJob.
Parameters:
- name (
str | tuple[str, ...]): Pipeline name (string or tuple). Used as the path prefix for all cache/pipeline paths when the pipeline is run via aPipelineJob. - auto_compile (
bool) defaultTrue: IfTrue(default),compile()is called automatically when the context manager exits.
method · orcapod.pipeline.graph.Pipeline.save
def save(path: str | Path) -> NoneSerialize the pure pipeline blueprint to a JSON file.
Saves the full pipeline topology: SourceNode declarations, function and operator pod configurations, and all edge connections. Runtime state — databases, execution context, and run metadata — is not persisted.
Parameters:
- path (
str | Path): File path to write JSON output to.
Raises:
ValueError: If the pipeline has not been compiled.
classmethod · orcapod.pipeline.graph.Pipeline.load
def load(path: str | Path) -> 'Pipeline'Deserialize a pure pipeline blueprint from a JSON file.
Reconstructs topology and SourceNode declarations. The loaded
pipeline is topology-only — to run it, use
PipelineJob.from_pipeline(pipeline, sources=..., database=...).
Parameters:
- path (
str | Path): Path to the JSON file produced bysave().
Returns:
'Pipeline': A compiledPipelineinstance with SourceNode leaf nodes.
Raises:
ValueError: If the file's format version is unsupported.
PipelineJob
Section titled “PipelineJob”class · orcapod.pipeline.job.PipelineJob
class PipelineJob( name: str | tuple[str, ...] = 'pipeline', database: 'ArrowDatabaseProtocol | None' = None, execution_context: 'ExecutionContext | None' = None, tracker_manager: cp.TrackerManagerProtocol | None = None, auto_compile: bool = True,)Bases: AbstractPipelineBase[JobNode]
Pipeline + source bindings + execution context.
PipelineJob is the everyday working object. It is built incrementally:
its with-block records both the DAG structure and any concrete source
bindings simultaneously. Concrete sources are automatically promoted to
SourceJobNode declarations in the underlying graph, with their
concrete instances stored in job.sources.
After the with block, job.pipeline returns a lightweight
Pipeline blueprint (SourceNode-only leaves). job.run() executes
the resolvable subgraph — nodes whose upstream SourceNodes are all bound.
PipelineJob can also be created from a Pipeline via
PipelineJob.from_pipeline(pipeline, sources=..., database=...) for the
"explicit blueprint" workflow.
Parameters:
- name (
str | tuple[str, ...]) default'pipeline': Pipeline name (string or tuple). Used as the path prefix for all cache/pipeline paths when the pipeline is run. - database (
'ArrowDatabaseProtocol | None') defaultNone: Database for result caching and operator records. - execution_context (
'ExecutionContext | None') defaultNone: Optional execution configuration. - tracker_manager (
cp.TrackerManagerProtocol | None) defaultNone: Optional tracker manager override. - auto_compile (
bool) defaultTrue: IfTrue(default),compile()is called automatically when the context manager exits.
compile
Section titled “compile”method · orcapod.pipeline.job.PipelineJob.compile
def compile() -> NoneCompile recorded invocations into a frozen DAG of job nodes.
Delegates to AbstractPipelineBase.compile() which uses
source_node_class, function_node_class, and
operator_node_class to create job node instances.
Concrete source bindings already in _sources are preserved.
Any SourceJobNode in _persistent_node_map that has a bound
source also contributes to _sources (covers both the
direct-recording path and the bind()-before-compile path).
The lazy compiled_pipeline cache is invalidated.
If _database is set, distributes databases to all function and
operator job nodes.
last_run
Section titled “last_run”property · orcapod.pipeline.job.PipelineJob.last_run
Type: RunSummary | None
The summary of the last run this job completed, or None before one.
The same object run and run_async returned: per function node, the rows
computed, read from the database and failed, and each node's first error. Use
job.last_run.print_traceback("<node label>") to see a failed row's full
traceback. A run that raised leaves it unchanged. A loaded job has none.
compiled_pipeline
Section titled “compiled_pipeline”property · orcapod.pipeline.job.PipelineJob.compiled_pipeline
Type: 'Pipeline'
The lightweight Pipeline blueprint for this job (lazy cache).
Computed on first access by calling as_pipeline(). Invalidated
by compile(), which sets _compiled_pipeline to None so the
next access recomputes a fresh schema-normalised blueprint.
bind() does not invalidate this cache: the blueprint contains
only SourceNode schema-slot declarations with no concrete data, so
it is unaffected by changes to concrete source bindings.
Raises:
RuntimeError: If the job has not been compiled yet.
pipeline
Section titled “pipeline”property · orcapod.pipeline.job.PipelineJob.pipeline
Type: 'Pipeline'
The compiled pure Pipeline (SourceNode-only leaves).
Raises:
RuntimeError: If the with-block has not been completed yet.
sources
Section titled “sources”property · orcapod.pipeline.job.PipelineJob.sources
Type: dict[str, cp.StreamProtocol]
Mapping of SourceNode name to bound concrete source.
database
Section titled “database”property · orcapod.pipeline.job.PipelineJob.database
Type: 'ArrowDatabaseProtocol | None'
The database used for result caching.
execution_context
Section titled “execution_context”property · orcapod.pipeline.job.PipelineJob.execution_context
Type: 'ExecutionContext | None'
The execution configuration, or None if unset.
from_pipeline
Section titled “from_pipeline”classmethod · orcapod.pipeline.job.PipelineJob.from_pipeline
def from_pipeline( pipeline: 'Pipeline', database: 'ArrowDatabaseProtocol | None' = None, sources: 'dict[str, cp.StreamProtocol] | None' = None, execution_context: 'ExecutionContext | None' = None,) -> 'PipelineJob'Create a runnable PipelineJob from a compiled Pipeline.
Converts the Pipeline blueprint to an InvocationGraph via
pipeline.to_invocations(), then reconstructs a PipelineJob
using from_invocations() (which calls SourceJobNode.from_stream
for each source, creating unbound job nodes). If sources are
provided, validates and binds them.
Parameters:
- pipeline (
'Pipeline'): A compiledPipeline(pipeline._compiledmust beTrue). - database (
'ArrowDatabaseProtocol | None') defaultNone: Database for result caching and operator records. - sources (
'dict[str, cp.StreamProtocol] | None') defaultNone: Mapping ofSourceNode.name→ concrete source. - execution_context (
'ExecutionContext | None') defaultNone: Optional execution configuration.
Returns:
'PipelineJob': A newPipelineJobready to run (orbind()further).
Raises:
ValueError: If pipeline has not been compiled.
method · orcapod.pipeline.job.PipelineJob.bind
def bind( sources: 'dict[str, cp.StreamProtocol] | None' = None, database: 'ArrowDatabaseProtocol | None' = None, execution_context: 'ExecutionContext | None' = None,) -> NoneUpdate bindings in place. Returns None.
Mutating — modifies self directly. Existing bindings not mentioned
in this call are preserved.
When sources is provided, each concrete source is validated against
its matching SourceJobNode slot schema, then the corresponding
SourceJobNode.bound_source is updated in-place. A source fits its slot
when its tag and data columns have the declared names, roles and Python
types, so a bind never changes a pipeline hash (ITL-1112). Every source is
validated before anything is changed: a refused source leaves the job's
bindings and database as they were.
When database is provided and differs from the current database,
_distribute_databases() is called so that all job nodes receive
live DB references immediately.
Parameters:
- sources (
'dict[str, cp.StreamProtocol] | None') defaultNone: Mapping ofSourceNode.name→ concrete source. - database (
'ArrowDatabaseProtocol | None') defaultNone: Replaces the current database and triggers DB redistribution. - execution_context (
'ExecutionContext | None') defaultNone: Replaces the current execution context.
Raises:
SourceSpecMismatchError: If any source's columns differ from its slot's declared names, roles or types.ValueError: If a source key has no matchingSourceJobNodeslot.UnstorableColumnError: If database cannot store a table of a node (ITL-1162); the job is left as it was.
apply_node_config
Section titled “apply_node_config”method · orcapod.pipeline.job.PipelineJob.apply_node_config
def apply_node_config(node_config: NodeConfig, replace_existing: bool = False) -> NoneApply a NodeConfig to all FunctionJobNodes in this pipeline.
Parameters:
- node_config (
NodeConfig): The config to apply. - replace_existing (
bool) defaultFalse: IfFalse, directly setsnode_configon every node, replacing whatever config each node currently holds. IfTrue, mergesnode_configinto each node's existing config — non-Nonefields innode_configoverride the node's current values, whileNonefields leave existing values unchanged.
Raises:
RuntimeError: If the job has not been compiled yet (usewith job:to record a DAG before applying node config).
as_pipeline
Section titled “as_pipeline”method · orcapod.pipeline.job.PipelineJob.as_pipeline
def as_pipeline() -> 'Pipeline'Return the lightweight Pipeline blueprint for this job.
Walks _persistent_node_map topologically and rewires each job
node to a fresh blueprint node whose upstreams point at the
already-built blueprint nodes in node_map (not at job nodes).
This ensures that the content_hash of every node in the returned
Pipeline is consistent with the hash keys in
_persistent_node_map.
Returns:
'Pipeline': A compiledPipelinewhose_persistent_node_mapcontains'Pipeline': only lightweightSourceNode/FunctionNode/OperatorNode'Pipeline': objects with blueprint (non-job) upstream references. Keys in'Pipeline':_persistent_node_mapmatch those inself._persistent_node_map.
Raises:
RuntimeError: If this job has not been compiled.
to_invocations
Section titled “to_invocations”method · orcapod.pipeline.job.PipelineJob.to_invocations
def to_invocations()Extract an InvocationGraph from the schema-based blueprint.
Delegates to as_pipeline().to_invocations() so that the
produced invocations always use schema-based (not data-based)
hashes, regardless of whether sources are currently bound.
This ensures stable serialization across bind/rebind cycles.
Raises:
RuntimeError: If this job has not been compiled.
unbound_sources
Section titled “unbound_sources”property · orcapod.pipeline.job.PipelineJob.unbound_sources
Type: list[str]
Names of source slots not yet bound in this job.
Computed live from _persistent_node_map — reflects the current
binding state at all times (before and after run).
Returns:
list[str]: List of unbound source slot names, in order of appearance inlist[str]: the pipeline graph. Empty list if the job is not yet compiled.
is_complete
Section titled “is_complete”method · orcapod.pipeline.job.PipelineJob.is_complete
def is_complete() -> boolReturn True when all source nodes are bound and a database is set.
Returns:
bool:Trueif all SourceNode slots are bound and a database is set.
is_runnable
Section titled “is_runnable”method · orcapod.pipeline.job.PipelineJob.is_runnable
def is_runnable(node_label: str) -> boolReturn whether a live job has all upstream inputs of node_label resolved.
A loaded job is read-only and always returns False.
Uses hash-based traversal of _hash_graph combined with
_persistent_node_map to check whether every upstream source slot
is bound. This avoids relying on the object-identity graph
(_node_graph), which breaks after bind() because
SourceJobNode.__hash__() changes when bound_source is mutated.
Parameters:
- node_label (
str): Label of the node to check.
Returns:
bool:Trueif the node can be executed with current bindings.
method · orcapod.pipeline.job.PipelineJob.run
def run( orchestrator: 'SyncPipelineOrchestrator | AsyncPipelineOrchestrator | None' = None, observer: 'ExecutionObserverProtocol | None' = None, materialize_results: bool = True, commit_every_rows: int = 100, *, load_only: bool = False,) -> RunSummaryExecute the resolvable subgraph of this job in place.
Nodes whose upstream includes an unbound source (and all their
dependents) are excluded from execution. Partial execution is a
first-class outcome — unbound_sources reports which source
slots were excluded.
After a successful run, job.nodes returns the execution-ready
nodes with populated database caches.
State-driven, as every job node's run() (developer, November 14, 2026;
ITL-1095): a node that holds a current run (completed in this process, not
stale, and for a full run not filled by a load-only run) is held, not run; its
rows feed the nodes after it, and nothing is computed, read, written or
delivered for it. A source is read only for the nodes that run. So a second run
of an unchanged job does nothing. To run a node again, call clear_cache() on
it (every node below it runs too), or bind a new source object.
job.run(orchestrator=AsyncPipelineOrchestrator()) works the same in a
script and inside a running event loop, such as a Jupyter kernel: there the
run goes on a separate thread while the call waits, and blocks that loop
until it ends. From async code, await job.run_async() instead.
Parameters:
- orchestrator (
'SyncPipelineOrchestrator | AsyncPipelineOrchestrator | None') defaultNone: Orchestrator to use for execution. Defaults toSyncPipelineOrchestratorwhenNone. - observer (
'ExecutionObserverProtocol | None') defaultNone: Optional execution observer. - materialize_results (
bool) defaultTrue: Controls whether the orchestrator retains node outputs in its transient in-process buffer during execution. WhenTrue(the default) the buffer is kept for the duration of the run. WhenFalseeach node's buffer is released as soon as the node completes, reducing peak memory. This setting does not affect database persistence or post-run result access: DB records are written regardless, andjob.nodes["label"]returns the same rows fromas_table()anditer_data()in either case, on either orchestrator. An operator node with the defaultcache_mode=OFF(the only mode a job sets) keeps its output in memory and stores nothing. Itsget_all_records()returnsNonewhen no matching records exist; records from an earlier LOG run remain readable after switching to OFF. - commit_every_rows (
int) default100: How many rows a function node processes between commits of its results and pipeline records to the database (default 100). Each node also commits when it finishes, before a failed run raises, and the job commits at its end. On a durable database (DeltaTableDatabase) a hard crash (power loss, killed process) loses at most the rows of the running node since its last commit; re-running the job recomputes only those. A larger value commits less often (faster, fewer files);1commits after every row. A node run on its own, outsiderun(), commits after every row. It must be smaller than the database'sbatch_sizewhen the database has one (DeltaTableDatabase, default 1000): the database commits every pending table by itself once one table's pending rows reach that size, which would break the node's results-before-records commit order. - load_only (
bool) defaultFalse: Fill the nodes from what is already stored and compute nothing (ITL-1095). Sources are read and operators run on the rows that reach them; each function node takes the results stored for this run's inputs (its own records under its content hash) and skips the inputs without one, which the summary counts (not_stored); a function node forwards only the rows it loaded. No user function is called, no side effect is delivered (a side-effect node passes its rows on) and nothing is written to the database, aLOGoperator's cache included. Afterwards each node shows the rows it loaded.
Rows whose function raised (with the default error_policy="continue") are
skipped, and the run goes on. At the end, if any row failed, a single
RowFailuresWarning names each node with failed rows, how many failed, the
first error, and how to print its traceback. A run without failures is quiet.
Returns:
RunSummary: The run'sRunSummary: per function node, the rows computed, read fromRunSummary: the database and failed, with each node's first error. Shown at the end of aRunSummary: notebook cell; also kept asjob.last_run. Results are read fromRunSummary:job.nodes.
Raises:
ValueError: If no database is set, orcommit_every_rowsis not a positive integer smaller than the database'sbatch_size.RuntimeError: If no pipeline has been compiled, or the job was built byPipelineJob.load(a loaded job is read-only).OrcapodDatabaseWriteError: If a node's results or records could not be written to the database, whatever the orchestrator'serror_policy. The run stops at the first failed write. Writes that succeeded before it are flushed first, so rows stored before the failure are kept, and re-running the job does not recompute stored function results. A failed commit of a node's batch names the node but no row (flush=True): the batch's writes stay pending and are retried once by the flush that follows the failure.OrcapodDatabaseReadError: If a stored table exists but could not be read, whatever the orchestrator'serror_policy, also when the read was needed to write a node's results. Its rows are not recomputed and the table is not overwritten.
run_async
Section titled “run_async”method · orcapod.pipeline.job.PipelineJob.run_async
async def run_async( orchestrator: 'AsyncPipelineOrchestrator | None' = None, observer: 'ExecutionObserverProtocol | None' = None, materialize_results: bool = True, commit_every_rows: int = 100, *, load_only: bool = False,) -> RunSummaryExecute the job with the async orchestrator, from async code.
The awaitable form of run: await job.run_async() runs on the caller's
event loop instead of blocking it. job.run(orchestrator=AsyncPipelineOrchestrator())
also works inside a running loop (a Jupyter kernel), but blocks that loop until
the run ends. Both store the same rows and record ids. Arguments, errors and
result are those of run.
Parameters:
- orchestrator (
'AsyncPipelineOrchestrator | None') defaultNone: The async orchestrator to run with. Defaults toAsyncPipelineOrchestrator(). - observer (
'ExecutionObserverProtocol | None') defaultNone: Optional execution observer. - materialize_results (
bool) defaultTrue: As forrun. - commit_every_rows (
int) default100: As forrun. - load_only (
bool) defaultFalse: As forrun.
Returns:
RunSummary: The run'sRunSummary, asrunreturns.
Raises:
TypeError: Iforchestratoris not anAsyncPipelineOrchestrator; run aSyncPipelineOrchestratorwithjob.run.ValueError: As forrun.RuntimeError: As forrun.OrcapodDatabaseWriteError: As forrun.OrcapodDatabaseReadError: As forrun.
method · orcapod.pipeline.job.PipelineJob.save
def save(path: 'str | Path') -> NoneSerialize this job to a JSON file.
Saves topology (via the embedded Pipeline) plus bindings metadata and
run state. The format covers both "template" (pre-run) and
"completed" (post-run) states — distinguished by run.status.
Parameters:
- path (
'str | Path'): File path to write JSON output to.
Raises:
ValueError: If no compiled pipeline exists.
classmethod · orcapod.pipeline.job.PipelineJob.load
def load( path: 'str | Path', database: 'ArrowDatabaseProtocol | None' = None,) -> 'PipelineJob'Deserialize a PipelineJob from a JSON file to read its stored results.
A loaded job is for reading the results its saved run stored; it cannot be
run again, because function pods are not saved with the job. Its nodes have
not run in this process, so they are empty (ITL-1095): read a node's stored
results through its stored view, job.nodes[label].as_source(), or with
get_all_records(). Operators store outputs only when caching is enabled;
the stored view of a loaded operator whose run's rows cannot be named raises
RuntimeError.
The embedded Pipeline blueprint is always restored. Sources whose data
lives in files (CSV, Delta) are rebuilt from the saved config. In-memory
sources (DictSource, ListSource, ArrowTableSource,
DataFrameSource) are not saved; loading logs that at INFO, and their
source nodes can be read after job.bind(sources=...). A source that
cannot be rebuilt for another reason (a moved file) is logged as a
warning. Neither affects reading the other nodes.
Parameters:
- path (
'str | Path'): Path to the JSON file produced bysave(). - database (
'ArrowDatabaseProtocol | None') defaultNone: Optional database override. When provided, takes precedence over any database configuration in the file.
Returns:
'PipelineJob': A read-onlyPipelineJobwhose nodes' stored views serve their stored'PipelineJob': results.
Raises:
ValueError: If the file's format version is unsupported, or if it was saved before the database key was renamed (bindings.storeinstead ofbindings.database) and nodatabaseis given. An explicitdatabasemakes the saved block irrelevant, so such a file then loads.
PipelineDiagram
Section titled “PipelineDiagram”class · orcapod.pipeline.diagram.PipelineDiagram
class PipelineDiagram( pipeline: _Drawable, *, direction: Direction = 'LR', collapse: bool = False, max_nodes: int = 400,)The DAG of a pipeline or job, drawn for a notebook.
In development: this interface may change.
Shown at the end of a notebook cell it is an SVG picture of the nodes (sources,
operators, function pods, side-effect pods) and the streams between them. Where the
notebook runs scripts, the picture can be dragged and zoomed, and selecting a node
shows its tag and data schema and, for a job, its run status and the last run's row
counts. Without scripts the picture and a table of the nodes remain.
print gives a plain-text listing.
Making a diagram computes nothing: it reads each node's predicted output_schema(),
and for a job what each node holds in memory and job.last_run.
Use Pipeline.diagram() or PipelineJob.diagram() to make one. The first
diagram made in a process issues InDevelopmentWarning (once per process).
Parameters:
- pipeline (
_Drawable): A compiledPipelineorPipelineJob. - direction (
Direction) default'LR':"LR"lays the graph out left to right,"TB"top to bottom. - collapse (
bool) defaultFalse: Fold the operators that only choose, rename or join columns (joins, column selections, drops and maps) and feed exactly one node into the edges they lie on: such an edge is dashed, and the folded steps are listed in the details of the node it leads to. - max_nodes (
int) default400: Above this many nodes (after folding), the notebook shows the plain-text listing instead of the picture.
Raises:
ValueError: Ifdirectionis not"LR"or"TB", ormax_nodesis not positive.RuntimeError: If the pipeline has not been compiled.
property · orcapod.pipeline.diagram.PipelineDiagram.title
Type: str
Pipeline '<name>' or Job '<name>'.
In development: this interface may change.
to_dict
Section titled “to_dict”method · orcapod.pipeline.diagram.PipelineDiagram.to_dict
def to_dict() -> dict[str, Any]The diagram's model: plain data that any renderer can draw.
In development: this interface may change.
Its shape may change: it carries no version marker yet.
Returns:
dict[str, Any]:{"kind", "name", "title", "nodes", "edges"}. Each node hasindex,dict[str, Any]:label,kind(itsnode_type),kind_text,detail,dict[str, Any]:status(""for a pipeline),tag_schemaanddata_schemadict[str, Any]: (lists of[column, type text]) andfacts(lists of[name, text]);dict[str, Any]: each edge hassourceandtarget(node indices) andfoldeddict[str, Any]: (the labels of the steps folded into it).
to_text
Section titled “to_text”method · orcapod.pipeline.diagram.PipelineDiagram.to_text
def to_text() -> strThe nodes as plain text, one per line, each with its inputs.
In development: this interface may change.
Returns:
str: A title line, then<label> (<kind> · <detail> · <status>) <- <inputs>str: per node, in topological order with ties broken by label.
to_svg
Section titled “to_svg”method · orcapod.pipeline.diagram.PipelineDiagram.to_svg
def to_svg() -> strThe picture alone, as an SVG element.
In development: this interface may change.
Returns:
str: The<svg>element; it uses no style sheet and no script.
to_html
Section titled “to_html”method · orcapod.pipeline.diagram.PipelineDiagram.to_html
def to_html() -> strThe HTML fragment a notebook shows: picture, node table, model and script.
In development: this interface may change.
Returns:
str: The fragment. Abovemax_nodesnodes, the plain-text listing in astr:<pre>element instead.
method · orcapod.pipeline.diagram.PipelineDiagram.save
def save(path: str | Path) -> NoneWrite the diagram as a standalone HTML page, viewable in any browser offline.
In development: this interface may change.
Parameters:
- path (
str | Path): The file to write.