Skip to content
This site documents the development version of Orcapod (the main branch), which may differ from the latest release. For the releases, see PyPI and the changelog.

Pipeline

Top-level objects for composing and executing Orcapod data pipelines. Pipeline is a pure blueprint; PipelineJob binds a pipeline to sources and a database and runs it.

class · orcapod.pipeline.graph.Pipeline

class Pipeline(
name: str | tuple[str, ...],
tracker_manager: cp.TrackerManagerProtocol | None = None,
auto_compile: bool = True,
)

Bases: AbstractPipelineBase[GraphNode]

A pure computational blueprint recording operator and function pod invocations.

During the with block, operator and function pod invocations are recorded into an internal graph via the unified _record_invocation() path inherited from AbstractPipelineBase. On context exit, compile() (also inherited) rewires the graph into a frozen DAG:

  • Leaf streams not registered as invocations → SourceNode declarations
  • Function pod invocations → FunctionNode
  • Operator invocations → OperatorNode

To run a Pipeline, use PipelineJob.from_pipeline(pipeline, sources=..., database=...) to create a PipelineJob.

Parameters:

  • name (str | tuple[str, ...]): Pipeline name (string or tuple). Used as the path prefix for all cache/pipeline paths when the pipeline is run via a PipelineJob.
  • auto_compile (bool) default True: If True (default), compile() is called automatically when the context manager exits.

method · orcapod.pipeline.graph.Pipeline.save

def save(path: str | Path) -> None

Serialize the pure pipeline blueprint to a JSON file.

Saves the full pipeline topology: SourceNode declarations, function and operator pod configurations, and all edge connections. Runtime state — databases, execution context, and run metadata — is not persisted.

Parameters:

  • path (str | Path): File path to write JSON output to.

Raises:

  • ValueError: If the pipeline has not been compiled.

classmethod · orcapod.pipeline.graph.Pipeline.load

def load(path: str | Path) -> 'Pipeline'

Deserialize a pure pipeline blueprint from a JSON file.

Reconstructs topology and SourceNode declarations. The loaded pipeline is topology-only — to run it, use PipelineJob.from_pipeline(pipeline, sources=..., database=...).

Parameters:

  • path (str | Path): Path to the JSON file produced by save().

Returns:

  • 'Pipeline': A compiled Pipeline instance with SourceNode leaf nodes.

Raises:

  • ValueError: If the file's format version is unsupported.

class · orcapod.pipeline.job.PipelineJob

class PipelineJob(
name: str | tuple[str, ...] = 'pipeline',
database: 'ArrowDatabaseProtocol | None' = None,
execution_context: 'ExecutionContext | None' = None,
tracker_manager: cp.TrackerManagerProtocol | None = None,
auto_compile: bool = True,
)

Bases: AbstractPipelineBase[JobNode]

Pipeline + source bindings + execution context.

PipelineJob is the everyday working object. It is built incrementally: its with-block records both the DAG structure and any concrete source bindings simultaneously. Concrete sources are automatically promoted to SourceJobNode declarations in the underlying graph, with their concrete instances stored in job.sources.

After the with block, job.pipeline returns a lightweight Pipeline blueprint (SourceNode-only leaves). job.run() executes the resolvable subgraph — nodes whose upstream SourceNodes are all bound.

PipelineJob can also be created from a Pipeline via PipelineJob.from_pipeline(pipeline, sources=..., database=...) for the "explicit blueprint" workflow.

Parameters:

  • name (str | tuple[str, ...]) default 'pipeline': Pipeline name (string or tuple). Used as the path prefix for all cache/pipeline paths when the pipeline is run.
  • database ('ArrowDatabaseProtocol | None') default None: Database for result caching and operator records.
  • execution_context ('ExecutionContext | None') default None: Optional execution configuration.
  • tracker_manager (cp.TrackerManagerProtocol | None) default None: Optional tracker manager override.
  • auto_compile (bool) default True: If True (default), compile() is called automatically when the context manager exits.

method · orcapod.pipeline.job.PipelineJob.compile

def compile() -> None

Compile recorded invocations into a frozen DAG of job nodes.

Delegates to AbstractPipelineBase.compile() which uses source_node_class, function_node_class, and operator_node_class to create job node instances.

Concrete source bindings already in _sources are preserved. Any SourceJobNode in _persistent_node_map that has a bound source also contributes to _sources (covers both the direct-recording path and the bind()-before-compile path).

The lazy compiled_pipeline cache is invalidated.

If _database is set, distributes databases to all function and operator job nodes.

property · orcapod.pipeline.job.PipelineJob.last_run

Type: RunSummary | None

The summary of the last run this job completed, or None before one.

The same object run and run_async returned: per function node, the rows computed, read from the database and failed, and each node's first error. Use job.last_run.print_traceback("<node label>") to see a failed row's full traceback. A run that raised leaves it unchanged. A loaded job has none.

property · orcapod.pipeline.job.PipelineJob.compiled_pipeline

Type: 'Pipeline'

The lightweight Pipeline blueprint for this job (lazy cache).

Computed on first access by calling as_pipeline(). Invalidated by compile(), which sets _compiled_pipeline to None so the next access recomputes a fresh schema-normalised blueprint.

bind() does not invalidate this cache: the blueprint contains only SourceNode schema-slot declarations with no concrete data, so it is unaffected by changes to concrete source bindings.

Raises:

  • RuntimeError: If the job has not been compiled yet.

property · orcapod.pipeline.job.PipelineJob.pipeline

Type: 'Pipeline'

The compiled pure Pipeline (SourceNode-only leaves).

Raises:

  • RuntimeError: If the with-block has not been completed yet.

property · orcapod.pipeline.job.PipelineJob.sources

Type: dict[str, cp.StreamProtocol]

Mapping of SourceNode name to bound concrete source.

property · orcapod.pipeline.job.PipelineJob.database

Type: 'ArrowDatabaseProtocol | None'

The database used for result caching.

property · orcapod.pipeline.job.PipelineJob.execution_context

Type: 'ExecutionContext | None'

The execution configuration, or None if unset.

classmethod · orcapod.pipeline.job.PipelineJob.from_pipeline

def from_pipeline(
pipeline: 'Pipeline',
database: 'ArrowDatabaseProtocol | None' = None,
sources: 'dict[str, cp.StreamProtocol] | None' = None,
execution_context: 'ExecutionContext | None' = None,
) -> 'PipelineJob'

Create a runnable PipelineJob from a compiled Pipeline.

Converts the Pipeline blueprint to an InvocationGraph via pipeline.to_invocations(), then reconstructs a PipelineJob using from_invocations() (which calls SourceJobNode.from_stream for each source, creating unbound job nodes). If sources are provided, validates and binds them.

Parameters:

  • pipeline ('Pipeline'): A compiled Pipeline (pipeline._compiled must be True).
  • database ('ArrowDatabaseProtocol | None') default None: Database for result caching and operator records.
  • sources ('dict[str, cp.StreamProtocol] | None') default None: Mapping of SourceNode.name → concrete source.
  • execution_context ('ExecutionContext | None') default None: Optional execution configuration.

Returns:

  • 'PipelineJob': A new PipelineJob ready to run (or bind() further).

Raises:

  • ValueError: If pipeline has not been compiled.

method · orcapod.pipeline.job.PipelineJob.bind

def bind(
sources: 'dict[str, cp.StreamProtocol] | None' = None,
database: 'ArrowDatabaseProtocol | None' = None,
execution_context: 'ExecutionContext | None' = None,
) -> None

Update bindings in place. Returns None.

Mutating — modifies self directly. Existing bindings not mentioned in this call are preserved.

When sources is provided, each concrete source is validated against its matching SourceJobNode slot schema, then the corresponding SourceJobNode.bound_source is updated in-place. A source fits its slot when its tag and data columns have the declared names, roles and Python types, so a bind never changes a pipeline hash (ITL-1112). Every source is validated before anything is changed: a refused source leaves the job's bindings and database as they were.

When database is provided and differs from the current database, _distribute_databases() is called so that all job nodes receive live DB references immediately.

Parameters:

  • sources ('dict[str, cp.StreamProtocol] | None') default None: Mapping of SourceNode.name → concrete source.
  • database ('ArrowDatabaseProtocol | None') default None: Replaces the current database and triggers DB redistribution.
  • execution_context ('ExecutionContext | None') default None: Replaces the current execution context.

Raises:

  • SourceSpecMismatchError: If any source's columns differ from its slot's declared names, roles or types.
  • ValueError: If a source key has no matching SourceJobNode slot.
  • UnstorableColumnError: If database cannot store a table of a node (ITL-1162); the job is left as it was.

method · orcapod.pipeline.job.PipelineJob.apply_node_config

def apply_node_config(node_config: NodeConfig, replace_existing: bool = False) -> None

Apply a NodeConfig to all FunctionJobNodes in this pipeline.

Parameters:

  • node_config (NodeConfig): The config to apply.
  • replace_existing (bool) default False: If False, directly sets node_config on every node, replacing whatever config each node currently holds. If True, merges node_config into each node's existing config — non-None fields in node_config override the node's current values, while None fields leave existing values unchanged.

Raises:

  • RuntimeError: If the job has not been compiled yet (use with job: to record a DAG before applying node config).

method · orcapod.pipeline.job.PipelineJob.as_pipeline

def as_pipeline() -> 'Pipeline'

Return the lightweight Pipeline blueprint for this job.

Walks _persistent_node_map topologically and rewires each job node to a fresh blueprint node whose upstreams point at the already-built blueprint nodes in node_map (not at job nodes). This ensures that the content_hash of every node in the returned Pipeline is consistent with the hash keys in _persistent_node_map.

Returns:

  • 'Pipeline': A compiled Pipeline whose _persistent_node_map contains
  • 'Pipeline': only lightweight SourceNode / FunctionNode / OperatorNode
  • 'Pipeline': objects with blueprint (non-job) upstream references. Keys in
  • 'Pipeline': _persistent_node_map match those in self._persistent_node_map.

Raises:

  • RuntimeError: If this job has not been compiled.

method · orcapod.pipeline.job.PipelineJob.to_invocations

def to_invocations()

Extract an InvocationGraph from the schema-based blueprint.

Delegates to as_pipeline().to_invocations() so that the produced invocations always use schema-based (not data-based) hashes, regardless of whether sources are currently bound.

This ensures stable serialization across bind/rebind cycles.

Raises:

  • RuntimeError: If this job has not been compiled.

property · orcapod.pipeline.job.PipelineJob.unbound_sources

Type: list[str]

Names of source slots not yet bound in this job.

Computed live from _persistent_node_map — reflects the current binding state at all times (before and after run).

Returns:

  • list[str]: List of unbound source slot names, in order of appearance in
  • list[str]: the pipeline graph. Empty list if the job is not yet compiled.

method · orcapod.pipeline.job.PipelineJob.is_complete

def is_complete() -> bool

Return True when all source nodes are bound and a database is set.

Returns:

  • bool: True if all SourceNode slots are bound and a database is set.

method · orcapod.pipeline.job.PipelineJob.is_runnable

def is_runnable(node_label: str) -> bool

Return whether a live job has all upstream inputs of node_label resolved.

A loaded job is read-only and always returns False.

Uses hash-based traversal of _hash_graph combined with _persistent_node_map to check whether every upstream source slot is bound. This avoids relying on the object-identity graph (_node_graph), which breaks after bind() because SourceJobNode.__hash__() changes when bound_source is mutated.

Parameters:

  • node_label (str): Label of the node to check.

Returns:

  • bool: True if the node can be executed with current bindings.

method · orcapod.pipeline.job.PipelineJob.run

def run(
orchestrator: 'SyncPipelineOrchestrator | AsyncPipelineOrchestrator | None' = None,
observer: 'ExecutionObserverProtocol | None' = None,
materialize_results: bool = True,
commit_every_rows: int = 100,
*,
load_only: bool = False,
) -> RunSummary

Execute the resolvable subgraph of this job in place.

Nodes whose upstream includes an unbound source (and all their dependents) are excluded from execution. Partial execution is a first-class outcome — unbound_sources reports which source slots were excluded.

After a successful run, job.nodes returns the execution-ready nodes with populated database caches.

State-driven, as every job node's run() (developer, November 14, 2026; ITL-1095): a node that holds a current run (completed in this process, not stale, and for a full run not filled by a load-only run) is held, not run; its rows feed the nodes after it, and nothing is computed, read, written or delivered for it. A source is read only for the nodes that run. So a second run of an unchanged job does nothing. To run a node again, call clear_cache() on it (every node below it runs too), or bind a new source object.

job.run(orchestrator=AsyncPipelineOrchestrator()) works the same in a script and inside a running event loop, such as a Jupyter kernel: there the run goes on a separate thread while the call waits, and blocks that loop until it ends. From async code, await job.run_async() instead.

Parameters:

  • orchestrator ('SyncPipelineOrchestrator | AsyncPipelineOrchestrator | None') default None: Orchestrator to use for execution. Defaults to SyncPipelineOrchestrator when None.
  • observer ('ExecutionObserverProtocol | None') default None: Optional execution observer.
  • materialize_results (bool) default True: Controls whether the orchestrator retains node outputs in its transient in-process buffer during execution. When True (the default) the buffer is kept for the duration of the run. When False each node's buffer is released as soon as the node completes, reducing peak memory. This setting does not affect database persistence or post-run result access: DB records are written regardless, and job.nodes["label"] returns the same rows from as_table() and iter_data() in either case, on either orchestrator. An operator node with the default cache_mode=OFF (the only mode a job sets) keeps its output in memory and stores nothing. Its get_all_records() returns None when no matching records exist; records from an earlier LOG run remain readable after switching to OFF.
  • commit_every_rows (int) default 100: How many rows a function node processes between commits of its results and pipeline records to the database (default 100). Each node also commits when it finishes, before a failed run raises, and the job commits at its end. On a durable database (DeltaTableDatabase) a hard crash (power loss, killed process) loses at most the rows of the running node since its last commit; re-running the job recomputes only those. A larger value commits less often (faster, fewer files); 1 commits after every row. A node run on its own, outside run(), commits after every row. It must be smaller than the database's batch_size when the database has one (DeltaTableDatabase, default 1000): the database commits every pending table by itself once one table's pending rows reach that size, which would break the node's results-before-records commit order.
  • load_only (bool) default False: Fill the nodes from what is already stored and compute nothing (ITL-1095). Sources are read and operators run on the rows that reach them; each function node takes the results stored for this run's inputs (its own records under its content hash) and skips the inputs without one, which the summary counts (not_stored); a function node forwards only the rows it loaded. No user function is called, no side effect is delivered (a side-effect node passes its rows on) and nothing is written to the database, a LOG operator's cache included. Afterwards each node shows the rows it loaded.

Rows whose function raised (with the default error_policy="continue") are skipped, and the run goes on. At the end, if any row failed, a single RowFailuresWarning names each node with failed rows, how many failed, the first error, and how to print its traceback. A run without failures is quiet.

Returns:

  • RunSummary: The run's RunSummary: per function node, the rows computed, read from
  • RunSummary: the database and failed, with each node's first error. Shown at the end of a
  • RunSummary: notebook cell; also kept as job.last_run. Results are read from
  • RunSummary: job.nodes.

Raises:

  • ValueError: If no database is set, or commit_every_rows is not a positive integer smaller than the database's batch_size.
  • RuntimeError: If no pipeline has been compiled, or the job was built by PipelineJob.load (a loaded job is read-only).
  • OrcapodDatabaseWriteError: If a node's results or records could not be written to the database, whatever the orchestrator's error_policy. The run stops at the first failed write. Writes that succeeded before it are flushed first, so rows stored before the failure are kept, and re-running the job does not recompute stored function results. A failed commit of a node's batch names the node but no row (flush=True): the batch's writes stay pending and are retried once by the flush that follows the failure.
  • OrcapodDatabaseReadError: If a stored table exists but could not be read, whatever the orchestrator's error_policy, also when the read was needed to write a node's results. Its rows are not recomputed and the table is not overwritten.

method · orcapod.pipeline.job.PipelineJob.run_async

async def run_async(
orchestrator: 'AsyncPipelineOrchestrator | None' = None,
observer: 'ExecutionObserverProtocol | None' = None,
materialize_results: bool = True,
commit_every_rows: int = 100,
*,
load_only: bool = False,
) -> RunSummary

Execute the job with the async orchestrator, from async code.

The awaitable form of run: await job.run_async() runs on the caller's event loop instead of blocking it. job.run(orchestrator=AsyncPipelineOrchestrator()) also works inside a running loop (a Jupyter kernel), but blocks that loop until the run ends. Both store the same rows and record ids. Arguments, errors and result are those of run.

Parameters:

  • orchestrator ('AsyncPipelineOrchestrator | None') default None: The async orchestrator to run with. Defaults to AsyncPipelineOrchestrator().
  • observer ('ExecutionObserverProtocol | None') default None: Optional execution observer.
  • materialize_results (bool) default True: As for run.
  • commit_every_rows (int) default 100: As for run.
  • load_only (bool) default False: As for run.

Returns:

  • RunSummary: The run's RunSummary, as run returns.

Raises:

  • TypeError: If orchestrator is not an AsyncPipelineOrchestrator; run a SyncPipelineOrchestrator with job.run.
  • ValueError: As for run.
  • RuntimeError: As for run.
  • OrcapodDatabaseWriteError: As for run.
  • OrcapodDatabaseReadError: As for run.

method · orcapod.pipeline.job.PipelineJob.save

def save(path: 'str | Path') -> None

Serialize this job to a JSON file.

Saves topology (via the embedded Pipeline) plus bindings metadata and run state. The format covers both "template" (pre-run) and "completed" (post-run) states — distinguished by run.status.

Parameters:

  • path ('str | Path'): File path to write JSON output to.

Raises:

  • ValueError: If no compiled pipeline exists.

classmethod · orcapod.pipeline.job.PipelineJob.load

def load(
path: 'str | Path',
database: 'ArrowDatabaseProtocol | None' = None,
) -> 'PipelineJob'

Deserialize a PipelineJob from a JSON file to read its stored results.

A loaded job is for reading the results its saved run stored; it cannot be run again, because function pods are not saved with the job. Its nodes have not run in this process, so they are empty (ITL-1095): read a node's stored results through its stored view, job.nodes[label].as_source(), or with get_all_records(). Operators store outputs only when caching is enabled; the stored view of a loaded operator whose run's rows cannot be named raises RuntimeError.

The embedded Pipeline blueprint is always restored. Sources whose data lives in files (CSV, Delta) are rebuilt from the saved config. In-memory sources (DictSource, ListSource, ArrowTableSource, DataFrameSource) are not saved; loading logs that at INFO, and their source nodes can be read after job.bind(sources=...). A source that cannot be rebuilt for another reason (a moved file) is logged as a warning. Neither affects reading the other nodes.

Parameters:

  • path ('str | Path'): Path to the JSON file produced by save().
  • database ('ArrowDatabaseProtocol | None') default None: Optional database override. When provided, takes precedence over any database configuration in the file.

Returns:

  • 'PipelineJob': A read-only PipelineJob whose nodes' stored views serve their stored
  • 'PipelineJob': results.

Raises:

  • ValueError: If the file's format version is unsupported, or if it was saved before the database key was renamed (bindings.store instead of bindings.database) and no database is given. An explicit database makes the saved block irrelevant, so such a file then loads.

class · orcapod.pipeline.diagram.PipelineDiagram

class PipelineDiagram(
pipeline: _Drawable,
*,
direction: Direction = 'LR',
collapse: bool = False,
max_nodes: int = 400,
)

The DAG of a pipeline or job, drawn for a notebook.

In development: this interface may change.

Shown at the end of a notebook cell it is an SVG picture of the nodes (sources, operators, function pods, side-effect pods) and the streams between them. Where the notebook runs scripts, the picture can be dragged and zoomed, and selecting a node shows its tag and data schema and, for a job, its run status and the last run's row counts. Without scripts the picture and a table of the nodes remain. print gives a plain-text listing.

Making a diagram computes nothing: it reads each node's predicted output_schema(), and for a job what each node holds in memory and job.last_run.

Use Pipeline.diagram() or PipelineJob.diagram() to make one. The first diagram made in a process issues InDevelopmentWarning (once per process).

Parameters:

  • pipeline (_Drawable): A compiled Pipeline or PipelineJob.
  • direction (Direction) default 'LR': "LR" lays the graph out left to right, "TB" top to bottom.
  • collapse (bool) default False: Fold the operators that only choose, rename or join columns (joins, column selections, drops and maps) and feed exactly one node into the edges they lie on: such an edge is dashed, and the folded steps are listed in the details of the node it leads to.
  • max_nodes (int) default 400: Above this many nodes (after folding), the notebook shows the plain-text listing instead of the picture.

Raises:

  • ValueError: If direction is not "LR" or "TB", or max_nodes is not positive.
  • RuntimeError: If the pipeline has not been compiled.

property · orcapod.pipeline.diagram.PipelineDiagram.title

Type: str

Pipeline '<name>' or Job '<name>'.

In development: this interface may change.

method · orcapod.pipeline.diagram.PipelineDiagram.to_dict

def to_dict() -> dict[str, Any]

The diagram's model: plain data that any renderer can draw.

In development: this interface may change.

Its shape may change: it carries no version marker yet.

Returns:

  • dict[str, Any]: {"kind", "name", "title", "nodes", "edges"}. Each node has index,
  • dict[str, Any]: label, kind (its node_type), kind_text, detail,
  • dict[str, Any]: status ("" for a pipeline), tag_schema and data_schema
  • dict[str, Any]: (lists of [column, type text]) and facts (lists of [name, text]);
  • dict[str, Any]: each edge has source and target (node indices) and folded
  • dict[str, Any]: (the labels of the steps folded into it).

method · orcapod.pipeline.diagram.PipelineDiagram.to_text

def to_text() -> str

The nodes as plain text, one per line, each with its inputs.

In development: this interface may change.

Returns:

  • str: A title line, then <label> (<kind> · <detail> · <status>) <- <inputs>
  • str: per node, in topological order with ties broken by label.

method · orcapod.pipeline.diagram.PipelineDiagram.to_svg

def to_svg() -> str

The picture alone, as an SVG element.

In development: this interface may change.

Returns:

  • str: The <svg> element; it uses no style sheet and no script.

method · orcapod.pipeline.diagram.PipelineDiagram.to_html

def to_html() -> str

The HTML fragment a notebook shows: picture, node table, model and script.

In development: this interface may change.

Returns:

  • str: The fragment. Above max_nodes nodes, the plain-text listing in a
  • str: <pre> element instead.

method · orcapod.pipeline.diagram.PipelineDiagram.save

def save(path: str | Path) -> None

Write the diagram as a standalone HTML page, viewable in any browser offline.

In development: this interface may change.

Parameters:

  • path (str | Path): The file to write.