Streams
A stream is an immutable sequence of (Tag, Data) pairs backed by a PyArrow Table. Streams are the universal data currency in Orcapod -- every source produces a stream, every operator consumes and produces streams, and every function pod transforms data within a stream. Immutability guarantees that once a stream is created, its data cannot change, which is essential for reproducible pipelines.
Tag columns vs Data columns
Section titled “Tag columns vs Data columns”Every stream divides its columns into two groups:
Tag columns are join keys and metadata. They identify which record you are looking at
(e.g., subject_id, session_date). Operators like Join match rows across
streams using shared tag columns.
Data columns are the data payload. They hold the actual values being processed
(e.g., age, weight, spike_count). Function pods read data
columns as function inputs and write new data columns as outputs.
This separation is enforced throughout the framework:
- Operators inspect and restructure tags but never look inside data
- Function pods inspect and transform data but never look at tags
Key classes
Section titled “Key classes”ArrowTableStream
Section titled “ArrowTableStream”The primary stream implementation. Wraps a PyArrow Table with designated tag and data columns. Created internally by sources and operators -- you rarely construct one directly.
StreamBase
Section titled “StreamBase”Abstract base class providing the stream interface. Both ArrowTableStream and stream-like
objects (sources, function pod streams, nodes) inherit from it.
A stream must have at least one data column; a table made only of tag columns is rejected
with ValueError.
Core methods
Section titled “Core methods”Every stream exposes four key methods. The snippets below use this stream (a source is a stream):
from orcapod.sources import DictSource
stream = DictSource( data=[ {"subject_id": "mouse_01", "age": 12, "weight": 25.3}, {"subject_id": "mouse_02", "age": 8, "weight": 22.1}, ], tag_columns=["subject_id"], tag_only_columns=True,)output_schema()
Section titled “output_schema()”Returns the (tag_schema, data_schema) tuple describing column names and their Python types:
tag_schema, data_schema = stream.output_schema()print(dict(tag_schema)) # {'subject_id': <class 'str'>}print(dict(data_schema)) # {'age': <class 'int'>, 'weight': <class 'float'>}keys()
Section titled “keys()”Returns column names as (tag_keys, data_keys):
tag_keys, data_keys = stream.keys()# tag_keys = ('subject_id',)# data_keys = ('age', 'weight')iter_data()
Section titled “iter_data()”Iterates over (Tag, Data) pairs. A stream is itself iterable, and iterating it is the same as
calling iter_data(), so a loop is written for tag, data in stream:; the method is there
for when the iterator is passed on as a value (list(stream.iter_data())). Each Tag and Data
is an immutable datagram that you can inspect with .as_dict():
for tag, data in stream: print(tag.as_dict(), data.as_dict())# {'subject_id': 'mouse_01'} {'age': 12, 'weight': 25.3}# {'subject_id': 'mouse_02'} {'age': 8, 'weight': 22.1}as_table()
Section titled “as_table()”Returns the full stream as a PyArrow Table, which integrates with Pandas, Polars, and other Arrow-compatible tools:
table = stream.as_table()df = table.to_pandas()Streams also have direct shortcuts: as_pandas_df(), as_polars_df() and as_lazy_frame().
Looking at data
Section titled “Looking at data”In a notebook, a stream, source or job node shown at the end of a cell renders a short
table: its first rows in tag order, with a line saying what they are. That preview is
head() asked for sort_by_tags and rendered, and head(n, columns={"sort_by_tags": True})
gives the same rows to work with. Neither ever computes anything: they show
only what is readily available, never compute, poll or deliver a row (they read only what is held or stored, or make a bounded table read), never call your
code (a function, a predicate, a side effect) and never write to a database. run() computes
(see "Computing a stream outside a job" below).
With no rows to show, a head still has the columns and Arrow types the rows would have. To
learn them it applies each operator to empty (zero-row) tables, which processes no data: a
chain of N operators applies up to N operators to empty tables (about 60 ms for 20). A filter
is not applied at all, since its columns are its input's. A user-defined operator's
static_process is applied to zero-row tables and must not have effects. Two consequences: a pipeline whose
inputs disagree on a column's Arrow type (say int8 against int64) raises from head()
the same InputValidationError its run raises (the notebook preview shows it as "could not
read"), and a filter whose predicate does not fit its columns shows a table in head() and
raises when it runs.
from orcapod import function_podfrom orcapod.sources import DictSource
subjects = DictSource( data=[ {"subject_id": f"mouse_{i:02d}", "age": 8 + i, "weight": 20.0 + i} for i in range(1, 26) ], tag_columns=["subject_id"], tag_only_columns=True, label="subjects",)
head = subjects.head()print(head.header())# DictSource[subjects] — first 10 of 25 rowsprint(head.status.value, len(head))# rows 10print(subjects.head(3).as_table().column("age").to_pylist())# [9, 10, 11]head() returns a StreamView: the rows as a PyArrow table (.table, .as_table(),
.as_polars_df()), the row count when it is known without computing (.total_rows), and a
.status that says where the rows came from. A stream downstream of a computation shows rows
only once they are computed:
@function_pod(output_keys="age_months")def to_months(age: int) -> int: return age * 12
months = to_months.pod(subjects, label="months")print(months.head().header())# FunctionPodStream[months] — not computed; run() computes itprint(months.head().status.value, months.head().table.column_names)# not_computed ['subject_id', 'age_months']
view = months.run() # computes every rowprint(view.header())# FunctionPodStream[months] — all 25 computed rowsprint(months.head(3).header())# FunctionPodStream[months] — first 3 of 25 computed rowsComputing a stream outside a job: run()
Section titled “Computing a stream outside a job: run()”Every stream, source and node has run(), with no arguments: it is the one way to compute
any of them outside a job (developer, November 13, 2026). It does exactly the work
as_table() does, through the same cache, and returns a StreamView of every row with the
default columns. A stream already computed is not computed a second time; one whose input
changed is recomputed. In a notebook, stream.run() at the end of a cell computes and displays.
The view's own methods choose what is shown. Each returns a new view and never reads or computes anything; the header says which rows are shown:
print(view.head(5).header())# FunctionPodStream[months] — first 5 of 25 computed rowsprint(view.tail(5).header())# FunctionPodStream[months] — last 5 of 25 computed rowsprint(view.head(20).tail(10).header())# FunctionPodStream[months] — rows 11–20 of 25 computed rowsprint(view.sort_by_tags().head(3).header())# FunctionPodStream[months] — first 3 of 25 computed rows, sorted by tagsprint(view.head(3).sort_by_tags().header())# FunctionPodStream[months] — first 3 of 25 computed rows, then sorted by tagsview.sort_by_tags() orders rows as as_table(columns={"sort_by_tags": True}) does,
except for rows with equal tags: those are ordered by their system tags, which a run()
view does not hold, so it keeps them in the order it has them. The display of a view cut by
head(n) or tail(n) (on the stream or on a view) shows all its rows; a view nobody cut,
such as the one run() returns, shows its first and last five rows when it has more than
ten, and its header gives the count.
A view holds the default column groups only. For system tags, source info, the data context
or content hashes, ask the stream after run(). A function pod's or operator's output stream,
a job node (a job's side-effect node included), an in-memory source and a CachedSource keep
the rows run() gave: their as_table(columns=...) and head(columns=...) read those rows,
compute nothing and deliver no side effect. Other streams do their as_table() work again: a
side-effect stream outside a job delivers its side effects again (its head() shows no rows),
a polling source polls again and a SpiralDBTableSource reads its whole table again. A view never asks its stream for more columns, because for those
streams asking again is not a read.
What run() does on each kind of object:
| Object | run() |
|---|---|
An in-memory source (DictSource, ArrowTableSource, ...), a Delta or SQL table source |
nothing to compute: a view of the rows it holds |
A SpiralDBTableSource |
reads the whole table into memory (head() is the bounded read); every call reads again |
A PollingSource |
polls, as as_table() does |
A CachedSource |
reads its cache and writes the wrapped source's new rows to it |
| A function pod's or operator's output stream | computes what is not computed yet |
| A job node | runs the node as a job run would, after every job node it depends on that holds no current run (each once), commits, and returns the view of that run's rows; a node that holds a current run (completed in this process and not stale) runs nothing. run(load_only=True) loads stored results only. as_table() of a job node only reads |
| A job's side-effect node | runs as a job node does (row above): delivers what its invocation log does not record (every row again without track_completion), then returns the rows it passed on; a node that holds a current run delivers nothing (clear_cache() it first to deliver again); every read only reads |
| A side-effect stream outside a job | delivers its side effects for every row, on every call, and returns the rows it passed on; head() still shows none |
| A pipeline blueprint node, an unbound input slot | raises, as as_table() does |
What each kind of object shows:
| Object | head() shows |
|---|---|
| A source | its first rows (a bounded read for a source that reads its table lazily) |
| An operator's or function pod's output stream | the rows already computed, or none with status not_computed |
| A job node | the rows of its current run (after job.run() or node.run()); before a run in this process, or once stale, none (not_computed), pointing to run() and to node.as_source(), the view of every stored result |
| A pipeline blueprint node, an unbound input slot | none (unavailable) |
| A job's side-effect node | the rows its last run passed on, else none (not_computed); it never delivers, and neither do its as_table(), iter_data() and other reads |
| A side-effect stream outside a job | none: showing it never delivers its side effects |
head(n, columns=..., all_info=...) takes the same column configuration as as_table,
sorting included, and applies it to the rows it has before taking the first n. The default
does not sort, as as_table() does not, so head() shows rows in the order the stream
holds them. The notebook preview asks for the sorted head, so it shows the same rows
whatever order they were computed in. With sort_by_tags=True, or all_info=True, rows are
sorted by tag first (nulls last; rows with equal tags are ordered by their system tags):
from orcapod.sources import DictSource
unordered = DictSource( data=[{"subject_id": f"mouse_{i:02d}", "age": 8 + i} for i in (3, 1, 2)], tag_columns=["subject_id"], label="unordered",)print(unordered.head(2).as_table().column("subject_id").to_pylist())# ['mouse_03', 'mouse_01']print(unordered.head(2, all_info=True).as_table().column("subject_id").to_pylist())# ['mouse_01', 'mouse_02']A stream that has all of its rows (a job node after its run, a computed stream, an in-memory
source) then gives exactly the first n rows of as_table with the same configuration,
whatever order the rows were computed in. A stream that has only some of its rows sorts
those: a source that reads a bounded number of rows (such as a SpiralDBTableSource) shows
the rows it read, in tag order, and a stream computed only in part shows the rows computed so
far in tag order. It is not "the first n rows in tag order" of rows it never read, and the
preview's header line says so ("10 rows read, sorted by tags, of about 63,349"). A stream
with no tag columns previews its rows in stream order.
A head of an operator is never the operator applied to the heads of its inputs: a group over
the first ten rows has the wrong members. So an uncomputed stream shows its columns and
"not computed" until run(), as_table(), iter_data() or a job run computes it. An empty result
shows status rows with zero rows, so it is never mistaken for "not computed". A head without
rows because nothing is computed or stored yet shows the stream's declared columns: the
source-info (_source_*), _context_key and content-hash columns, which are known only from
rows, can be missing from it even when columns or all_info asks for them.
A stream or node records that its results are current when it has read its input to the end.
So after job.run() on either orchestrator, sync or async, every node shows the rows of that
run: head(), as_table() and iter_data() serve them from memory, without reading the
store. Before its run in this process a job node holds nothing and shows no rows, whatever
its database holds; node.as_source() reads the database. Likewise, once a chain of streams such as g.pod(f.pod(source)) has been read to the
end, reading it again computes nothing. A change upstream (clear_cache()) makes every stream
after it out of date, and its next read computes it again.
Two reads of one stream open at the same time, or a read that picks up one left partway, share
the stream's results: each row is computed once and every read yields the same rows, whatever
feeds the stream. An upstream finishing its first computation during the read is not a change;
a clear_cache() upstream between the reads is, and the next read starts again.
The display marks the tag columns, which come first: in a notebook their headers are bold
(in an accent colour), a vertical rule separates them from the data columns and a line under
the header says "bold = tag"; as text, each tag column's name starts with * ("* = tag").
Only the display is marked: the view's table and as_table() keep the plain names.
In the preview, large cells are summarised (an array as ndarray float64 (2, 3), bytes as
bytes (300 B), long text and long lists shortened); head().table keeps the full values.
In a terminal, print(stream.head()) prints the same preview as text.
Controlling column visibility with ColumnConfig
Section titled “Controlling column visibility with ColumnConfig”By default, streams only expose user-facing tag and data columns. Orcapod also maintains
hidden columns for provenance tracking and metadata. Use ColumnConfig (or the all_info
shortcut) to control which column groups are included.
| Config field | What it reveals | Column prefix |
|---|---|---|
meta |
System metadata columns, when the stream has any | __ |
system_tags |
System tag columns (provenance tracking) | _tag_source_id::, _tag_record_id:: |
source |
Source-info columns (per-data provenance tokens) | _source_ |
context |
Data context column | _context_key |
content_hash |
Content hash column | _content_hash |
sort_by_tags |
Sort rows by tag columns | (ordering only) |
Pass config as a dict or a ColumnConfig object:
from orcapod.sources import DictSource
source = DictSource( data=[ {"subject_id": "mouse_01", "age": 12, "weight": 25.3}, {"subject_id": "mouse_02", "age": 8, "weight": 22.1}, ], tag_columns=["subject_id"], tag_only_columns=True,)
# Default: user-facing columns onlytable = source.as_table()print(table.column_names)# ['subject_id', 'age', 'weight']
# Include source-info columnstable = source.as_table(columns={"source": True})print(table.column_names)# ['subject_id', 'age', 'weight', '_source_age', '_source_weight']
# Include everythingtable = source.as_table(all_info=True)print(table.column_names)# ['subject_id', 'age', 'weight', '_tag_source_id::...', '_tag_record_id::...', '_content_hash', '_context_key', '_source_age', '_source_weight']Code example
Section titled “Code example”Inspect a stream produced by a source:
from orcapod.sources import DictSource
source = DictSource( data=[ {"subject_id": "mouse_01", "age": 12, "weight": 25.3}, {"subject_id": "mouse_02", "age": 8, "weight": 22.1}, {"subject_id": "mouse_03", "age": 15, "weight": 27.8}, ], tag_columns=["subject_id"], tag_only_columns=True,)
# Schema inspectiontag_schema, data_schema = source.output_schema()print("Tag schema:", dict(tag_schema))# Tag schema: {'subject_id': <class 'str'>}print("Data schema:", dict(data_schema))# Data schema: {'age': <class 'int'>, 'weight': <class 'float'>}
# Iterate over (Tag, Data) pairsfor tag, data in source: print(f" {tag.as_dict()} -> {data.as_dict()}")# {'subject_id': 'mouse_01'} -> {'age': 12, 'weight': 25.3}# {'subject_id': 'mouse_02'} -> {'age': 8, 'weight': 22.1}# {'subject_id': 'mouse_03'} -> {'age': 15, 'weight': 27.8}
# Convert to a PyArrow table (interops with Pandas)table = source.as_table()print(table.to_pandas())# subject_id age weight# 0 mouse_01 12 25.3# 1 mouse_02 8 22.1# 2 mouse_03 15 27.8How it connects to other concepts
Section titled “How it connects to other concepts”- Sources produce streams from external data
- Operators consume one or more streams and produce a new stream
- Function Pods transform data values within a stream
- Every stream carries identity hashes for deduplication and DB-scoping