Streams
Streams are immutable sequences of (Tag, Data) pairs backed by PyArrow tables.
ArrowTableStream
Section titled “ArrowTableStream”class · orcapod.core.streams.arrow_table_stream.ArrowTableStream
class ArrowTableStream( table: 'pa.Table', tag_columns: Collection[str] = (), system_tag_columns: Collection[str] = (), source_info: dict[str, SourceInfoValue] | None = None, producer: PodProtocol | None = None, upstreams: tuple[StreamProtocol, ...] = (), data_tags: Collection[str] = (), **kwargs: Any,)Bases: StreamBase
An immutable stream based on a PyArrow Table. This stream is designed to be used with data that is already in a tabular format, such as data loaded from a file or database. The columns to be treated as tags are specified at initialization, and the rest of the columns are treated as data. The stream is immutable, meaning that once it is created, it cannot be modified. This is useful for ensuring that the data in the stream remains consistent and unchanging.
The types of the tag and data columns are inferred from the PyArrow Table schema.
Every column takes the large variant of its Arrow type (developer, November 14,
2026, ITL-1122): string is large_string, binary is large_binary,
list is large_list, recursively. This is the one place a table becomes a
stream, so every source (each builds its stream here, from a user's Arrow table
too) and every operator result presents large types. Width, nullability, field
metadata and logical types are kept; large and non-large variants hash identically.
data_tags
Section titled “data_tags”property · orcapod.core.streams.arrow_table_stream.ArrowTableStream.data_tags
Type: tuple[str, ...]
The data-tags: tag columns that are also data columns (tag order).
method · orcapod.core.streams.arrow_table_stream.ArrowTableStream.keys
def keys( *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False,) -> tuple[tuple[str, ...], tuple[str, ...]]Returns the keys of the tag and data columns in the stream. This is useful for accessing the columns in the stream.
output_schema
Section titled “output_schema”method · orcapod.core.streams.arrow_table_stream.ArrowTableStream.output_schema
def output_schema( *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False,) -> tuple[Schema, Schema]Returns the types of the tag and data columns in the stream. This is useful for accessing the types of the columns in the stream.
as_table
Section titled “as_table”method · orcapod.core.streams.arrow_table_stream.ArrowTableStream.as_table
def as_table( *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False,) -> 'pa.Table'Returns the underlying table representation of the stream. This is useful for converting the stream to a table format.
method · orcapod.core.streams.arrow_table_stream.ArrowTableStream.head
def head( n: int = 10, *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False,) -> StreamViewReturn the first n rows of the table this stream holds.
Without sorting only those rows are assembled: content hashes, when asked for,
are computed for them alone. With sort_by_tags (or all_info) the rows are
sorted first, as as_table sorts them, and the first n of those are
returned, so the head is always as_table(...) cut to n rows. See
StreamBase.head.
clear_cache
Section titled “clear_cache”method · orcapod.core.streams.arrow_table_stream.ArrowTableStream.clear_cache
def clear_cache() -> NoneResets the cached elements of the stream. This is useful for re-iterating over the stream.
iter_data
Section titled “iter_data”method · orcapod.core.streams.arrow_table_stream.ArrowTableStream.iter_data
def iter_data() -> Iterator[tuple[TagProtocol, Data]]Iterates over the rows of the stream as (Tag, Data) pairs.
Every row of the backing table is yielded exactly once, in table order, regardless of how the tag, data and source-info columns are chunked.
Yields:
tuple[TagProtocol, Data]: A(Tag, Data)tuple for each row.
StreamBase
Section titled “StreamBase”class · orcapod.core.streams.base.StreamBase
class StreamBase( label: str | None = None, data_context: str | contexts.DataContext | None = None, config: OrcapodConfig | None = None,)Bases: TraceableBase
is_stale
Section titled “is_stale”property · orcapod.core.streams.base.StreamBase.is_stale
Type: bool
True if any upstream stream or the source pod has a last_modified
timestamp strictly newer than this stream's own last_modified,
indicating that any in-memory cached content should be discarded and
repopulated.
Semantics:
- A
Nonetimestamp on this stream means "content not yet established" → always stale. - A
Nonetimestamp on an upstream or source means "modification time unknown" → conservatively treat as stale. - Immutable streams with no upstreams and no source (e.g.
ArrowTableStream) always returnFalse.
method · orcapod.core.streams.base.StreamBase.join
def join(other_stream: StreamProtocol, label: str | None = None) -> StreamBaseJoins this stream with another stream, returning a new stream that contains the combined data from both streams.
semi_join
Section titled “semi_join”method · orcapod.core.streams.base.StreamBase.semi_join
def semi_join(other_stream: StreamProtocol, label: str | None = None) -> StreamBasePerforms a semi-join with another stream, returning a new stream that contains only the data from this stream that have matching tags in the other stream.
map_tags
Section titled “map_tags”method · orcapod.core.streams.base.StreamBase.map_tags
def map_tags( name_map: Mapping[str, str] | None = None, /, *, drop_unmapped: bool = False, strict: bool = True, label: str | None = None, tag_only: bool = False, **renames: str,) -> StreamBaseRename tag columns. See orcapod.core.operators.MapTags.
The renames are one dict, the only positional argument (map_tags({"subject": "animal"})), or keyword arguments (map_tags(subject="animal")), not both.
A name that is not a Python identifier, or a column named drop_unmapped,
strict, label or tag_only (the options), is renamed through the
dict. A data-tag is renamed in both roles unless tag_only is
True; see MapTags.
Parameters:
- name_map (
Mapping[str, str] | None) defaultNone: Each tag column to rename, mapped to its new name. - drop_unmapped (
bool) defaultFalse: WhenTrue, the tags not inname_mapare dropped. - strict (
bool) defaultTrue: WhenTrue(the default), a key ofname_mapthat is not a tag column raisesInputValidationError; whenFalseit is ignored. - label (
str | None) defaultNone: Optional node label for the pipeline graph. - tag_only (
bool) defaultFalse: Rename data-tags in the tag role only (ITL-1023). - **renames (
str) default{}: Each column to rename, mapped to its new name, as keyword arguments.
Returns:
StreamBase: The stream with the tag columns renamed.
Raises:
TypeError: When both a dict and keyword renames are given, or a name is not a string.InputValidationError: When astrictname is not a tag column, or a new name collides with an existing column.
map_data
Section titled “map_data”method · orcapod.core.streams.base.StreamBase.map_data
def map_data( name_map: Mapping[str, str] | None = None, /, *, drop_unmapped: bool = False, strict: bool = True, label: str | None = None, **renames: str,) -> StreamBaseRename data columns. See orcapod.core.operators.MapData.
The renames are one dict, the only positional argument (map_data({"age": "age_weeks"})), or keyword arguments (map_data(age="age_weeks")), not
both. A name that is not a Python identifier, or a column named
drop_unmapped, strict or label (the options), is renamed through the
dict.
Parameters:
- name_map (
Mapping[str, str] | None) defaultNone: Each data column to rename, mapped to its new name. - drop_unmapped (
bool) defaultFalse: WhenTrue, the data columns not inname_mapare dropped. - strict (
bool) defaultTrue: WhenTrue(the default), a key ofname_mapthat is not a data column raisesInputValidationError; whenFalseit is ignored. - label (
str | None) defaultNone: Optional node label for the pipeline graph. - **renames (
str) default{}: Each column to rename, mapped to its new name, as keyword arguments.
Returns:
StreamBase: The stream with the data columns renamed.
Raises:
TypeError: When both a dict and keyword renames are given, or a name is not a string.InputValidationError: When astrictname is not a data column, a new name collides with an existing column, or no data column would remain.
map_columns
Section titled “map_columns”method · orcapod.core.streams.base.StreamBase.map_columns
def map_columns( name_map: Mapping[str, str] | None = None, /, *, drop_unmapped: bool = False, strict: bool = True, label: str | None = None, **renames: str,) -> StreamBaseRename columns, tag or data, in one step.
A renamed tag stays a tag and a renamed data column stays data. See
orcapod.core.operators.MapColumns.
The renames are one dict, the only positional argument (map_columns({"age": "age_weeks"})), or keyword arguments (map_columns(age="age_weeks")), not
both. A name that is not a Python identifier, or a column named
drop_unmapped, strict or label (the options), is renamed through the
dict.
Parameters:
- name_map (
Mapping[str, str] | None) defaultNone: Each column to rename, tag or data, mapped to its new name. - drop_unmapped (
bool) defaultFalse: WhenTrue, every tag and data column not inname_mapis dropped; system tags are kept. - strict (
bool) defaultTrue: WhenTrue(the default), a key ofname_mapthat is neither a tag nor a data column raisesInputValidationError; whenFalseit is ignored. - label (
str | None) defaultNone: Optional node label for the pipeline graph. - **renames (
str) default{}: Each column to rename, mapped to its new name, as keyword arguments.
Returns:
StreamBase: The stream with the columns renamed.
Raises:
TypeError: When both a dict and keyword renames are given, or a name is not a string.InputValidationError: When astrictname is missing, a new name collides with another column, or no data column would remain.
method · orcapod.core.streams.base.StreamBase.batch
def batch( batch_size: int = 0, drop_partial_batch: bool = False, label: str | None = None,) -> StreamBaseBatch stream into fixed-size chunks, each of size batch_size. If drop_last is True, any remaining elements that don't fit into a full batch will be dropped.
group_by
Section titled “group_by”method · orcapod.core.streams.base.StreamBase.group_by
def group_by(by: str | Collection[str], label: str | None = None) -> StreamBaseReduce rows sharing a key into one row per group.
Group-key columns stay scalar tags and are the only tags of the output;
every other tag is dropped, and every data column becomes list-valued.
A dropped tag's values are not carried: make the column data before
grouping if a later step needs it. See
orcapod.core.operators.GroupBy.
Parameters:
- by (
str | Collection[str]): Tag column names to group on, or one name as a string. - label (
str | None) defaultNone: Optional node label for the pipeline graph.
Returns:
StreamBase: A stream with one row per distinct group-key tuple.
drop_nulls
Section titled “drop_nulls”method · orcapod.core.streams.base.StreamBase.drop_nulls
def drop_nulls( columns: str | Collection[str] | None = None, label: str | None = None,) -> StreamBaseKeep the rows where every named column is non-null; narrow those columns to T.
Each named column, tag or data, is non-nullable afterwards (T
instead of T | None), so a pod whose parameter is a plain T
can consume it. Without names, every tag and data column of this
stream is tested and narrowed: a row with a null in any of them is
dropped. See orcapod.core.operators.DropNulls.
Parameters:
- columns (
str | Collection[str] | None) defaultNone: One column name, a collection of names, orNone(the default) for every tag and data column. An empty collection raisesValueError. - label (
str | None) defaultNone: Optional node label for the pipeline graph.
Returns:
StreamBase: A stream over the surviving rows, in input order.
tag_by
Section titled “tag_by”method · orcapod.core.streams.base.StreamBase.tag_by
def tag_by(*columns: str | Sequence[str], label: str | None = None) -> StreamBaseGive the named data columns the tag role, as data-tags.
s.tag_by("x", "y") equals s.tag_by(["x", "y"]). Each named data
column becomes a data-tag: one column, both a tag and the data column
it was, with its values and source info unchanged. The new tags follow this
stream's tags, in the order given. See orcapod.core.operators.TagBy.
Parameters:
- *columns (
str | Sequence[str]) default(): One or more data column names, or one list or tuple of names as the only positional argument (not a set: the order matters). - label (
str | None) defaultNone: Optional node label for the pipeline graph.
Returns:
StreamBase: The stream with the named columns as data-tags.
Raises:
TypeError: When no name is given, the names are a set, or a name is not a string.ValueError: When a name is repeated.InputValidationError: When a name is reserved, is not a data column of this stream, is already a tag, or has a type that cannot be a tag.
add_tags
Section titled “add_tags”method · orcapod.core.streams.base.StreamBase.add_tags
def add_tags( tags: Mapping[str, bool | int | float | str | bytes] | None = None, /, *, label: str | None = None, **values: bool | int | float | str | bytes,) -> StreamBaseAdd tags with constant values to every row. See orcapod.core.operators.AddTags.
The tags are one dict, the only positional argument (add_tags({"cohort": "A"})), or keyword arguments (add_tags(cohort="A", run=3)), not both. A
tag named label, or a name that is not a Python identifier, is given
through the dict. Each new tag holds the same value on every row and follows
the existing tags, in the order given. A value is a bool, int,
float, str or bytes, and its type is the tag's type. Data
columns, their values and their source info are unchanged.
Parameters:
- tags (
Mapping[str, bool | int | float | str | bytes] | None) defaultNone: Each new tag's name mapped to its value. - label (
str | None) defaultNone: Optional node label for the pipeline graph. - **values (
bool | int | float | str | bytes) default{}: Each new tag's name mapped to its value, as keyword arguments.
Returns:
StreamBase: The stream with the new tag columns.
Raises:
TypeError: When both a dict and keyword tags are given, a name is not a string, or a value's type is not accepted.ValueError: When no tag is given, or a value is out of range.InputValidationError: When a name is reserved, or already a tag or data column of this stream.
select_tag_columns
Section titled “select_tag_columns”method · orcapod.core.streams.base.StreamBase.select_tag_columns
def select_tag_columns( tag_columns: str | Collection[str], strict: bool = True, label: str | None = None,) -> StreamBaseSelect the specified tag columns from the stream. A ValueError is raised if one or more specified tag columns do not exist in the stream unless strict = False.
select_data_columns
Section titled “select_data_columns”method · orcapod.core.streams.base.StreamBase.select_data_columns
def select_data_columns( data_columns: str | Collection[str], strict: bool = True, label: str | None = None,) -> StreamBaseSelect the specified data columns from the stream. A ValueError is raised if one or more specified data columns do not exist in the stream unless strict = False.
drop_columns
Section titled “drop_columns”method · orcapod.core.streams.base.StreamBase.drop_columns
def drop_columns( *columns: str | Collection[str], strict: bool = True, label: str | None = None,) -> StreamBaseDrop the named columns, whether each is a tag or a data column.
s.drop_columns("a", "b") equals s.drop_columns(["a", "b"]). A
dropped data column's source-info column goes with it; system tags are
kept. See orcapod.core.operators.DropColumns.
Parameters:
- *columns (
str | Collection[str]) default(): One or more column names, or one collection of names as the only positional argument. - strict (
bool) defaultTrue: WhenTrue(the default), a name that is neither a tag nor a data column raisesInputValidationError; whenFalseit is ignored. - label (
str | None) defaultNone: Optional node label for the pipeline graph.
Returns:
StreamBase: The stream without the named columns.
Raises:
TypeError: When no name is given, a collection is mixed with further positional arguments, or a name is not a string.ValueError: When the one collection given is empty.InputValidationError: When astrictname is missing, or when no data column would remain.
select_columns
Section titled “select_columns”method · orcapod.core.streams.base.StreamBase.select_columns
def select_columns( *columns: str | Collection[str], strict: bool = True, label: str | None = None,) -> StreamBaseKeep exactly the named columns, tag or data; drop every other one.
s.select_columns("a", "b") equals s.select_columns(["a", "b"]).
Every user tag and data column not named is dropped, with the source-info
column of each dropped data column; system tags are kept. Naming no tag
is allowed. See orcapod.core.operators.SelectColumns.
Parameters:
- *columns (
str | Collection[str]) default(): One or more column names, or one collection of names as the only positional argument. - strict (
bool) defaultTrue: WhenTrue(the default), a name that is neither a tag nor a data column raisesInputValidationError; whenFalseit is ignored. - label (
str | None) defaultNone: Optional node label for the pipeline graph.
Returns:
StreamBase: The stream with only the named columns.
Raises:
TypeError: When no name is given, a collection is mixed with further positional arguments, or a name is not a string.ValueError: When the one collection given is empty.InputValidationError: When astrictname is missing, or when no data column is named.
method · orcapod.core.streams.base.StreamBase.pick
def pick( column: str, key: str, out: str | None = None, fail_on_miss: bool = False, label: str | None = None,) -> StreamBaseExtract a value from a struct- or dict-typed data column by key.
For dict[K, V] columns the lookup is per-packet; if the key is absent
the packet is skipped (or an error is raised when fail_on_miss=True).
Parameters:
- column (
str): Name of the data column to project into. - key (
str): Dict key or struct field name to extract. - out (
str | None) defaultNone: Output column name.None(default) replacescolumnin-place; a string adds a new column alongside the original. - fail_on_miss (
bool) defaultFalse: IfTrue, raiseRuntimeErrorwhen the key is absent in a packet instead of skipping. - label (
str | None) defaultNone: Optional label for the operator node.
Returns:
StreamBase: A new stream with the projected column.
method · orcapod.core.streams.base.StreamBase.index
def index( column: str, i: int, out: str | None = None, fail_on_miss: bool = False, label: str | None = None,) -> StreamBaseExtract an element from a list-typed data column by position.
Out-of-bounds access causes the packet to be skipped (or an error
when fail_on_miss=True). Negative indices follow Python semantics
(-1 is the last element).
Parameters:
- column (
str): Name of the data column to project into. - i (
int): Position to extract. Negative indices follow Python semantics. - out (
str | None) defaultNone: Output column name.None(default) replacescolumnin-place; a string adds a new column alongside the original. - fail_on_miss (
bool) defaultFalse: IfTrue, raiseRuntimeErroron out-of-bounds instead of skipping. - label (
str | None) defaultNone: Optional label for the operator node.
Returns:
StreamBase: A new stream with the projected column.
async_iter_data
Section titled “async_iter_data”method · orcapod.core.streams.base.StreamBase.async_iter_data
async def async_iter_data() -> AsyncIterator[tuple[TagProtocol, DataProtocol]]Async iterator over (tag, data) pairs.
Default implementation wraps iter_data as an async generator.
Subclasses override this to provide true async streaming behaviour
(e.g. PollingSource).
as_polars_df
Section titled “as_polars_df”method · orcapod.core.streams.base.StreamBase.as_polars_df
def as_polars_df( *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False,) -> 'pl.DataFrame'Convert the entire stream to a Polars DataFrame.
method · orcapod.core.streams.base.StreamBase.as_df
def as_df( *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False,) -> 'pl.DataFrame'Convert the entire stream to a Polars DataFrame.
as_lazy_frame
Section titled “as_lazy_frame”method · orcapod.core.streams.base.StreamBase.as_lazy_frame
def as_lazy_frame( *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False,) -> 'pl.LazyFrame'Convert the entire stream to a Polars LazyFrame.
method · orcapod.core.streams.base.StreamBase.flow
def flow() -> list[tuple[TagProtocol, DataProtocol]]Materialize the stream into a concrete collection of
(TagProtocol, DataProtocol) pairs.
This is implemented by iterating over iter_data. Depending on
the concrete stream implementation, iterating may trigger computation or
upstream work, or it may simply materialize already-computed results.
method · orcapod.core.streams.base.StreamBase.run
def run() -> StreamViewCompute the whole stream and return a StreamView of all its rows.
The one way to compute any stream outside a job (developer, November 13, 2026,
ITL-1081): head() and the notebook preview never compute a row (see
head). run() does
exactly the work as_table() does, through the same code path and cache, so
a stream already computed is not computed again, and one whose inputs changed is
computed again. The view holds every row with the default column configuration;
its head, tail and sort_by_tags choose what is shown. For other
column groups, use as_table(columns=...) or head(columns=...) after
run(). On a stream that keeps its rows (a pod's or operator's output stream,
a job node, a job's side-effect node included, an in-memory source) they read the
rows run() gave and deliver no side effect; a stream that keeps none does its
as_table() work again (a side-effect stream outside a job delivers again, a
polling source polls again).
Returns:
StreamView: AStreamViewof all the rowsas_table()returns, with statusStreamView:ROWS.
method · orcapod.core.streams.base.StreamBase.head
def head( n: int = 10, *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False,) -> StreamViewReturn at most n rows that are readily available, without computing.
The rows are ones the stream already holds, has cached from an earlier
computation, has stored, or can read with a bounded read of its table. head
never computes, polls or delivers a row (it reads only what is held or stored,
or makes a bounded table read), never calls user code (a
function, a predicate, a side effect) and never writes. With no rows to show, it
learns the column types without data (ITL-912): each operator of the stream is
applied to zero-row inputs, which processes no data, so a chain of N
operators applies up to N operators to empty tables (about 60 ms for 20); an
operator whose output schema is its input's (OUTPUT_SCHEMA_IS_INPUT_SCHEMA,
such as PolarsFilter) is not applied at all. A user-defined operator's
static_process is applied to zero-row tables and must not have effects. So a pipeline whose inputs
disagree on a column's Arrow type raises from head the
InputValidationError its run raises, and a filter with an ill-typed
predicate shows its input's columns: the error appears at the run. head takes the column configuration of as_table,
sorting included, and applies it to the rows the stream has, then takes the first
n. The default configuration does not sort, as as_table() does not, so
the rows are the first n that as_table() would return. With
sort_by_tags (or all_info) a stream that has all of its rows (a node's
stored slice, a computed stream, an in-memory source) returns the first n
of as_table with that configuration. A stream that has read or computed only
part of its rows (a source's bounded read, a partly computed stream) sorts those
rows: the rows read, in tag order, not the first n in tag order. The
notebook preview of a stream is its head() asked for sort_by_tags.
This default shows no rows: a stream class that can serve rows without computing
overrides it. The zero-row table then has the stream's columns, typed from its
declared (Python) schema, so a column's Arrow width can differ from the one the
computed stream will have (int shows as int64). That zero-row table has
the declared columns only: never a _source_*, _context_key or
content-hash column, whatever columns asks for, because whether such a column
exists and, for _source_*, its type are known only from rows.
Parameters:
- n (
int) default10: The largest number of rows to return. - columns (
ColumnConfig| dict[str, Any] | None) defaultNone: Column groups to include and whether to sort by tags, exactly as foras_table. - all_info (
bool) defaultFalse: Include every column group and sort by tags, as foras_table.
Returns:
StreamView: AStreamViewwith the rows and astatusthat tells an empty resultStreamView: from one that is not computed.
Raises:
TypeError: Ifnis not anint.ValueError: Ifnis negative.