Skip to content
This site documents the development version of Orcapod (the main branch), which may differ from the latest release. For the releases, see PyPI and the changelog.

Streams

Streams are immutable sequences of (Tag, Data) pairs backed by PyArrow tables.

class · orcapod.core.streams.arrow_table_stream.ArrowTableStream

class ArrowTableStream(
table: 'pa.Table',
tag_columns: Collection[str] = (),
system_tag_columns: Collection[str] = (),
source_info: dict[str, SourceInfoValue] | None = None,
producer: PodProtocol | None = None,
upstreams: tuple[StreamProtocol, ...] = (),
data_tags: Collection[str] = (),
**kwargs: Any,
)

Bases: StreamBase

An immutable stream based on a PyArrow Table. This stream is designed to be used with data that is already in a tabular format, such as data loaded from a file or database. The columns to be treated as tags are specified at initialization, and the rest of the columns are treated as data. The stream is immutable, meaning that once it is created, it cannot be modified. This is useful for ensuring that the data in the stream remains consistent and unchanging.

The types of the tag and data columns are inferred from the PyArrow Table schema.

Every column takes the large variant of its Arrow type (developer, November 14, 2026, ITL-1122): string is large_string, binary is large_binary, list is large_list, recursively. This is the one place a table becomes a stream, so every source (each builds its stream here, from a user's Arrow table too) and every operator result presents large types. Width, nullability, field metadata and logical types are kept; large and non-large variants hash identically.

property · orcapod.core.streams.arrow_table_stream.ArrowTableStream.data_tags

Type: tuple[str, ...]

The data-tags: tag columns that are also data columns (tag order).

method · orcapod.core.streams.arrow_table_stream.ArrowTableStream.keys

def keys(
*,
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> tuple[tuple[str, ...], tuple[str, ...]]

Returns the keys of the tag and data columns in the stream. This is useful for accessing the columns in the stream.

method · orcapod.core.streams.arrow_table_stream.ArrowTableStream.output_schema

def output_schema(
*,
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> tuple[Schema, Schema]

Returns the types of the tag and data columns in the stream. This is useful for accessing the types of the columns in the stream.

method · orcapod.core.streams.arrow_table_stream.ArrowTableStream.as_table

def as_table(
*,
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> 'pa.Table'

Returns the underlying table representation of the stream. This is useful for converting the stream to a table format.

method · orcapod.core.streams.arrow_table_stream.ArrowTableStream.head

def head(
n: int = 10,
*,
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> StreamView

Return the first n rows of the table this stream holds.

Without sorting only those rows are assembled: content hashes, when asked for, are computed for them alone. With sort_by_tags (or all_info) the rows are sorted first, as as_table sorts them, and the first n of those are returned, so the head is always as_table(...) cut to n rows. See StreamBase.head.

method · orcapod.core.streams.arrow_table_stream.ArrowTableStream.clear_cache

def clear_cache() -> None

Resets the cached elements of the stream. This is useful for re-iterating over the stream.

method · orcapod.core.streams.arrow_table_stream.ArrowTableStream.iter_data

def iter_data() -> Iterator[tuple[TagProtocol, Data]]

Iterates over the rows of the stream as (Tag, Data) pairs.

Every row of the backing table is yielded exactly once, in table order, regardless of how the tag, data and source-info columns are chunked.

Yields:

  • tuple[TagProtocol, Data]: A (Tag, Data) tuple for each row.

class · orcapod.core.streams.base.StreamBase

class StreamBase(
label: str | None = None,
data_context: str | contexts.DataContext | None = None,
config: OrcapodConfig | None = None,
)

Bases: TraceableBase

property · orcapod.core.streams.base.StreamBase.is_stale

Type: bool

True if any upstream stream or the source pod has a last_modified timestamp strictly newer than this stream's own last_modified, indicating that any in-memory cached content should be discarded and repopulated.

Semantics:

  • A None timestamp on this stream means "content not yet established" → always stale.
  • A None timestamp on an upstream or source means "modification time unknown" → conservatively treat as stale.
  • Immutable streams with no upstreams and no source (e.g. ArrowTableStream) always return False.

method · orcapod.core.streams.base.StreamBase.join

def join(other_stream: StreamProtocol, label: str | None = None) -> StreamBase

Joins this stream with another stream, returning a new stream that contains the combined data from both streams.

method · orcapod.core.streams.base.StreamBase.semi_join

def semi_join(other_stream: StreamProtocol, label: str | None = None) -> StreamBase

Performs a semi-join with another stream, returning a new stream that contains only the data from this stream that have matching tags in the other stream.

method · orcapod.core.streams.base.StreamBase.map_tags

def map_tags(
name_map: Mapping[str, str] | None = None,
/,
*,
drop_unmapped: bool = False,
strict: bool = True,
label: str | None = None,
tag_only: bool = False,
**renames: str,
) -> StreamBase

Rename tag columns. See orcapod.core.operators.MapTags.

The renames are one dict, the only positional argument (map_tags({"subject": "animal"})), or keyword arguments (map_tags(subject="animal")), not both. A name that is not a Python identifier, or a column named drop_unmapped, strict, label or tag_only (the options), is renamed through the dict. A data-tag is renamed in both roles unless tag_only is True; see MapTags.

Parameters:

  • name_map (Mapping[str, str] | None) default None: Each tag column to rename, mapped to its new name.
  • drop_unmapped (bool) default False: When True, the tags not in name_map are dropped.
  • strict (bool) default True: When True (the default), a key of name_map that is not a tag column raises InputValidationError; when False it is ignored.
  • label (str | None) default None: Optional node label for the pipeline graph.
  • tag_only (bool) default False: Rename data-tags in the tag role only (ITL-1023).
  • **renames (str) default {}: Each column to rename, mapped to its new name, as keyword arguments.

Returns:

  • StreamBase: The stream with the tag columns renamed.

Raises:

  • TypeError: When both a dict and keyword renames are given, or a name is not a string.
  • InputValidationError: When a strict name is not a tag column, or a new name collides with an existing column.

method · orcapod.core.streams.base.StreamBase.map_data

def map_data(
name_map: Mapping[str, str] | None = None,
/,
*,
drop_unmapped: bool = False,
strict: bool = True,
label: str | None = None,
**renames: str,
) -> StreamBase

Rename data columns. See orcapod.core.operators.MapData.

The renames are one dict, the only positional argument (map_data({"age": "age_weeks"})), or keyword arguments (map_data(age="age_weeks")), not both. A name that is not a Python identifier, or a column named drop_unmapped, strict or label (the options), is renamed through the dict.

Parameters:

  • name_map (Mapping[str, str] | None) default None: Each data column to rename, mapped to its new name.
  • drop_unmapped (bool) default False: When True, the data columns not in name_map are dropped.
  • strict (bool) default True: When True (the default), a key of name_map that is not a data column raises InputValidationError; when False it is ignored.
  • label (str | None) default None: Optional node label for the pipeline graph.
  • **renames (str) default {}: Each column to rename, mapped to its new name, as keyword arguments.

Returns:

  • StreamBase: The stream with the data columns renamed.

Raises:

  • TypeError: When both a dict and keyword renames are given, or a name is not a string.
  • InputValidationError: When a strict name is not a data column, a new name collides with an existing column, or no data column would remain.

method · orcapod.core.streams.base.StreamBase.map_columns

def map_columns(
name_map: Mapping[str, str] | None = None,
/,
*,
drop_unmapped: bool = False,
strict: bool = True,
label: str | None = None,
**renames: str,
) -> StreamBase

Rename columns, tag or data, in one step.

A renamed tag stays a tag and a renamed data column stays data. See orcapod.core.operators.MapColumns.

The renames are one dict, the only positional argument (map_columns({"age": "age_weeks"})), or keyword arguments (map_columns(age="age_weeks")), not both. A name that is not a Python identifier, or a column named drop_unmapped, strict or label (the options), is renamed through the dict.

Parameters:

  • name_map (Mapping[str, str] | None) default None: Each column to rename, tag or data, mapped to its new name.
  • drop_unmapped (bool) default False: When True, every tag and data column not in name_map is dropped; system tags are kept.
  • strict (bool) default True: When True (the default), a key of name_map that is neither a tag nor a data column raises InputValidationError; when False it is ignored.
  • label (str | None) default None: Optional node label for the pipeline graph.
  • **renames (str) default {}: Each column to rename, mapped to its new name, as keyword arguments.

Returns:

  • StreamBase: The stream with the columns renamed.

Raises:

  • TypeError: When both a dict and keyword renames are given, or a name is not a string.
  • InputValidationError: When a strict name is missing, a new name collides with another column, or no data column would remain.

method · orcapod.core.streams.base.StreamBase.batch

def batch(
batch_size: int = 0,
drop_partial_batch: bool = False,
label: str | None = None,
) -> StreamBase

Batch stream into fixed-size chunks, each of size batch_size. If drop_last is True, any remaining elements that don't fit into a full batch will be dropped.

method · orcapod.core.streams.base.StreamBase.group_by

def group_by(by: str | Collection[str], label: str | None = None) -> StreamBase

Reduce rows sharing a key into one row per group.

Group-key columns stay scalar tags and are the only tags of the output; every other tag is dropped, and every data column becomes list-valued. A dropped tag's values are not carried: make the column data before grouping if a later step needs it. See orcapod.core.operators.GroupBy.

Parameters:

  • by (str | Collection[str]): Tag column names to group on, or one name as a string.
  • label (str | None) default None: Optional node label for the pipeline graph.

Returns:

  • StreamBase: A stream with one row per distinct group-key tuple.

method · orcapod.core.streams.base.StreamBase.drop_nulls

def drop_nulls(
columns: str | Collection[str] | None = None,
label: str | None = None,
) -> StreamBase

Keep the rows where every named column is non-null; narrow those columns to T.

Each named column, tag or data, is non-nullable afterwards (T instead of T | None), so a pod whose parameter is a plain T can consume it. Without names, every tag and data column of this stream is tested and narrowed: a row with a null in any of them is dropped. See orcapod.core.operators.DropNulls.

Parameters:

  • columns (str | Collection[str] | None) default None: One column name, a collection of names, or None (the default) for every tag and data column. An empty collection raises ValueError.
  • label (str | None) default None: Optional node label for the pipeline graph.

Returns:

  • StreamBase: A stream over the surviving rows, in input order.

method · orcapod.core.streams.base.StreamBase.tag_by

def tag_by(*columns: str | Sequence[str], label: str | None = None) -> StreamBase

Give the named data columns the tag role, as data-tags.

s.tag_by("x", "y") equals s.tag_by(["x", "y"]). Each named data column becomes a data-tag: one column, both a tag and the data column it was, with its values and source info unchanged. The new tags follow this stream's tags, in the order given. See orcapod.core.operators.TagBy.

Parameters:

  • *columns (str | Sequence[str]) default (): One or more data column names, or one list or tuple of names as the only positional argument (not a set: the order matters).
  • label (str | None) default None: Optional node label for the pipeline graph.

Returns:

  • StreamBase: The stream with the named columns as data-tags.

Raises:

  • TypeError: When no name is given, the names are a set, or a name is not a string.
  • ValueError: When a name is repeated.
  • InputValidationError: When a name is reserved, is not a data column of this stream, is already a tag, or has a type that cannot be a tag.

method · orcapod.core.streams.base.StreamBase.add_tags

def add_tags(
tags: Mapping[str, bool | int | float | str | bytes] | None = None,
/,
*,
label: str | None = None,
**values: bool | int | float | str | bytes,
) -> StreamBase

Add tags with constant values to every row. See orcapod.core.operators.AddTags.

The tags are one dict, the only positional argument (add_tags({"cohort": "A"})), or keyword arguments (add_tags(cohort="A", run=3)), not both. A tag named label, or a name that is not a Python identifier, is given through the dict. Each new tag holds the same value on every row and follows the existing tags, in the order given. A value is a bool, int, float, str or bytes, and its type is the tag's type. Data columns, their values and their source info are unchanged.

Parameters:

  • tags (Mapping[str, bool | int | float | str | bytes] | None) default None: Each new tag's name mapped to its value.
  • label (str | None) default None: Optional node label for the pipeline graph.
  • **values (bool | int | float | str | bytes) default {}: Each new tag's name mapped to its value, as keyword arguments.

Returns:

  • StreamBase: The stream with the new tag columns.

Raises:

  • TypeError: When both a dict and keyword tags are given, a name is not a string, or a value's type is not accepted.
  • ValueError: When no tag is given, or a value is out of range.
  • InputValidationError: When a name is reserved, or already a tag or data column of this stream.

method · orcapod.core.streams.base.StreamBase.select_tag_columns

def select_tag_columns(
tag_columns: str | Collection[str],
strict: bool = True,
label: str | None = None,
) -> StreamBase

Select the specified tag columns from the stream. A ValueError is raised if one or more specified tag columns do not exist in the stream unless strict = False.

method · orcapod.core.streams.base.StreamBase.select_data_columns

def select_data_columns(
data_columns: str | Collection[str],
strict: bool = True,
label: str | None = None,
) -> StreamBase

Select the specified data columns from the stream. A ValueError is raised if one or more specified data columns do not exist in the stream unless strict = False.

method · orcapod.core.streams.base.StreamBase.drop_columns

def drop_columns(
*columns: str | Collection[str],
strict: bool = True,
label: str | None = None,
) -> StreamBase

Drop the named columns, whether each is a tag or a data column.

s.drop_columns("a", "b") equals s.drop_columns(["a", "b"]). A dropped data column's source-info column goes with it; system tags are kept. See orcapod.core.operators.DropColumns.

Parameters:

  • *columns (str | Collection[str]) default (): One or more column names, or one collection of names as the only positional argument.
  • strict (bool) default True: When True (the default), a name that is neither a tag nor a data column raises InputValidationError; when False it is ignored.
  • label (str | None) default None: Optional node label for the pipeline graph.

Returns:

  • StreamBase: The stream without the named columns.

Raises:

  • TypeError: When no name is given, a collection is mixed with further positional arguments, or a name is not a string.
  • ValueError: When the one collection given is empty.
  • InputValidationError: When a strict name is missing, or when no data column would remain.

method · orcapod.core.streams.base.StreamBase.select_columns

def select_columns(
*columns: str | Collection[str],
strict: bool = True,
label: str | None = None,
) -> StreamBase

Keep exactly the named columns, tag or data; drop every other one.

s.select_columns("a", "b") equals s.select_columns(["a", "b"]). Every user tag and data column not named is dropped, with the source-info column of each dropped data column; system tags are kept. Naming no tag is allowed. See orcapod.core.operators.SelectColumns.

Parameters:

  • *columns (str | Collection[str]) default (): One or more column names, or one collection of names as the only positional argument.
  • strict (bool) default True: When True (the default), a name that is neither a tag nor a data column raises InputValidationError; when False it is ignored.
  • label (str | None) default None: Optional node label for the pipeline graph.

Returns:

  • StreamBase: The stream with only the named columns.

Raises:

  • TypeError: When no name is given, a collection is mixed with further positional arguments, or a name is not a string.
  • ValueError: When the one collection given is empty.
  • InputValidationError: When a strict name is missing, or when no data column is named.

method · orcapod.core.streams.base.StreamBase.pick

def pick(
column: str,
key: str,
out: str | None = None,
fail_on_miss: bool = False,
label: str | None = None,
) -> StreamBase

Extract a value from a struct- or dict-typed data column by key.

For dict[K, V] columns the lookup is per-packet; if the key is absent the packet is skipped (or an error is raised when fail_on_miss=True).

Parameters:

  • column (str): Name of the data column to project into.
  • key (str): Dict key or struct field name to extract.
  • out (str | None) default None: Output column name. None (default) replaces column in-place; a string adds a new column alongside the original.
  • fail_on_miss (bool) default False: If True, raise RuntimeError when the key is absent in a packet instead of skipping.
  • label (str | None) default None: Optional label for the operator node.

Returns:

  • StreamBase: A new stream with the projected column.

method · orcapod.core.streams.base.StreamBase.index

def index(
column: str,
i: int,
out: str | None = None,
fail_on_miss: bool = False,
label: str | None = None,
) -> StreamBase

Extract an element from a list-typed data column by position.

Out-of-bounds access causes the packet to be skipped (or an error when fail_on_miss=True). Negative indices follow Python semantics (-1 is the last element).

Parameters:

  • column (str): Name of the data column to project into.
  • i (int): Position to extract. Negative indices follow Python semantics.
  • out (str | None) default None: Output column name. None (default) replaces column in-place; a string adds a new column alongside the original.
  • fail_on_miss (bool) default False: If True, raise RuntimeError on out-of-bounds instead of skipping.
  • label (str | None) default None: Optional label for the operator node.

Returns:

  • StreamBase: A new stream with the projected column.

method · orcapod.core.streams.base.StreamBase.async_iter_data

async def async_iter_data() -> AsyncIterator[tuple[TagProtocol, DataProtocol]]

Async iterator over (tag, data) pairs.

Default implementation wraps iter_data as an async generator. Subclasses override this to provide true async streaming behaviour (e.g. PollingSource).

method · orcapod.core.streams.base.StreamBase.as_polars_df

def as_polars_df(
*,
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> 'pl.DataFrame'

Convert the entire stream to a Polars DataFrame.

method · orcapod.core.streams.base.StreamBase.as_df

def as_df(
*,
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> 'pl.DataFrame'

Convert the entire stream to a Polars DataFrame.

method · orcapod.core.streams.base.StreamBase.as_lazy_frame

def as_lazy_frame(
*,
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> 'pl.LazyFrame'

Convert the entire stream to a Polars LazyFrame.

method · orcapod.core.streams.base.StreamBase.flow

def flow() -> list[tuple[TagProtocol, DataProtocol]]

Materialize the stream into a concrete collection of (TagProtocol, DataProtocol) pairs.

This is implemented by iterating over iter_data. Depending on the concrete stream implementation, iterating may trigger computation or upstream work, or it may simply materialize already-computed results.

method · orcapod.core.streams.base.StreamBase.run

def run() -> StreamView

Compute the whole stream and return a StreamView of all its rows.

The one way to compute any stream outside a job (developer, November 13, 2026, ITL-1081): head() and the notebook preview never compute a row (see head). run() does exactly the work as_table() does, through the same code path and cache, so a stream already computed is not computed again, and one whose inputs changed is computed again. The view holds every row with the default column configuration; its head, tail and sort_by_tags choose what is shown. For other column groups, use as_table(columns=...) or head(columns=...) after run(). On a stream that keeps its rows (a pod's or operator's output stream, a job node, a job's side-effect node included, an in-memory source) they read the rows run() gave and deliver no side effect; a stream that keeps none does its as_table() work again (a side-effect stream outside a job delivers again, a polling source polls again).

Returns:

  • StreamView: A StreamView of all the rows as_table() returns, with status
  • StreamView: ROWS.

method · orcapod.core.streams.base.StreamBase.head

def head(
n: int = 10,
*,
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> StreamView

Return at most n rows that are readily available, without computing.

The rows are ones the stream already holds, has cached from an earlier computation, has stored, or can read with a bounded read of its table. head never computes, polls or delivers a row (it reads only what is held or stored, or makes a bounded table read), never calls user code (a function, a predicate, a side effect) and never writes. With no rows to show, it learns the column types without data (ITL-912): each operator of the stream is applied to zero-row inputs, which processes no data, so a chain of N operators applies up to N operators to empty tables (about 60 ms for 20); an operator whose output schema is its input's (OUTPUT_SCHEMA_IS_INPUT_SCHEMA, such as PolarsFilter) is not applied at all. A user-defined operator's static_process is applied to zero-row tables and must not have effects. So a pipeline whose inputs disagree on a column's Arrow type raises from head the InputValidationError its run raises, and a filter with an ill-typed predicate shows its input's columns: the error appears at the run. head takes the column configuration of as_table, sorting included, and applies it to the rows the stream has, then takes the first n. The default configuration does not sort, as as_table() does not, so the rows are the first n that as_table() would return. With sort_by_tags (or all_info) a stream that has all of its rows (a node's stored slice, a computed stream, an in-memory source) returns the first n of as_table with that configuration. A stream that has read or computed only part of its rows (a source's bounded read, a partly computed stream) sorts those rows: the rows read, in tag order, not the first n in tag order. The notebook preview of a stream is its head() asked for sort_by_tags.

This default shows no rows: a stream class that can serve rows without computing overrides it. The zero-row table then has the stream's columns, typed from its declared (Python) schema, so a column's Arrow width can differ from the one the computed stream will have (int shows as int64). That zero-row table has the declared columns only: never a _source_*, _context_key or content-hash column, whatever columns asks for, because whether such a column exists and, for _source_*, its type are known only from rows.

Parameters:

  • n (int) default 10: The largest number of rows to return.
  • columns (ColumnConfig | dict[str, Any] | None) default None: Column groups to include and whether to sort by tags, exactly as for as_table.
  • all_info (bool) default False: Include every column group and sort by tags, as for as_table.

Returns:

  • StreamView: A StreamView with the rows and a status that tells an empty result
  • StreamView: from one that is not computed.

Raises:

  • TypeError: If n is not an int.
  • ValueError: If n is negative.