Skip to content
This site documents the development version of Orcapod (the main branch), which may differ from the latest release. For the releases, see PyPI and the changelog.

Operators

Operators perform structural transformations on streams without inspecting or synthesizing data values. An operator may add a tag (AddTags), never data.

class · orcapod.core.operators.join.Join

class Join(tracker_manager: TrackerManagerProtocol | None = None, **kwargs)

Bases: NonZeroInputOperator

property · orcapod.core.operators.join.Join.kernel_id

Type: tuple[str, ...]

Returns a unique identifier for the kernel. This is used to identify the kernel in the computational graph.

method · orcapod.core.operators.join.Join.validate_nonzero_inputs

def validate_nonzero_inputs(*streams: StreamProtocol) -> None

Validate that input streams are compatible for joining.

method · orcapod.core.operators.join.Join.static_process

def static_process(*streams: StreamProtocol) -> StreamProtocol

Join the streams on their shared tag columns.

The result holds every input's tag and data columns, source info and system tags (renamed per input). It holds none of the inputs' meta (__-prefixed) columns, which Join drops (meta_columns.without_meta_columns).

Raises:

  • InputValidationError: If two inputs share a data column, or a shared tag key has different Arrow types in two inputs.

method · orcapod.core.operators.join.Join.async_execute

async def async_execute(
inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]],
output: WritableChannel[tuple[TagProtocol, DataProtocol]],
*,
input_pipeline_hashes: Sequence[ContentHash],
) -> None

Async streaming join with pairwise iterative semantics.

Single input: streams through directly without any buffering.

Two inputs: binary symmetric hash join — each arriving row is probed against the opposite side's buffer, emitting matches as soon as found.

Three or more inputs: staggered pairwise binary joins in canonical order — join(join(x, y), z) — matching static_process's iterative accumulation. Each binary join uses the per-pair intersection of tag keys, so partially overlapping tag schemas are handled correctly.

Parameters:

  • inputs (Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]]): Readable channels, one per upstream.
  • output (WritableChannel[tuple[TagProtocol, DataProtocol]]): Writable channel for downstream.
  • input_pipeline_hashes (Sequence[ContentHash]): Pipeline hash for each input, positionally matching inputs. With 2+ inputs, each input's system-tag columns are renamed after its hash, as on the sync path.

Raises:

  • ValueError: If input_pipeline_hashes does not have one hash per input.

class · orcapod.core.operators.merge_join.MergeJoin

class MergeJoin(tracker_manager: TrackerManagerProtocol | None = None, **kwargs)

Bases: BinaryOperator

Binary operator that joins two streams, merging colliding data columns into sorted lists.

For data columns that exist in both streams:

  • Values are combined into a list and sorted independently per column.
  • Corresponding source columns are reordered to match the sort order of their data column.

For non-colliding columns, values are kept as scalars (same as regular Join).

Tag columns use inner join on shared tags, with union of tag schemas.

Arrow types: every output column takes the large-normalized form of its input Arrow type (string becomes large_string, list becomes large_list, recursively; the Arrow hasher treats the two variants as identical), and nothing is inferred from Python values. Integer and float width, nullability and logical (extension) types are kept. A merged column is large_list<T> where T is the normalized input element type, built by arrow_utils.aggregated_list_type (the construction Batch and GroupBy use), with or without rows; a logical T yields the logical list type of T. Its source-info column is large_list<large_string>.

When the two inputs' Arrow types for the same column differ, both are normalized to their large variants; if they are then equal the join proceeds, otherwise it raises InputValidationError naming the column and both types (require_shared_arrow_type). Validation cannot detect such a disagreement, because streams expose only Python schemas; it is detected in binary_static_process before the join.

MergeJoin is commutative: MergeJoin(A, B) produces the same result as MergeJoin(B, A), achieved by sorting merged values and system tag values.

Data-tags (ITL-1194) follow Join, with Join's helpers. A tag x that is a data-tag in both inputs is a join key, so its two values are equal: the output has ONE scalar data-tag column x, not a two-element list, and its source info is one token, the join_v1 fold of the two inputs' tokens (sorted). A tag that is a data-tag in one input stays a data-tag, with its own token, whether the other input has it as a plain tag or not at all. A data-tag of one input named like a data-only column of the other is refused (the tag and data role check). A shared key that is T in one input and T | None in the other is T in both roles (ITL-829). Every other colliding data column is merged into a sorted list, as for plain inputs.

method · orcapod.core.operators.merge_join.MergeJoin.binary_static_process

def binary_static_process(
left_stream: StreamProtocol,
right_stream: StreamProtocol,
) -> StreamProtocol

Inner-join two streams on their shared tags, merging colliding data columns.

A data column present in both streams becomes a sorted two-element list per row, with its source-info column reordered to match. System tag columns are renamed with each stream's pipeline hash and canonical position. A zero-row input yields a zero-row result with the same column types as a non-empty one.

Parameters:

  • left_stream (StreamProtocol): The left input stream.
  • right_stream (StreamProtocol): The right input stream.

Returns:

  • StreamProtocol: The joined stream, with tag columns first.

class · orcapod.core.operators.semijoin.SemiJoin

class SemiJoin(tracker_manager: TrackerManagerProtocol | None = None, **kwargs)

Bases: BinaryOperator

Binary operator that performs a semi-join between two streams.

A semi-join returns only the entries from the left stream that have matching entries in the right stream. The match keys are exactly the columns that are tag columns of both streams (ITL-1088): a row matches when its values in all of them equal a right row's. A name shared as a data column, or a tag on one side and a data column on the other, is not a key and is ignored, with a SemiJoinIgnoredColumnWarning (provisional in v0.3); the right stream's data columns are never read.

If the streams share no tag column, the entire left stream is returned unchanged.

The output stream preserves the schema of the left stream exactly. A shared tag key that is T in one input and T | None in the other is accepted (a null key matches nothing) and keeps the left input's type (ITL-829).

method · orcapod.core.operators.semijoin.SemiJoin.binary_static_process

def binary_static_process(
left_stream: StreamProtocol,
right_stream: StreamProtocol,
) -> StreamProtocol

Performs a semi-join between left and right streams. Returns entries from left stream that have matching entries in right stream.

method · orcapod.core.operators.semijoin.SemiJoin.binary_output_schema

def binary_output_schema(
left_stream: StreamProtocol,
right_stream: StreamProtocol,
*,
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> tuple[Schema, Schema]

Returns the output types for the semi-join operation. The output preserves the exact schema of the left stream.

method · orcapod.core.operators.semijoin.SemiJoin.validate_binary_inputs

def validate_binary_inputs(
left_stream: StreamProtocol,
right_stream: StreamProtocol,
) -> None

Validates that the input streams are compatible for semi-join. Checks that the match keys (the shared tag columns) have compatible types; data columns are not keys and are not checked.

Stores the common keys so that async_execute can use them to determine the correct empty-right behavior without data.

Warns:

  • SemiJoinIgnoredColumnWarning: Once per operator and set of names, when the inputs share a name that is not a tag on both sides (ITL-1088).

method · orcapod.core.operators.semijoin.SemiJoin.async_execute

async def async_execute(
inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]],
output: WritableChannel[tuple[TagProtocol, DataProtocol]],
*,
input_pipeline_hashes: Sequence[ContentHash],
) -> None

Build-probe: collect right input, then stream left through a hash lookup.

Phase 1 — Build: collect all rows from the right (filter) channel and index them by the common-key values. The left channel is read at the same time (ITL-1075): the left rows that arrive before the right input ends are kept, at most the whole left input, as the sync path and MergeJoin's barrier path hold their inputs. Reading the left input only after the right one ended deadlocked a run in which one node fed both inputs. Phase 2 — Probe: the kept left rows, then the rest of the left input one at a time with backpressure; for each row whose common-key values appear in the right-side index, emit immediately. Left rows keep their arrival order.

Falls back to barrier mode when the right input is empty (schema cannot be inferred from data) or when there are no common keys.

Key types are checked (require_shared_arrow_type) on the first row of each input, since channels carry no schema. When either input is empty there is nothing to check, so a key type mismatch the sync path reports gives an empty result here (as in Join's async path).

Parameters:

  • inputs (Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]]): The left and right readable input channels.
  • output (WritableChannel[tuple[TagProtocol, DataProtocol]]): Writable channel for downstream consumption.
  • input_pipeline_hashes (Sequence[ContentHash]): Not used: the left rows pass through with their system-tag names, so no column is named after an input's pipeline hash.

class · orcapod.core.operators.batch.Batch

class Batch(batch_size: int = 0, drop_partial_batch: bool = False, **kwargs)

Bases: UnaryOperator

Group consecutive rows into batches of batch_size rows, list-wrapping them.

Batch has no key, so it keeps every tag, list-valued: those lists are what tells a batch's members apart. Every user tag, data and source-info column becomes a list with one element per member; system-tag columns fold to a scalar digest and gain a ::{pipeline_hash} name suffix.

A data-tag (ITL-1017, developer decision of November 16, 2026: "Batch always simply takes anything and puts into list") is one column, so it becomes the same list in both roles and stays a data-tag. Its source info is one token per member, as for any data column: nothing is merged, so no token is folded.

Parameters:

  • batch_size (int) default 0: Rows per batch; 0 puts every row in one batch.
  • drop_partial_batch (bool) default False: Drop a last batch with fewer than batch_size rows.

method · orcapod.core.operators.batch.Batch.validate_unary_input

def validate_unary_input(stream: StreamProtocol) -> None

Batch works on any input stream, so no validation is needed.

method · orcapod.core.operators.batch.Batch.unary_static_process

def unary_static_process(stream: StreamProtocol) -> StreamProtocol

Group rows into fixed-size batches, list-wrapping their values.

Tag and data columns become list-valued; a data-tag, one column, becomes the same list in both roles and stays a data-tag. Source-info columns become list-valued too, one element per batch member. System-tag columns are folded to a scalar instead -- record identity hashes them directly, so they must not become lists.

Parameters:

  • stream (StreamProtocol): The upstream stream.

Returns:

  • StreamProtocol: A stream with one row per batch.

method · orcapod.core.operators.batch.Batch.unary_output_schema

def unary_output_schema(
stream: StreamProtocol,
*,
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> tuple[Schema, Schema]

Predict the batched output schemas without batching.

Every user tag, data, and source column becomes list[T], so a data-tag is list[T] in both halves. System tag columns keep their scalar type and gain a ::{pipeline_hash} name suffix.

Parameters:

  • stream (StreamProtocol): The upstream stream.
  • columns (ColumnConfig | dict[str, Any] | None) default None: Column inclusion config.
  • all_info (bool) default False: Include all info columns.

Returns:

method · orcapod.core.operators.batch.Batch.async_execute

async def async_execute(
inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]],
output: WritableChannel[tuple[TagProtocol, DataProtocol]],
*,
input_pipeline_hashes: Sequence[ContentHash],
) -> None

Streaming batch: emit full batches as they accumulate.

When batch_size > 0, each group of batch_size rows is materialized and emitted immediately, allowing downstream consumers to start processing before all input is consumed. When batch_size == 0 (batch everything), falls back to barrier mode.

Parameters:

  • inputs (Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]]): The one readable input channel.
  • output (WritableChannel[tuple[TagProtocol, DataProtocol]]): Writable channel for downstream consumption.
  • input_pipeline_hashes (Sequence[ContentHash]): The input stream's pipeline hash, as a one-element sequence. Every rebuilt batch reports it, so the folded system-tag names end in it as on the sync path (ITL-906).

Raises:

  • ValueError: If input_pipeline_hashes does not have one hash per input.

method · orcapod.core.operators.batch.Batch.to_config

def to_config() -> dict[str, Any]

Serialize this Batch operator to a config dict.

Returns:

  • dict[str, Any]: A dict with class_name, module_path, and config keys,
  • dict[str, Any]: where config contains batch_size and drop_partial_batch.

class · orcapod.core.operators.group_by.GroupBy

class GroupBy(by: str | Collection[str], **kwargs: Any)

Bases: UnaryOperator

Reduce rows sharing a key into one row with list-valued data members.

This is the only many->one operator. Every other operator preserves one row per tag; GroupBy collapses N rows into one, which is what lets a downstream pod receive a whole group at once (for example, all of a recording session's per-probe result parquets).

Given tags (subject, date, probe) and data (path), grouping by ["subject", "date"] emits one row per distinct (subject, date):

  • subject and date stay scalar tag columns, in the input's tag order; they are the only tag columns of the output
  • probe is dropped (ITL-1070): a list of the members' values is not a key for the group, as SQL drops a non-key, non-aggregated column after GROUP BY. A data-tag probe loses its tag role only: its data column stays and becomes list-valued like path
  • path becomes list-valued
  • the _source_* column of each data column becomes list-valued, one element per member, except that of a data-tag group key: its value is the merge of the members' equal values, so its source info is one token, the group_v1 fold of the members' tokens in member order (ITL-1017)
  • system-tag columns fold to a scalar digest and gain a ::{pipeline_hash} name suffix

No data element ever comes from a tag value: GroupBy uses tags to form groups, but the data columns of the output are exactly the data columns of the input. The members' values of a dropped plain tag are not carried anywhere. If a downstream step needs to know which member each element came from, make that value a data column before grouping: declare the tag a data-tag at the source (the default), declare it as data, or use DerivedSource(origin, tags_as_data={...}).

Members are sorted by the input's non-group-key tag values, with record_id appended as a final tiebreaker, so the emitted lists are stable across runs. The list of a data-tag non-key is therefore in its own value order, and every other data list is aligned with it. This matters because orcapod hashes those lists to build the cache key -- an unsorted list would make an identical member set hash differently and trigger a spurious recompute. Groups themselves are emitted in group-key order for the same reason: a reordered input must produce an identical output.

Tag tuples are expected to be unique within a stream, but nothing enforces that. If the input does contain duplicate tag tuples, the members sharing a tuple are separated only by record_id, which is fixed when the source is materialized -- so their order is stable for a given source table, but undefined if the source table itself is permuted.

A grouped stream can be grouped again on a coarser key: its data lists nest one level deeper (list[list[T]]), as source info does. The members of the second grouping are ordered by the first grouping's keys that are not in the coarser key (scalar tags of the grouped stream), then by record_id; a tag the first grouping dropped no longer takes part. A list-valued tag of the input can order members but cannot be a grouping key.

Contrast with Batch, which partitions by row count for throughput. It has no key, so it keeps every tag column, list-valued: those lists are what tells a batch's members apart.

Parameters:

  • by (str | Collection[str]): Tag column names to group on, or one name as a string (a bare string is one name, never its characters). Must be non-empty, free of duplicates, and must all be scalar tag columns of the input stream.

method · orcapod.core.operators.group_by.GroupBy.to_config

def to_config() -> dict[str, Any]

Serialize this GroupBy operator to a config dict.

by is emitted as a list rather than a tuple so the config stays JSON-serializable; __init__ normalizes it back to a tuple.

Returns:

  • dict[str, Any]: A dict with class_name, module_path, and config keys,
  • dict[str, Any]: where config contains by.

method · orcapod.core.operators.group_by.GroupBy.validate_unary_input

def validate_unary_input(stream: StreamProtocol) -> None

Verify every grouping column is a scalar tag column of the input.

Parameters:

  • stream (StreamProtocol): The upstream stream to validate.

Raises:

  • InputValidationError: If any name in by is not a tag column, or names a list-valued tag column.

method · orcapod.core.operators.group_by.GroupBy.unary_static_process

def unary_static_process(stream: StreamProtocol) -> StreamProtocol

Partition rows by group key and emit one row per group.

Parameters:

  • stream (StreamProtocol): The upstream stream.

Returns:

  • StreamProtocol: A stream with one row per distinct group-key tuple, whose tag
  • StreamProtocol: columns are the group keys, scalar, in the input's tag order.

method · orcapod.core.operators.group_by.GroupBy.unary_output_schema

def unary_output_schema(
stream: StreamProtocol,
*,
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> tuple[Schema, Schema]

Predict the grouped output schemas without grouping.

Parameters:

  • stream (StreamProtocol): The upstream stream.
  • columns (ColumnConfig | dict[str, Any] | None) default None: Column inclusion config.
  • all_info (bool) default False: Include all info columns.

Returns:

  • Schema: A (tag_schema, data_schema) tuple. The tag schema holds the
  • Schema: group keys, scalar, in the input's tag order, and the system tags,
  • tuple[Schema, Schema]: scalar under their suffixed names; every other user tag is
  • tuple[Schema, Schema]: dropped. The data schema holds the input's data columns as
  • tuple[Schema, Schema]: list[T] (a data-tag non-key among them), except that a
  • tuple[Schema, Schema]: data-tag key stays scalar.

class · orcapod.core.operators.column_selection.SelectTagColumns

class SelectTagColumns(columns: str | Collection[str], strict: bool = True, **kwargs)

Bases: UnaryOperator

Operator that selects specified columns from a stream.

method · orcapod.core.operators.column_selection.SelectTagColumns.to_config

def to_config() -> dict[str, Any]

Serialize this SelectTagColumns operator to a config dict.

Returns:

  • dict[str, Any]: A dict with class_name, module_path, and config keys,
  • dict[str, Any]: where config contains columns and strict.

method · orcapod.core.operators.column_selection.SelectTagColumns.validate_unary_input

def validate_unary_input(stream: StreamProtocol) -> None

This method should be implemented by subclasses to validate the inputs to the operator. It takes two streams as input and raises an error if the inputs are not valid.

method · orcapod.core.operators.column_selection.SelectTagColumns.async_execute

async def async_execute(
inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]],
output: WritableChannel[tuple[TagProtocol, DataProtocol]],
*,
input_pipeline_hashes: Sequence[ContentHash],
) -> None

Streaming: select tag columns per row without materializing.

Parameters:

  • inputs (Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]]): The one readable input channel.
  • output (WritableChannel[tuple[TagProtocol, DataProtocol]]): Writable channel for downstream consumption.
  • input_pipeline_hashes (Sequence[ContentHash]): Not used: this operator keeps system-tag names as they are, so no column is named after an input's pipeline hash.

class · orcapod.core.operators.column_selection.SelectDataColumns

class SelectDataColumns(columns: str | Collection[str], strict: bool = True, **kwargs)

Bases: UnaryOperator

Operator that selects specified columns from a stream.

method · orcapod.core.operators.column_selection.SelectDataColumns.to_config

def to_config() -> dict[str, Any]

Serialize this SelectDataColumns operator to a config dict.

Returns:

  • dict[str, Any]: A dict with class_name, module_path, and config keys,
  • dict[str, Any]: where config contains columns and strict.

method · orcapod.core.operators.column_selection.SelectDataColumns.validate_unary_input

def validate_unary_input(stream: StreamProtocol) -> None

This method should be implemented by subclasses to validate the inputs to the operator. It takes two streams as input and raises an error if the inputs are not valid.

method · orcapod.core.operators.column_selection.SelectDataColumns.async_execute

async def async_execute(
inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]],
output: WritableChannel[tuple[TagProtocol, DataProtocol]],
*,
input_pipeline_hashes: Sequence[ContentHash],
) -> None

Streaming: select data columns per row without materializing.

Parameters:

  • inputs (Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]]): The one readable input channel.
  • output (WritableChannel[tuple[TagProtocol, DataProtocol]]): Writable channel for downstream consumption.
  • input_pipeline_hashes (Sequence[ContentHash]): Not used: this operator keeps system-tag names as they are, so no column is named after an input's pipeline hash.

class · orcapod.core.operators.column_selection.DropTagColumns

class DropTagColumns(columns: str | Collection[str], strict: bool = True, **kwargs)

Bases: UnaryOperator

Operator that drops specified columns from a stream.

method · orcapod.core.operators.column_selection.DropTagColumns.to_config

def to_config() -> dict[str, Any]

Serialize this DropTagColumns operator to a config dict.

Returns:

  • dict[str, Any]: A dict with class_name, module_path, and config keys,
  • dict[str, Any]: where config contains columns and strict.

method · orcapod.core.operators.column_selection.DropTagColumns.validate_unary_input

def validate_unary_input(stream: StreamProtocol) -> None

This method should be implemented by subclasses to validate the inputs to the operator. It takes two streams as input and raises an error if the inputs are not valid.

method · orcapod.core.operators.column_selection.DropTagColumns.async_execute

async def async_execute(
inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]],
output: WritableChannel[tuple[TagProtocol, DataProtocol]],
*,
input_pipeline_hashes: Sequence[ContentHash],
) -> None

Streaming: drop tag columns per row without materializing.

Parameters:

  • inputs (Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]]): The one readable input channel.
  • output (WritableChannel[tuple[TagProtocol, DataProtocol]]): Writable channel for downstream consumption.
  • input_pipeline_hashes (Sequence[ContentHash]): Not used: this operator keeps system-tag names as they are, so no column is named after an input's pipeline hash.

class · orcapod.core.operators.column_selection.DropDataColumns

class DropDataColumns(columns: str | Collection[str], strict: bool = True, **kwargs)

Bases: UnaryOperator

Operator that drops specified columns from a stream.

method · orcapod.core.operators.column_selection.DropDataColumns.to_config

def to_config() -> dict[str, Any]

Serialize this DropDataColumns operator to a config dict.

Returns:

  • dict[str, Any]: A dict with class_name, module_path, and config keys,
  • dict[str, Any]: where config contains columns and strict.

method · orcapod.core.operators.column_selection.DropDataColumns.validate_unary_input

def validate_unary_input(stream: StreamProtocol) -> None

This method should be implemented by subclasses to validate the inputs to the operator. It takes two streams as input and raises an error if the inputs are not valid.

method · orcapod.core.operators.column_selection.DropDataColumns.async_execute

async def async_execute(
inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]],
output: WritableChannel[tuple[TagProtocol, DataProtocol]],
*,
input_pipeline_hashes: Sequence[ContentHash],
) -> None

Streaming: drop data columns per row without materializing.

Parameters:

  • inputs (Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]]): The one readable input channel.
  • output (WritableChannel[tuple[TagProtocol, DataProtocol]]): Writable channel for downstream consumption.
  • input_pipeline_hashes (Sequence[ContentHash]): Not used: this operator keeps system-tag names as they are, so no column is named after an input's pipeline hash.

class · orcapod.core.operators.column_selection.DropColumns

class DropColumns(columns: str | Collection[str], strict: bool = True, **kwargs)

Bases: _TagAndDataColumnsOperator

Drop the named columns, whether each is a tag or a data column (ITL-1084).

For tag names only it gives the rows of DropTagColumns, for data names only those of DropDataColumns. A dropped data column's source-info column goes with it; system tags are kept.

Parameters:

  • columns (str | Collection[str]): One column name or a collection of names.
  • strict (bool) default True: When True (the default), a name that is neither a tag nor a data column of the input raises InputValidationError; when False it is ignored.

Raises:

  • InputValidationError: When called on a stream whose data columns would all be dropped: a stream needs at least one data column.

class · orcapod.core.operators.column_selection.SelectColumns

class SelectColumns(columns: str | Collection[str], strict: bool = True, **kwargs)

Bases: _TagAndDataColumnsOperator

Keep exactly the named columns, tag or data; drop every other one (ITL-1084).

Every user tag and data column not named is dropped, with the source-info column of each dropped data column; system tags are kept. Naming no tag is allowed (the result has no tag column).

Parameters:

  • columns (str | Collection[str]): One column name or a collection of names.
  • strict (bool) default True: When True (the default), a name that is neither a tag nor a data column of the input raises InputValidationError; when False it is ignored.

Raises:

  • InputValidationError: When called on a stream none of whose data columns is named: a stream needs at least one data column.

class · orcapod.core.operators.mappers.MapTags

class MapTags(
name_map: Mapping[str, str],
drop_unmapped: bool = False,
strict: bool = True,
tag_only: bool = False,
**kwargs,
)

Bases: UnaryOperator

Rename tag columns.

Data-tags (developer, November 12, 2026, ITL-1023): renaming a data-tag renames its one column in both roles, so the data-tag survives under the new name. With tag_only=True only the tag is renamed: the data column keeps its name and the renamed tag is a plain tag with the same values. A plain tag is renamed as before. A data-tag dropped as unmapped keeps its data column.

Parameters:

  • name_map (Mapping[str, str]): Each tag column to rename, mapped to its new name.
  • drop_unmapped (bool) default False: When True, the tag columns not in name_map are dropped.
  • strict (bool) default True: When True (the default), a key of name_map that is not a tag column of the input raises InputValidationError; when False it is ignored. Part of the operator's identity, as for SelectTagColumns and the other column operators.
  • tag_only (bool) default False: Rename data-tags in the tag role only.

method · orcapod.core.operators.mappers.MapTags.to_config

def to_config() -> dict[str, Any]

Serialize this MapTags operator to a config dict.

Returns:

  • dict[str, Any]: A dict with class_name, module_path, and config keys,
  • dict[str, Any]: where config contains name_map, drop_unmapped and
  • dict[str, Any]: strict.

method · orcapod.core.operators.mappers.MapTags.validate_unary_input

def validate_unary_input(stream: StreamProtocol) -> None

This method should be implemented by subclasses to validate the inputs to the operator. It takes two streams as input and raises an error if the inputs are not valid.

method · orcapod.core.operators.mappers.MapTags.async_execute

async def async_execute(
inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]],
output: WritableChannel[tuple[TagProtocol, DataProtocol]],
*,
input_pipeline_hashes: Sequence[ContentHash],
) -> None

Streaming: rename tag columns per row without materializing.

Parameters:

  • inputs (Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]]): The one readable input channel.
  • output (WritableChannel[tuple[TagProtocol, DataProtocol]]): Writable channel for downstream consumption.
  • input_pipeline_hashes (Sequence[ContentHash]): Not used: this operator keeps system-tag names as they are, so no column is named after an input's pipeline hash.

class · orcapod.core.operators.mappers.MapData

class MapData(
name_map: Mapping[str, str],
drop_unmapped: bool = False,
strict: bool = True,
**kwargs,
)

Bases: UnaryOperator

Rename data columns.

Renaming data is always a data-only move (developer, November 12, 2026, ITL-1023): renaming the data column of a data-tag x gives the new data column and leaves x a plain tag with the same values.

Parameters:

  • name_map (Mapping[str, str]): Each data column to rename, mapped to its new name.
  • drop_unmapped (bool) default False: When True, the data columns not in name_map are dropped.
  • strict (bool) default True: When True (the default), a key of name_map that is not a data column of the input raises InputValidationError; when False it is ignored. Part of the operator's identity, as for SelectTagColumns and the other column operators.

method · orcapod.core.operators.mappers.MapData.to_config

def to_config() -> dict[str, Any]

Serialize this MapData operator to a config dict.

Returns:

  • dict[str, Any]: A dict with class_name, module_path, and config keys,
  • dict[str, Any]: where config contains name_map, drop_unmapped and
  • dict[str, Any]: strict.

method · orcapod.core.operators.mappers.MapData.async_execute

async def async_execute(
inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]],
output: WritableChannel[tuple[TagProtocol, DataProtocol]],
*,
input_pipeline_hashes: Sequence[ContentHash],
) -> None

Streaming: rename data columns per row without materializing.

Parameters:

  • inputs (Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]]): The one readable input channel.
  • output (WritableChannel[tuple[TagProtocol, DataProtocol]]): Writable channel for downstream consumption.
  • input_pipeline_hashes (Sequence[ContentHash]): Not used: this operator keeps system-tag names as they are, so no column is named after an input's pipeline hash.

class · orcapod.core.operators.mappers.MapColumns

class MapColumns(
name_map: Mapping[str, str],
drop_unmapped: bool = False,
strict: bool = True,
**kwargs,
)

Bases: UnaryOperator

Rename columns, whether each is a tag or a data column (ITL-1160).

The counterpart of SelectColumns and DropColumns for renaming: one step renames tag and data columns alike. A renamed tag stays a tag and a renamed data column stays data; a renamed data column's source-info column is renamed with it. System tags pass through, the stream's column order is kept, and, as for SelectColumns, no data meta (__) column of the input is carried.

Parameters:

  • name_map (Mapping[str, str]): Each column to rename, tag or data, mapped to its new name.
  • drop_unmapped (bool) default False: When True, every user tag and data column not in name_map is dropped (with the source-info column of a dropped data column); system tags are kept.
  • strict (bool) default True: When True (the default), a key of name_map that is neither a tag nor a data column of the input raises InputValidationError; when False it is ignored. Part of the operator's identity, as for SelectColumns.

Raises:

  • InputValidationError: When called on a stream where a new name is taken by a column that is not renamed, two columns get the same new name, or (with drop_unmapped) no data column would remain.

method · orcapod.core.operators.mappers.MapColumns.to_config

def to_config() -> dict[str, Any]

Serialize this MapColumns operator to a config dict.

Returns:

  • dict[str, Any]: A dict with class_name, module_path, and config keys,
  • dict[str, Any]: where config contains name_map, drop_unmapped and
  • dict[str, Any]: strict.

method · orcapod.core.operators.mappers.MapColumns.validate_unary_input

def validate_unary_input(stream: StreamProtocol) -> None

Validate the names against the input stream.

Raises:

  • InputValidationError: As _validate_columns describes.

method · orcapod.core.operators.mappers.MapColumns.async_execute

async def async_execute(
inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]],
output: WritableChannel[tuple[TagProtocol, DataProtocol]],
*,
input_pipeline_hashes: Sequence[ContentHash],
) -> None

Streaming: rename (and drop) the columns per row without materializing.

The names are validated against the first row, as validate_unary_input validates them against the stream.

Parameters:

  • inputs (Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]]): The one readable input channel.
  • output (WritableChannel[tuple[TagProtocol, DataProtocol]]): Writable channel for downstream consumption.
  • input_pipeline_hashes (Sequence[ContentHash]): Not used: this operator keeps system-tag names as they are, so no column is named after an input's pipeline hash.

class · orcapod.core.operators.add_tags.AddTags

class AddTags(tags: Mapping[str, TagConstant], **kwargs: Any)

Bases: UnaryOperator

Add tags with constant values to every row of a stream.

Each new tag is a plain tag column holding the same value on every row. A tag is not a data value, so an operator may add one (developer, November 16, 2026, ITL-1196); data columns, their values and their source info pass through unchanged, and no data column is added. System tags keep their names and values (name-preserving), so record ids are unchanged.

The new tags follow the input's tags, in the order given. A value's exact Python type is the tag's type, mapped to Arrow by the data context's type converter: bool to bool, int to int64, float to double, str to large_string, bytes to large_binary. Every new tag is non-nullable.

The operator's identity is its tags, names and values in order, so two different values are two operators with two stored paths.

Data-tags (ITL-1023): the input's data-tags stay data-tags, one column in both roles with its values and source info; the new tags are plain. A new tag named like any column of the input, a data-tag included, is refused.

Parameters:

  • tags (Mapping[str, TagConstant]): Each new tag's name mapped to its value.

Raises:

  • ValueError: When no tag is given, a name is empty, or a value is out of range (see _checked_value).
  • TypeError: When a name is not a string or a value's type is not accepted.
  • InputValidationError: When a name is reserved (meta or system prefix, :: separator).

property · orcapod.core.operators.add_tags.AddTags.tags

Type: Mapping[str, TagConstant]

The new tags, names mapped to values, in order; read-only.

Read-only so the identity, the saved config and the cached constants can never disagree.

method · orcapod.core.operators.add_tags.AddTags.to_config

def to_config() -> dict[str, Any]

Serialize this AddTags operator to a config dict.

Each tag is saved with its type token and its value; a bytes value as hex text.

Returns:

  • dict[str, Any]: A dict with class_name, module_path and config keys, where
  • dict[str, Any]: config holds tags, a list of {"name", "type", "value"}.

classmethod · orcapod.core.operators.add_tags.AddTags.from_config

def from_config(config: dict[str, Any]) -> AddTags

Reconstruct an AddTags operator from a config dict.

Parameters:

  • config (dict[str, Any]): Dict as returned by to_config().

Returns:

Raises:

  • ValueError: When a saved tag names an unknown type.

method · orcapod.core.operators.add_tags.AddTags.unary_static_process

def unary_static_process(stream: StreamProtocol) -> StreamProtocol

Return the stream's rows with the new tags added.

Parameters:

  • stream (StreamProtocol): The upstream stream.

Returns:

  • StreamProtocol: A stream over the same rows, in order, with the new tag columns after
  • StreamProtocol: the input's tags. A result with no rows has the same schema.

method · orcapod.core.operators.add_tags.AddTags.async_execute

async def async_execute(
inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]],
output: WritableChannel[tuple[TagProtocol, DataProtocol]],
*,
input_pipeline_hashes: Sequence[ContentHash],
) -> None

Streaming: add the new tags to each row as it arrives.

Each row's tag table goes through the rule of the sync path (_with_constants); its data passes through unchanged.

Parameters:

  • inputs (Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]]): Single-element sequence of readable channels.
  • output (WritableChannel[tuple[TagProtocol, DataProtocol]]): Channel to send the rows to.
  • input_pipeline_hashes (Sequence[ContentHash]): Not used: this operator keeps system-tag names as they are, so no column is named after an input's pipeline hash.

class · orcapod.core.operators.filters.PolarsFilter

class PolarsFilter(
predicates: Collection[pl_type.IntoExprColumn | Iterable[pl_type.IntoExprColumn] | bool | list[bool] | np.ndarray[Any, Any]] = (),
constraints: Mapping[str, Any] | None = None,
**kwargs,
)

Bases: UnaryOperator

Operator that applies Polars filtering to a stream

method · orcapod.core.operators.filters.PolarsFilter.validate_unary_input

def validate_unary_input(stream: StreamProtocol) -> None

This method should be implemented by subclasses to validate the inputs to the operator. It takes two streams as input and raises an error if the inputs are not valid.

method · orcapod.core.operators.filters.PolarsFilter.to_config

def to_config() -> dict[str, Any]

Serialize this PolarsFilter operator to a config dict.

Polars Expr predicates are serialized to JSON strings when possible. If any predicate cannot be serialized, reconstructable is set to False and predicates is set to None.

Returns:

  • dict[str, Any]: A dict with class_name, module_path, and config keys.

classmethod · orcapod.core.operators.filters.PolarsFilter.from_config

def from_config(config: dict[str, Any]) -> PolarsFilter

Reconstruct a PolarsFilter from a config dict.

Parameters:

  • config (dict[str, Any]): Dict as returned by to_config().

Returns:

Raises:

  • NotImplementedError: If reconstructable is False.

class · orcapod.core.operators.pick.Pick

class Pick(
column: str,
key: str,
out: str | None = None,
fail_on_miss: bool = False,
**kwargs: Any,
)

Bases: UnaryOperator

Extract a value from a struct- or dict-typed data column by key.

For dict[K, V] columns the lookup is per-packet; if the key is absent the packet is skipped (or an error is raised when fail_on_miss=True).

For extension-type struct columns (Pydantic, dataclass) the field existence is validated at build time and guaranteed at runtime.

The extracted column keeps the value's Arrow type (dict[str, int16] yields int16). Its nullability follows the predicted output type, which is not Optional for dict values, so the column is NOT NULL. A value that is None therefore raises ValueError (whatever fail_on_miss is), naming the operator, the column and the row's tag.

Parameters:

  • column (str): Name of the data column to project into.
  • key (str): Dict key or struct field name to extract.
  • out (str | None) default None: Output column name. None (default) replaces column in-place; a string adds a new column alongside the original.
  • fail_on_miss (bool) default False: If True, raise RuntimeError when the key is absent in a packet instead of skipping. Excluded from identity_structure — miss-handling does not affect functional output semantics. See ITL-439.

method · orcapod.core.operators.pick.Pick.validate_unary_input

def validate_unary_input(stream: StreamProtocol) -> None

Validate input schema and resolve output type.

Parameters:

  • stream (StreamProtocol): The upstream stream to validate.

Raises:

  • InputValidationError: If column is missing, out collides with an existing column, or the column type does not support pick (including a logical type without pick_field, such as numpy.ndarray).

method · orcapod.core.operators.pick.Pick.unary_output_schema

def unary_output_schema(
stream: StreamProtocol,
*,
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> tuple[Schema, Schema]

Return the (tag, data) schemas for the output stream.

Parameters:

  • stream (StreamProtocol): The upstream stream.
  • columns (ColumnConfig | dict[str, Any] | None) default None: Column inclusion config passed through to output_schema.
  • all_info (bool) default False: Whether to include all info columns.

Returns:

method · orcapod.core.operators.pick.Pick.unary_static_process

def unary_static_process(stream: StreamProtocol) -> StreamProtocol

Process the full stream packet by packet.

Iterates stream.iter_data(), applies _process_one per packet, and packs surviving packets back into a stream. Skipped packets (key absent, fail_on_miss=False) are silently dropped.

Parameters:

  • stream (StreamProtocol): The upstream stream.

Returns:

  • StreamProtocol: A new stream with the projection applied. If all packets were
  • StreamProtocol: skipped, an empty stream with the correct output schema is returned.

Raises:

  • RuntimeError: If fail_on_miss=True and any packet's dict lacks self.key.

method · orcapod.core.operators.pick.Pick.async_execute

async def async_execute(
inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]],
output: WritableChannel[tuple[TagProtocol, DataProtocol]],
*,
input_pipeline_hashes: Sequence[ContentHash],
) -> None

Process packets one at a time as they arrive (true streaming).

Each packet is processed independently via _process_one and forwarded immediately to the output channel — no buffering required.

Parameters:

  • inputs (Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]]): Single-element sequence of readable channels.
  • output (WritableChannel[tuple[TagProtocol, DataProtocol]]): Channel to send transformed packets to.
  • input_pipeline_hashes (Sequence[ContentHash]): Not used: this operator keeps system-tag names as they are, so no column is named after an input's pipeline hash.

class · orcapod.core.operators.index.Index

class Index(
column: str,
i: int,
out: str | None = None,
fail_on_miss: bool = False,
**kwargs: Any,
)

Bases: UnaryOperator

Extract an element from a list-typed data column by position.

The list length is per-packet data, so bounds are not checked at build time. Out-of-bounds access causes the packet to be skipped (or an error when fail_on_miss=True). Negative indices follow Python semantics (-1 is the last element).

The extracted column keeps the list's element Arrow type (list<int8> yields int8). Its nullability follows the predicted output type, which is not Optional for list elements, so the column is NOT NULL. An element that is None therefore raises ValueError (whatever fail_on_miss is), naming the operator, the column and the row's tag.

Parameters:

  • column (str): Name of the data column to project into.
  • i (int): Position to extract. Negative indices follow Python semantics.
  • out (str | None) default None: Output column name. None (default) replaces column in-place; a string adds a new column alongside the original.
  • fail_on_miss (bool) default False: If True, raise RuntimeError on out-of-bounds instead of skipping. Excluded from identity_structure. See ITL-439.

method · orcapod.core.operators.index.Index.validate_unary_input

def validate_unary_input(stream: StreamProtocol) -> None

Validate input schema and resolve output element type.

Parameters:

  • stream (StreamProtocol): The upstream stream to validate.

Raises:

  • InputValidationError: If column is missing, out collides, or the column type is not a supported index target.
  • NotImplementedError: If the column holds an extension type whose index_element is not yet implemented.

method · orcapod.core.operators.index.Index.unary_output_schema

def unary_output_schema(
stream: StreamProtocol,
*,
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> tuple[Schema, Schema]

Return the (tag, data) schemas for the output stream.

Parameters:

  • stream (StreamProtocol): The upstream stream.
  • columns (ColumnConfig | dict[str, Any] | None) default None: Column inclusion config.
  • all_info (bool) default False: Include all info columns.

Returns:

method · orcapod.core.operators.index.Index.unary_static_process

def unary_static_process(stream: StreamProtocol) -> StreamProtocol

Process the full stream packet by packet.

Iterates stream.iter_data(), applies _process_one per packet, and packs surviving packets back into a stream. Skipped packets (out-of-bounds, fail_on_miss=False) are silently dropped.

Parameters:

  • stream (StreamProtocol): The upstream stream.

Returns:

  • StreamProtocol: A new stream with the projection applied. If all packets were
  • StreamProtocol: skipped, an empty stream with the correct output schema is returned.

Raises:

  • RuntimeError: If fail_on_miss=True and any packet is out-of-bounds.
  • ValueError: If an extracted element is None but the predicted output type is not Optional.

method · orcapod.core.operators.index.Index.async_execute

async def async_execute(
inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]],
output: WritableChannel[tuple[TagProtocol, DataProtocol]],
*,
input_pipeline_hashes: Sequence[ContentHash],
) -> None

Process packets one at a time as they arrive (true streaming).

Each packet is processed independently via _process_one and forwarded immediately to the output channel — no buffering required.

Parameters:

  • inputs (Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]]): Single-element sequence of readable channels.
  • output (WritableChannel[tuple[TagProtocol, DataProtocol]]): Channel to send transformed packets to.
  • input_pipeline_hashes (Sequence[ContentHash]): Not used: this operator keeps system-tag names as they are, so no column is named after an input's pipeline hash.