Operators
Operators perform structural transformations on streams without inspecting or synthesizing data values. An operator may add a tag (AddTags), never data.
class · orcapod.core.operators.join.Join
class Join(tracker_manager: TrackerManagerProtocol | None = None, **kwargs)Bases: NonZeroInputOperator
kernel_id
Section titled “kernel_id”property · orcapod.core.operators.join.Join.kernel_id
Type: tuple[str, ...]
Returns a unique identifier for the kernel. This is used to identify the kernel in the computational graph.
validate_nonzero_inputs
Section titled “validate_nonzero_inputs”method · orcapod.core.operators.join.Join.validate_nonzero_inputs
def validate_nonzero_inputs(*streams: StreamProtocol) -> NoneValidate that input streams are compatible for joining.
static_process
Section titled “static_process”method · orcapod.core.operators.join.Join.static_process
def static_process(*streams: StreamProtocol) -> StreamProtocolJoin the streams on their shared tag columns.
The result holds every input's tag and data columns, source info and
system tags (renamed per input). It holds none of the inputs' meta
(__-prefixed) columns, which Join drops (meta_columns.without_meta_columns).
Raises:
InputValidationError: If two inputs share a data column, or a shared tag key has different Arrow types in two inputs.
async_execute
Section titled “async_execute”method · orcapod.core.operators.join.Join.async_execute
async def async_execute( inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], *, input_pipeline_hashes: Sequence[ContentHash],) -> NoneAsync streaming join with pairwise iterative semantics.
Single input: streams through directly without any buffering.
Two inputs: binary symmetric hash join — each arriving row is probed against the opposite side's buffer, emitting matches as soon as found.
Three or more inputs: staggered pairwise binary joins in
canonical order — join(join(x, y), z) — matching
static_process's iterative accumulation. Each binary join
uses the per-pair intersection of tag keys, so partially
overlapping tag schemas are handled correctly.
Parameters:
- inputs (
Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]]): Readable channels, one per upstream. - output (
WritableChannel[tuple[TagProtocol, DataProtocol]]): Writable channel for downstream. - input_pipeline_hashes (
Sequence[ContentHash]): Pipeline hash for each input, positionally matchinginputs. With 2+ inputs, each input's system-tag columns are renamed after its hash, as on the sync path.
Raises:
ValueError: Ifinput_pipeline_hashesdoes not have one hash per input.
MergeJoin
Section titled “MergeJoin”class · orcapod.core.operators.merge_join.MergeJoin
class MergeJoin(tracker_manager: TrackerManagerProtocol | None = None, **kwargs)Bases: BinaryOperator
Binary operator that joins two streams, merging colliding data columns into sorted lists.
For data columns that exist in both streams:
- Values are combined into a list
and sorted independently per column. - Corresponding source columns are reordered to match the sort order of their data column.
For non-colliding columns, values are kept as scalars (same as regular Join).
Tag columns use inner join on shared tags, with union of tag schemas.
Arrow types: every output column takes the large-normalized form of its
input Arrow type (string becomes large_string, list becomes
large_list, recursively; the Arrow hasher treats the two variants as
identical), and nothing is inferred from Python values. Integer and float
width, nullability and logical (extension) types are kept. A merged column
is large_list<T> where T is the normalized input element type,
built by arrow_utils.aggregated_list_type (the construction Batch
and GroupBy use), with or without rows; a logical T yields the
logical list type of T. Its source-info column is
large_list<large_string>.
When the two inputs' Arrow types for the same column differ, both are
normalized to their large variants; if they are then equal the join
proceeds, otherwise it raises InputValidationError naming the column
and both types (require_shared_arrow_type). Validation cannot detect
such a disagreement, because streams expose only Python schemas; it is
detected in binary_static_process before the join.
MergeJoin is commutative: MergeJoin(A, B) produces the same result as MergeJoin(B, A), achieved by sorting merged values and system tag values.
Data-tags (ITL-1194) follow Join, with Join's helpers. A tag
x that is a data-tag in both inputs is a join key, so its two values are equal: the
output has ONE scalar data-tag column x, not a two-element list, and its
source info is one token, the join_v1 fold of the two inputs' tokens
(sorted). A tag that is a data-tag in one input stays a data-tag, with its own token,
whether the other input has it as a plain tag or not at all. A data-tag
of one input named like a data-only column of the other is refused (the tag and
data role check). A shared key that is T in one input and T | None in
the other is T in both roles (ITL-829). Every other colliding data column is
merged into a sorted list, as for plain inputs.
binary_static_process
Section titled “binary_static_process”method · orcapod.core.operators.merge_join.MergeJoin.binary_static_process
def binary_static_process( left_stream: StreamProtocol, right_stream: StreamProtocol,) -> StreamProtocolInner-join two streams on their shared tags, merging colliding data columns.
A data column present in both streams becomes a sorted two-element list per row, with its source-info column reordered to match. System tag columns are renamed with each stream's pipeline hash and canonical position. A zero-row input yields a zero-row result with the same column types as a non-empty one.
Parameters:
- left_stream (
StreamProtocol): The left input stream. - right_stream (
StreamProtocol): The right input stream.
Returns:
StreamProtocol: The joined stream, with tag columns first.
SemiJoin
Section titled “SemiJoin”class · orcapod.core.operators.semijoin.SemiJoin
class SemiJoin(tracker_manager: TrackerManagerProtocol | None = None, **kwargs)Bases: BinaryOperator
Binary operator that performs a semi-join between two streams.
A semi-join returns only the entries from the left stream that have
matching entries in the right stream. The match keys are exactly the
columns that are tag columns of both streams (ITL-1088): a row matches
when its values in all of them equal a right row's. A name shared as a
data column, or a tag on one side and a data column on the other, is not a
key and is ignored, with a SemiJoinIgnoredColumnWarning (provisional in
v0.3); the right stream's data columns are never read.
If the streams share no tag column, the entire left stream is returned unchanged.
The output stream preserves the schema of the left stream exactly. A shared
tag key that is T in one input and T | None in the other is accepted
(a null key matches nothing) and keeps the left input's type (ITL-829).
binary_static_process
Section titled “binary_static_process”method · orcapod.core.operators.semijoin.SemiJoin.binary_static_process
def binary_static_process( left_stream: StreamProtocol, right_stream: StreamProtocol,) -> StreamProtocolPerforms a semi-join between left and right streams. Returns entries from left stream that have matching entries in right stream.
binary_output_schema
Section titled “binary_output_schema”method · orcapod.core.operators.semijoin.SemiJoin.binary_output_schema
def binary_output_schema( left_stream: StreamProtocol, right_stream: StreamProtocol, *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False,) -> tuple[Schema, Schema]Returns the output types for the semi-join operation. The output preserves the exact schema of the left stream.
validate_binary_inputs
Section titled “validate_binary_inputs”method · orcapod.core.operators.semijoin.SemiJoin.validate_binary_inputs
def validate_binary_inputs( left_stream: StreamProtocol, right_stream: StreamProtocol,) -> NoneValidates that the input streams are compatible for semi-join. Checks that the match keys (the shared tag columns) have compatible types; data columns are not keys and are not checked.
Stores the common keys so that async_execute can use them
to determine the correct empty-right behavior without data.
Warns:
SemiJoinIgnoredColumnWarning: Once per operator and set of names, when the inputs share a name that is not a tag on both sides (ITL-1088).
async_execute
Section titled “async_execute”method · orcapod.core.operators.semijoin.SemiJoin.async_execute
async def async_execute( inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], *, input_pipeline_hashes: Sequence[ContentHash],) -> NoneBuild-probe: collect right input, then stream left through a hash lookup.
Phase 1 — Build: collect all rows from the right (filter) channel and
index them by the common-key values. The left channel is read at the
same time (ITL-1075): the left rows that arrive before the right input
ends are kept, at most the whole left input, as the sync path and
MergeJoin's barrier path hold their inputs. Reading the left input
only after the right one ended deadlocked a run in which one node fed
both inputs.
Phase 2 — Probe: the kept left rows, then the rest of the left input
one at a time with backpressure; for each row whose common-key values
appear in the right-side index, emit immediately. Left rows keep their
arrival order.
Falls back to barrier mode when the right input is empty (schema cannot be inferred from data) or when there are no common keys.
Key types are checked (require_shared_arrow_type) on the first row
of each input, since channels carry no schema. When either input is
empty there is nothing to check, so a key type mismatch the sync path
reports gives an empty result here (as in Join's async path).
Parameters:
- inputs (
Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]]): The left and right readable input channels. - output (
WritableChannel[tuple[TagProtocol, DataProtocol]]): Writable channel for downstream consumption. - input_pipeline_hashes (
Sequence[ContentHash]): Not used: the left rows pass through with their system-tag names, so no column is named after an input's pipeline hash.
class · orcapod.core.operators.batch.Batch
class Batch(batch_size: int = 0, drop_partial_batch: bool = False, **kwargs)Bases: UnaryOperator
Group consecutive rows into batches of batch_size rows, list-wrapping them.
Batch has no key, so it keeps every tag, list-valued: those lists are what
tells a batch's members apart. Every user tag, data and source-info column
becomes a list with one element per member; system-tag columns fold to a scalar
digest and gain a ::{pipeline_hash} name suffix.
A data-tag (ITL-1017, developer decision of November 16, 2026: "Batch always simply takes anything and puts into list") is one column, so it becomes the same list in both roles and stays a data-tag. Its source info is one token per member, as for any data column: nothing is merged, so no token is folded.
Parameters:
- batch_size (
int) default0: Rows per batch;0puts every row in one batch. - drop_partial_batch (
bool) defaultFalse: Drop a last batch with fewer thanbatch_sizerows.
validate_unary_input
Section titled “validate_unary_input”method · orcapod.core.operators.batch.Batch.validate_unary_input
def validate_unary_input(stream: StreamProtocol) -> NoneBatch works on any input stream, so no validation is needed.
unary_static_process
Section titled “unary_static_process”method · orcapod.core.operators.batch.Batch.unary_static_process
def unary_static_process(stream: StreamProtocol) -> StreamProtocolGroup rows into fixed-size batches, list-wrapping their values.
Tag and data columns become list-valued; a data-tag, one column, becomes the same list in both roles and stays a data-tag. Source-info columns become list-valued too, one element per batch member. System-tag columns are folded to a scalar instead -- record identity hashes them directly, so they must not become lists.
Parameters:
- stream (
StreamProtocol): The upstream stream.
Returns:
StreamProtocol: A stream with one row per batch.
unary_output_schema
Section titled “unary_output_schema”method · orcapod.core.operators.batch.Batch.unary_output_schema
def unary_output_schema( stream: StreamProtocol, *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False,) -> tuple[Schema, Schema]Predict the batched output schemas without batching.
Every user tag, data, and source column becomes list[T], so a
data-tag is list[T] in both halves. System tag columns keep
their scalar type and gain a ::{pipeline_hash} name suffix.
Parameters:
- stream (
StreamProtocol): The upstream stream. - columns (
ColumnConfig| dict[str, Any] | None) defaultNone: Column inclusion config. - all_info (
bool) defaultFalse: Include all info columns.
Returns:
async_execute
Section titled “async_execute”method · orcapod.core.operators.batch.Batch.async_execute
async def async_execute( inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], *, input_pipeline_hashes: Sequence[ContentHash],) -> NoneStreaming batch: emit full batches as they accumulate.
When batch_size > 0, each group of batch_size rows is
materialized and emitted immediately, allowing downstream consumers
to start processing before all input is consumed. When
batch_size == 0 (batch everything), falls back to barrier mode.
Parameters:
- inputs (
Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]]): The one readable input channel. - output (
WritableChannel[tuple[TagProtocol, DataProtocol]]): Writable channel for downstream consumption. - input_pipeline_hashes (
Sequence[ContentHash]): The input stream's pipeline hash, as a one-element sequence. Every rebuilt batch reports it, so the folded system-tag names end in it as on the sync path (ITL-906).
Raises:
ValueError: Ifinput_pipeline_hashesdoes not have one hash per input.
to_config
Section titled “to_config”method · orcapod.core.operators.batch.Batch.to_config
def to_config() -> dict[str, Any]Serialize this Batch operator to a config dict.
Returns:
dict[str, Any]: A dict withclass_name,module_path, andconfigkeys,dict[str, Any]: whereconfigcontainsbatch_sizeanddrop_partial_batch.
GroupBy
Section titled “GroupBy”class · orcapod.core.operators.group_by.GroupBy
class GroupBy(by: str | Collection[str], **kwargs: Any)Bases: UnaryOperator
Reduce rows sharing a key into one row with list-valued data members.
This is the only many->one operator. Every other operator preserves one
row per tag; GroupBy collapses N rows into one, which is what lets a
downstream pod receive a whole group at once (for example, all of a
recording session's per-probe result parquets).
Given tags (subject, date, probe) and data (path), grouping by
["subject", "date"] emits one row per distinct (subject, date):
subjectanddatestay scalar tag columns, in the input's tag order; they are the only tag columns of the outputprobeis dropped (ITL-1070): a list of the members' values is not a key for the group, as SQL drops a non-key, non-aggregated column afterGROUP BY. A data-tagprobeloses its tag role only: its data column stays and becomes list-valued likepathpathbecomes list-valued- the
_source_*column of each data column becomes list-valued, one element per member, except that of a data-tag group key: its value is the merge of the members' equal values, so its source info is one token, thegroup_v1fold of the members' tokens in member order (ITL-1017) - system-tag columns fold to a scalar digest and gain a
::{pipeline_hash}name suffix
No data element ever comes from a tag value: GroupBy uses tags to form
groups, but the data columns of the output are exactly the data columns of
the input. The members' values of a dropped plain tag are not carried
anywhere. If a downstream step needs to know which member each element
came from, make that value a data column before grouping: declare the tag
a data-tag at the source (the default), declare it as data, or use
DerivedSource(origin, tags_as_data={...}).
Members are sorted by the input's non-group-key tag values, with record_id
appended as a final tiebreaker, so the emitted lists are stable across
runs. The list of a data-tag non-key is therefore in its own value
order, and every other data list is aligned with it. This matters because orcapod hashes those lists to build the cache
key -- an unsorted list would make an identical member set hash differently
and trigger a spurious recompute. Groups themselves are emitted in
group-key order for the same reason: a reordered input must produce an
identical output.
Tag tuples are expected to be unique within a stream, but nothing enforces
that. If the input does contain duplicate tag tuples, the members sharing
a tuple are separated only by record_id, which is fixed when the source
is materialized -- so their order is stable for a given source table, but
undefined if the source table itself is permuted.
A grouped stream can be grouped again on a coarser key: its data lists nest
one level deeper (list[list[T]]), as source info does. The members of
the second grouping are ordered by the first grouping's keys that are not
in the coarser key (scalar tags of the grouped stream), then by
record_id; a tag the first grouping dropped no longer takes part. A
list-valued tag of the input can order members but cannot be a grouping
key.
Contrast with Batch, which partitions by row count for throughput. It
has no key, so it keeps every tag column, list-valued: those lists are
what tells a batch's members apart.
Parameters:
- by (
str | Collection[str]): Tag column names to group on, or one name as a string (a bare string is one name, never its characters). Must be non-empty, free of duplicates, and must all be scalar tag columns of the input stream.
to_config
Section titled “to_config”method · orcapod.core.operators.group_by.GroupBy.to_config
def to_config() -> dict[str, Any]Serialize this GroupBy operator to a config dict.
by is emitted as a list rather than a tuple so the config stays
JSON-serializable; __init__ normalizes it back to a tuple.
Returns:
dict[str, Any]: A dict withclass_name,module_path, andconfigkeys,dict[str, Any]: whereconfigcontainsby.
validate_unary_input
Section titled “validate_unary_input”method · orcapod.core.operators.group_by.GroupBy.validate_unary_input
def validate_unary_input(stream: StreamProtocol) -> NoneVerify every grouping column is a scalar tag column of the input.
Parameters:
- stream (
StreamProtocol): The upstream stream to validate.
Raises:
InputValidationError: If any name inbyis not a tag column, or names a list-valued tag column.
unary_static_process
Section titled “unary_static_process”method · orcapod.core.operators.group_by.GroupBy.unary_static_process
def unary_static_process(stream: StreamProtocol) -> StreamProtocolPartition rows by group key and emit one row per group.
Parameters:
- stream (
StreamProtocol): The upstream stream.
Returns:
StreamProtocol: A stream with one row per distinct group-key tuple, whose tagStreamProtocol: columns are the group keys, scalar, in the input's tag order.
unary_output_schema
Section titled “unary_output_schema”method · orcapod.core.operators.group_by.GroupBy.unary_output_schema
def unary_output_schema( stream: StreamProtocol, *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False,) -> tuple[Schema, Schema]Predict the grouped output schemas without grouping.
Parameters:
- stream (
StreamProtocol): The upstream stream. - columns (
ColumnConfig| dict[str, Any] | None) defaultNone: Column inclusion config. - all_info (
bool) defaultFalse: Include all info columns.
Returns:
Schema: A(tag_schema, data_schema)tuple. The tag schema holds theSchema: group keys, scalar, in the input's tag order, and the system tags,tuple[Schema,Schema]: scalar under their suffixed names; every other user tag istuple[Schema,Schema]: dropped. The data schema holds the input's data columns astuple[Schema,Schema]:list[T](a data-tag non-key among them), except that atuple[Schema,Schema]: data-tag key stays scalar.
SelectTagColumns
Section titled “SelectTagColumns”class · orcapod.core.operators.column_selection.SelectTagColumns
class SelectTagColumns(columns: str | Collection[str], strict: bool = True, **kwargs)Bases: UnaryOperator
Operator that selects specified columns from a stream.
to_config
Section titled “to_config”method · orcapod.core.operators.column_selection.SelectTagColumns.to_config
def to_config() -> dict[str, Any]Serialize this SelectTagColumns operator to a config dict.
Returns:
dict[str, Any]: A dict withclass_name,module_path, andconfigkeys,dict[str, Any]: whereconfigcontainscolumnsandstrict.
validate_unary_input
Section titled “validate_unary_input”method · orcapod.core.operators.column_selection.SelectTagColumns.validate_unary_input
def validate_unary_input(stream: StreamProtocol) -> NoneThis method should be implemented by subclasses to validate the inputs to the operator. It takes two streams as input and raises an error if the inputs are not valid.
async_execute
Section titled “async_execute”method · orcapod.core.operators.column_selection.SelectTagColumns.async_execute
async def async_execute( inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], *, input_pipeline_hashes: Sequence[ContentHash],) -> NoneStreaming: select tag columns per row without materializing.
Parameters:
- inputs (
Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]]): The one readable input channel. - output (
WritableChannel[tuple[TagProtocol, DataProtocol]]): Writable channel for downstream consumption. - input_pipeline_hashes (
Sequence[ContentHash]): Not used: this operator keeps system-tag names as they are, so no column is named after an input's pipeline hash.
SelectDataColumns
Section titled “SelectDataColumns”class · orcapod.core.operators.column_selection.SelectDataColumns
class SelectDataColumns(columns: str | Collection[str], strict: bool = True, **kwargs)Bases: UnaryOperator
Operator that selects specified columns from a stream.
to_config
Section titled “to_config”method · orcapod.core.operators.column_selection.SelectDataColumns.to_config
def to_config() -> dict[str, Any]Serialize this SelectDataColumns operator to a config dict.
Returns:
dict[str, Any]: A dict withclass_name,module_path, andconfigkeys,dict[str, Any]: whereconfigcontainscolumnsandstrict.
validate_unary_input
Section titled “validate_unary_input”method · orcapod.core.operators.column_selection.SelectDataColumns.validate_unary_input
def validate_unary_input(stream: StreamProtocol) -> NoneThis method should be implemented by subclasses to validate the inputs to the operator. It takes two streams as input and raises an error if the inputs are not valid.
async_execute
Section titled “async_execute”method · orcapod.core.operators.column_selection.SelectDataColumns.async_execute
async def async_execute( inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], *, input_pipeline_hashes: Sequence[ContentHash],) -> NoneStreaming: select data columns per row without materializing.
Parameters:
- inputs (
Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]]): The one readable input channel. - output (
WritableChannel[tuple[TagProtocol, DataProtocol]]): Writable channel for downstream consumption. - input_pipeline_hashes (
Sequence[ContentHash]): Not used: this operator keeps system-tag names as they are, so no column is named after an input's pipeline hash.
DropTagColumns
Section titled “DropTagColumns”class · orcapod.core.operators.column_selection.DropTagColumns
class DropTagColumns(columns: str | Collection[str], strict: bool = True, **kwargs)Bases: UnaryOperator
Operator that drops specified columns from a stream.
to_config
Section titled “to_config”method · orcapod.core.operators.column_selection.DropTagColumns.to_config
def to_config() -> dict[str, Any]Serialize this DropTagColumns operator to a config dict.
Returns:
dict[str, Any]: A dict withclass_name,module_path, andconfigkeys,dict[str, Any]: whereconfigcontainscolumnsandstrict.
validate_unary_input
Section titled “validate_unary_input”method · orcapod.core.operators.column_selection.DropTagColumns.validate_unary_input
def validate_unary_input(stream: StreamProtocol) -> NoneThis method should be implemented by subclasses to validate the inputs to the operator. It takes two streams as input and raises an error if the inputs are not valid.
async_execute
Section titled “async_execute”method · orcapod.core.operators.column_selection.DropTagColumns.async_execute
async def async_execute( inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], *, input_pipeline_hashes: Sequence[ContentHash],) -> NoneStreaming: drop tag columns per row without materializing.
Parameters:
- inputs (
Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]]): The one readable input channel. - output (
WritableChannel[tuple[TagProtocol, DataProtocol]]): Writable channel for downstream consumption. - input_pipeline_hashes (
Sequence[ContentHash]): Not used: this operator keeps system-tag names as they are, so no column is named after an input's pipeline hash.
DropDataColumns
Section titled “DropDataColumns”class · orcapod.core.operators.column_selection.DropDataColumns
class DropDataColumns(columns: str | Collection[str], strict: bool = True, **kwargs)Bases: UnaryOperator
Operator that drops specified columns from a stream.
to_config
Section titled “to_config”method · orcapod.core.operators.column_selection.DropDataColumns.to_config
def to_config() -> dict[str, Any]Serialize this DropDataColumns operator to a config dict.
Returns:
dict[str, Any]: A dict withclass_name,module_path, andconfigkeys,dict[str, Any]: whereconfigcontainscolumnsandstrict.
validate_unary_input
Section titled “validate_unary_input”method · orcapod.core.operators.column_selection.DropDataColumns.validate_unary_input
def validate_unary_input(stream: StreamProtocol) -> NoneThis method should be implemented by subclasses to validate the inputs to the operator. It takes two streams as input and raises an error if the inputs are not valid.
async_execute
Section titled “async_execute”method · orcapod.core.operators.column_selection.DropDataColumns.async_execute
async def async_execute( inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], *, input_pipeline_hashes: Sequence[ContentHash],) -> NoneStreaming: drop data columns per row without materializing.
Parameters:
- inputs (
Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]]): The one readable input channel. - output (
WritableChannel[tuple[TagProtocol, DataProtocol]]): Writable channel for downstream consumption. - input_pipeline_hashes (
Sequence[ContentHash]): Not used: this operator keeps system-tag names as they are, so no column is named after an input's pipeline hash.
DropColumns
Section titled “DropColumns”class · orcapod.core.operators.column_selection.DropColumns
class DropColumns(columns: str | Collection[str], strict: bool = True, **kwargs)Bases: _TagAndDataColumnsOperator
Drop the named columns, whether each is a tag or a data column (ITL-1084).
For tag names only it gives the rows of DropTagColumns, for data names only
those of DropDataColumns. A dropped data column's source-info column goes
with it; system tags are kept.
Parameters:
- columns (
str | Collection[str]): One column name or a collection of names. - strict (
bool) defaultTrue: WhenTrue(the default), a name that is neither a tag nor a data column of the input raisesInputValidationError; whenFalseit is ignored.
Raises:
InputValidationError: When called on a stream whose data columns would all be dropped: a stream needs at least one data column.
SelectColumns
Section titled “SelectColumns”class · orcapod.core.operators.column_selection.SelectColumns
class SelectColumns(columns: str | Collection[str], strict: bool = True, **kwargs)Bases: _TagAndDataColumnsOperator
Keep exactly the named columns, tag or data; drop every other one (ITL-1084).
Every user tag and data column not named is dropped, with the source-info column of each dropped data column; system tags are kept. Naming no tag is allowed (the result has no tag column).
Parameters:
- columns (
str | Collection[str]): One column name or a collection of names. - strict (
bool) defaultTrue: WhenTrue(the default), a name that is neither a tag nor a data column of the input raisesInputValidationError; whenFalseit is ignored.
Raises:
InputValidationError: When called on a stream none of whose data columns is named: a stream needs at least one data column.
MapTags
Section titled “MapTags”class · orcapod.core.operators.mappers.MapTags
class MapTags( name_map: Mapping[str, str], drop_unmapped: bool = False, strict: bool = True, tag_only: bool = False, **kwargs,)Bases: UnaryOperator
Rename tag columns.
Data-tags (developer, November 12, 2026, ITL-1023): renaming a
data-tag renames its one column in both roles, so the data-tag
survives under the new name. With tag_only=True only the tag is renamed:
the data column keeps its name and the renamed tag is a plain tag with the
same values. A plain tag is renamed as before. A data-tag dropped as
unmapped keeps its data column.
Parameters:
- name_map (
Mapping[str, str]): Each tag column to rename, mapped to its new name. - drop_unmapped (
bool) defaultFalse: WhenTrue, the tag columns not inname_mapare dropped. - strict (
bool) defaultTrue: WhenTrue(the default), a key ofname_mapthat is not a tag column of the input raisesInputValidationError; whenFalseit is ignored. Part of the operator's identity, as forSelectTagColumnsand the other column operators. - tag_only (
bool) defaultFalse: Rename data-tags in the tag role only.
to_config
Section titled “to_config”method · orcapod.core.operators.mappers.MapTags.to_config
def to_config() -> dict[str, Any]Serialize this MapTags operator to a config dict.
Returns:
dict[str, Any]: A dict withclass_name,module_path, andconfigkeys,dict[str, Any]: whereconfigcontainsname_map,drop_unmappedanddict[str, Any]:strict.
validate_unary_input
Section titled “validate_unary_input”method · orcapod.core.operators.mappers.MapTags.validate_unary_input
def validate_unary_input(stream: StreamProtocol) -> NoneThis method should be implemented by subclasses to validate the inputs to the operator. It takes two streams as input and raises an error if the inputs are not valid.
async_execute
Section titled “async_execute”method · orcapod.core.operators.mappers.MapTags.async_execute
async def async_execute( inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], *, input_pipeline_hashes: Sequence[ContentHash],) -> NoneStreaming: rename tag columns per row without materializing.
Parameters:
- inputs (
Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]]): The one readable input channel. - output (
WritableChannel[tuple[TagProtocol, DataProtocol]]): Writable channel for downstream consumption. - input_pipeline_hashes (
Sequence[ContentHash]): Not used: this operator keeps system-tag names as they are, so no column is named after an input's pipeline hash.
MapData
Section titled “MapData”class · orcapod.core.operators.mappers.MapData
class MapData( name_map: Mapping[str, str], drop_unmapped: bool = False, strict: bool = True, **kwargs,)Bases: UnaryOperator
Rename data columns.
Renaming data is always a data-only move (developer, November 12, 2026,
ITL-1023): renaming the data column of a data-tag x gives the new
data column and leaves x a plain tag with the same values.
Parameters:
- name_map (
Mapping[str, str]): Each data column to rename, mapped to its new name. - drop_unmapped (
bool) defaultFalse: WhenTrue, the data columns not inname_mapare dropped. - strict (
bool) defaultTrue: WhenTrue(the default), a key ofname_mapthat is not a data column of the input raisesInputValidationError; whenFalseit is ignored. Part of the operator's identity, as forSelectTagColumnsand the other column operators.
to_config
Section titled “to_config”method · orcapod.core.operators.mappers.MapData.to_config
def to_config() -> dict[str, Any]Serialize this MapData operator to a config dict.
Returns:
dict[str, Any]: A dict withclass_name,module_path, andconfigkeys,dict[str, Any]: whereconfigcontainsname_map,drop_unmappedanddict[str, Any]:strict.
async_execute
Section titled “async_execute”method · orcapod.core.operators.mappers.MapData.async_execute
async def async_execute( inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], *, input_pipeline_hashes: Sequence[ContentHash],) -> NoneStreaming: rename data columns per row without materializing.
Parameters:
- inputs (
Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]]): The one readable input channel. - output (
WritableChannel[tuple[TagProtocol, DataProtocol]]): Writable channel for downstream consumption. - input_pipeline_hashes (
Sequence[ContentHash]): Not used: this operator keeps system-tag names as they are, so no column is named after an input's pipeline hash.
MapColumns
Section titled “MapColumns”class · orcapod.core.operators.mappers.MapColumns
class MapColumns( name_map: Mapping[str, str], drop_unmapped: bool = False, strict: bool = True, **kwargs,)Bases: UnaryOperator
Rename columns, whether each is a tag or a data column (ITL-1160).
The counterpart of SelectColumns and DropColumns for renaming: one
step renames tag and data columns alike. A renamed tag stays a tag and a
renamed data column stays data; a renamed data column's source-info column
is renamed with it. System tags pass through, the stream's column order is
kept, and, as for SelectColumns, no data meta (__) column of the
input is carried.
Parameters:
- name_map (
Mapping[str, str]): Each column to rename, tag or data, mapped to its new name. - drop_unmapped (
bool) defaultFalse: WhenTrue, every user tag and data column not inname_mapis dropped (with the source-info column of a dropped data column); system tags are kept. - strict (
bool) defaultTrue: WhenTrue(the default), a key ofname_mapthat is neither a tag nor a data column of the input raisesInputValidationError; whenFalseit is ignored. Part of the operator's identity, as forSelectColumns.
Raises:
InputValidationError: When called on a stream where a new name is taken by a column that is not renamed, two columns get the same new name, or (withdrop_unmapped) no data column would remain.
to_config
Section titled “to_config”method · orcapod.core.operators.mappers.MapColumns.to_config
def to_config() -> dict[str, Any]Serialize this MapColumns operator to a config dict.
Returns:
dict[str, Any]: A dict withclass_name,module_path, andconfigkeys,dict[str, Any]: whereconfigcontainsname_map,drop_unmappedanddict[str, Any]:strict.
validate_unary_input
Section titled “validate_unary_input”method · orcapod.core.operators.mappers.MapColumns.validate_unary_input
def validate_unary_input(stream: StreamProtocol) -> NoneValidate the names against the input stream.
Raises:
InputValidationError: As_validate_columnsdescribes.
async_execute
Section titled “async_execute”method · orcapod.core.operators.mappers.MapColumns.async_execute
async def async_execute( inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], *, input_pipeline_hashes: Sequence[ContentHash],) -> NoneStreaming: rename (and drop) the columns per row without materializing.
The names are validated against the first row, as validate_unary_input
validates them against the stream.
Parameters:
- inputs (
Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]]): The one readable input channel. - output (
WritableChannel[tuple[TagProtocol, DataProtocol]]): Writable channel for downstream consumption. - input_pipeline_hashes (
Sequence[ContentHash]): Not used: this operator keeps system-tag names as they are, so no column is named after an input's pipeline hash.
AddTags
Section titled “AddTags”class · orcapod.core.operators.add_tags.AddTags
class AddTags(tags: Mapping[str, TagConstant], **kwargs: Any)Bases: UnaryOperator
Add tags with constant values to every row of a stream.
Each new tag is a plain tag column holding the same value on every row. A tag is not a data value, so an operator may add one (developer, November 16, 2026, ITL-1196); data columns, their values and their source info pass through unchanged, and no data column is added. System tags keep their names and values (name-preserving), so record ids are unchanged.
The new tags follow the input's tags, in the order given. A value's exact
Python type is the tag's type, mapped to Arrow by the data context's type
converter: bool to bool, int to int64, float to
double, str to large_string, bytes to large_binary. Every
new tag is non-nullable.
The operator's identity is its tags, names and values in order, so two different values are two operators with two stored paths.
Data-tags (ITL-1023): the input's data-tags stay data-tags, one column in both roles with its values and source info; the new tags are plain. A new tag named like any column of the input, a data-tag included, is refused.
Parameters:
- tags (
Mapping[str, TagConstant]): Each new tag's name mapped to its value.
Raises:
ValueError: When no tag is given, a name is empty, or a value is out of range (see_checked_value).TypeError: When a name is not a string or a value's type is not accepted.InputValidationError: When a name is reserved (meta or system prefix,::separator).
property · orcapod.core.operators.add_tags.AddTags.tags
Type: Mapping[str, TagConstant]
The new tags, names mapped to values, in order; read-only.
Read-only so the identity, the saved config and the cached constants can never disagree.
to_config
Section titled “to_config”method · orcapod.core.operators.add_tags.AddTags.to_config
def to_config() -> dict[str, Any]Serialize this AddTags operator to a config dict.
Each tag is saved with its type token and its value; a bytes value
as hex text.
Returns:
dict[str, Any]: A dict withclass_name,module_pathandconfigkeys, wheredict[str, Any]:configholdstags, a list of{"name", "type", "value"}.
from_config
Section titled “from_config”classmethod · orcapod.core.operators.add_tags.AddTags.from_config
def from_config(config: dict[str, Any]) -> AddTagsReconstruct an AddTags operator from a config dict.
Parameters:
- config (
dict[str, Any]): Dict as returned byto_config().
Returns:
AddTags: A newAddTagsinstance.
Raises:
ValueError: When a saved tag names an unknown type.
unary_static_process
Section titled “unary_static_process”method · orcapod.core.operators.add_tags.AddTags.unary_static_process
def unary_static_process(stream: StreamProtocol) -> StreamProtocolReturn the stream's rows with the new tags added.
Parameters:
- stream (
StreamProtocol): The upstream stream.
Returns:
StreamProtocol: A stream over the same rows, in order, with the new tag columns afterStreamProtocol: the input's tags. A result with no rows has the same schema.
async_execute
Section titled “async_execute”method · orcapod.core.operators.add_tags.AddTags.async_execute
async def async_execute( inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], *, input_pipeline_hashes: Sequence[ContentHash],) -> NoneStreaming: add the new tags to each row as it arrives.
Each row's tag table goes through the rule of the sync path
(_with_constants); its data passes through unchanged.
Parameters:
- inputs (
Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]]): Single-element sequence of readable channels. - output (
WritableChannel[tuple[TagProtocol, DataProtocol]]): Channel to send the rows to. - input_pipeline_hashes (
Sequence[ContentHash]): Not used: this operator keeps system-tag names as they are, so no column is named after an input's pipeline hash.
PolarsFilter
Section titled “PolarsFilter”class · orcapod.core.operators.filters.PolarsFilter
class PolarsFilter( predicates: Collection[pl_type.IntoExprColumn | Iterable[pl_type.IntoExprColumn] | bool | list[bool] | np.ndarray[Any, Any]] = (), constraints: Mapping[str, Any] | None = None, **kwargs,)Bases: UnaryOperator
Operator that applies Polars filtering to a stream
validate_unary_input
Section titled “validate_unary_input”method · orcapod.core.operators.filters.PolarsFilter.validate_unary_input
def validate_unary_input(stream: StreamProtocol) -> NoneThis method should be implemented by subclasses to validate the inputs to the operator. It takes two streams as input and raises an error if the inputs are not valid.
to_config
Section titled “to_config”method · orcapod.core.operators.filters.PolarsFilter.to_config
def to_config() -> dict[str, Any]Serialize this PolarsFilter operator to a config dict.
Polars Expr predicates are serialized to JSON strings when possible.
If any predicate cannot be serialized, reconstructable is set to
False and predicates is set to None.
Returns:
dict[str, Any]: A dict withclass_name,module_path, andconfigkeys.
from_config
Section titled “from_config”classmethod · orcapod.core.operators.filters.PolarsFilter.from_config
def from_config(config: dict[str, Any]) -> PolarsFilterReconstruct a PolarsFilter from a config dict.
Parameters:
- config (
dict[str, Any]): Dict as returned byto_config().
Returns:
PolarsFilter: A newPolarsFilterinstance.
Raises:
NotImplementedError: IfreconstructableisFalse.
class · orcapod.core.operators.pick.Pick
class Pick( column: str, key: str, out: str | None = None, fail_on_miss: bool = False, **kwargs: Any,)Bases: UnaryOperator
Extract a value from a struct- or dict-typed data column by key.
For dict[K, V] columns the lookup is per-packet; if the key is absent
the packet is skipped (or an error is raised when fail_on_miss=True).
For extension-type struct columns (Pydantic, dataclass) the field existence is validated at build time and guaranteed at runtime.
The extracted column keeps the value's Arrow type (dict[str, int16]
yields int16). Its nullability follows the predicted output type, which
is not Optional for dict values, so the column is NOT NULL. A value that
is None therefore raises ValueError (whatever fail_on_miss is),
naming the operator, the column and the row's tag.
Parameters:
- column (
str): Name of the data column to project into. - key (
str): Dict key or struct field name to extract. - out (
str | None) defaultNone: Output column name.None(default) replacescolumnin-place; a string adds a new column alongside the original. - fail_on_miss (
bool) defaultFalse: IfTrue, raiseRuntimeErrorwhen the key is absent in a packet instead of skipping. Excluded fromidentity_structure— miss-handling does not affect functional output semantics. See ITL-439.
validate_unary_input
Section titled “validate_unary_input”method · orcapod.core.operators.pick.Pick.validate_unary_input
def validate_unary_input(stream: StreamProtocol) -> NoneValidate input schema and resolve output type.
Parameters:
- stream (
StreamProtocol): The upstream stream to validate.
Raises:
InputValidationError: Ifcolumnis missing,outcollides with an existing column, or the column type does not support pick (including a logical type withoutpick_field, such asnumpy.ndarray).
unary_output_schema
Section titled “unary_output_schema”method · orcapod.core.operators.pick.Pick.unary_output_schema
def unary_output_schema( stream: StreamProtocol, *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False,) -> tuple[Schema, Schema]Return the (tag, data) schemas for the output stream.
Parameters:
- stream (
StreamProtocol): The upstream stream. - columns (
ColumnConfig| dict[str, Any] | None) defaultNone: Column inclusion config passed through tooutput_schema. - all_info (
bool) defaultFalse: Whether to include all info columns.
Returns:
unary_static_process
Section titled “unary_static_process”method · orcapod.core.operators.pick.Pick.unary_static_process
def unary_static_process(stream: StreamProtocol) -> StreamProtocolProcess the full stream packet by packet.
Iterates stream.iter_data(), applies _process_one per packet,
and packs surviving packets back into a stream. Skipped packets
(key absent, fail_on_miss=False) are silently dropped.
Parameters:
- stream (
StreamProtocol): The upstream stream.
Returns:
StreamProtocol: A new stream with the projection applied. If all packets wereStreamProtocol: skipped, an empty stream with the correct output schema is returned.
Raises:
RuntimeError: Iffail_on_miss=Trueand any packet's dict lacksself.key.
async_execute
Section titled “async_execute”method · orcapod.core.operators.pick.Pick.async_execute
async def async_execute( inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], *, input_pipeline_hashes: Sequence[ContentHash],) -> NoneProcess packets one at a time as they arrive (true streaming).
Each packet is processed independently via _process_one and
forwarded immediately to the output channel — no buffering required.
Parameters:
- inputs (
Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]]): Single-element sequence of readable channels. - output (
WritableChannel[tuple[TagProtocol, DataProtocol]]): Channel to send transformed packets to. - input_pipeline_hashes (
Sequence[ContentHash]): Not used: this operator keeps system-tag names as they are, so no column is named after an input's pipeline hash.
class · orcapod.core.operators.index.Index
class Index( column: str, i: int, out: str | None = None, fail_on_miss: bool = False, **kwargs: Any,)Bases: UnaryOperator
Extract an element from a list-typed data column by position.
The list length is per-packet data, so bounds are not checked at build
time. Out-of-bounds access causes the packet to be skipped (or an error
when fail_on_miss=True). Negative indices follow Python semantics
(-1 is the last element).
The extracted column keeps the list's element Arrow type (list<int8>
yields int8). Its nullability follows the predicted output type, which
is not Optional for list elements, so the column is NOT NULL. An
element that is None therefore raises ValueError (whatever
fail_on_miss is), naming the operator, the column and the row's tag.
Parameters:
- column (
str): Name of the data column to project into. - i (
int): Position to extract. Negative indices follow Python semantics. - out (
str | None) defaultNone: Output column name.None(default) replacescolumnin-place; a string adds a new column alongside the original. - fail_on_miss (
bool) defaultFalse: IfTrue, raiseRuntimeErroron out-of-bounds instead of skipping. Excluded fromidentity_structure. See ITL-439.
validate_unary_input
Section titled “validate_unary_input”method · orcapod.core.operators.index.Index.validate_unary_input
def validate_unary_input(stream: StreamProtocol) -> NoneValidate input schema and resolve output element type.
Parameters:
- stream (
StreamProtocol): The upstream stream to validate.
Raises:
InputValidationError: Ifcolumnis missing,outcollides, or the column type is not a supported index target.NotImplementedError: If the column holds an extension type whoseindex_elementis not yet implemented.
unary_output_schema
Section titled “unary_output_schema”method · orcapod.core.operators.index.Index.unary_output_schema
def unary_output_schema( stream: StreamProtocol, *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False,) -> tuple[Schema, Schema]Return the (tag, data) schemas for the output stream.
Parameters:
- stream (
StreamProtocol): The upstream stream. - columns (
ColumnConfig| dict[str, Any] | None) defaultNone: Column inclusion config. - all_info (
bool) defaultFalse: Include all info columns.
Returns:
unary_static_process
Section titled “unary_static_process”method · orcapod.core.operators.index.Index.unary_static_process
def unary_static_process(stream: StreamProtocol) -> StreamProtocolProcess the full stream packet by packet.
Iterates stream.iter_data(), applies _process_one per packet,
and packs surviving packets back into a stream. Skipped packets
(out-of-bounds, fail_on_miss=False) are silently dropped.
Parameters:
- stream (
StreamProtocol): The upstream stream.
Returns:
StreamProtocol: A new stream with the projection applied. If all packets wereStreamProtocol: skipped, an empty stream with the correct output schema is returned.
Raises:
RuntimeError: Iffail_on_miss=Trueand any packet is out-of-bounds.ValueError: If an extracted element isNonebut the predicted output type is not Optional.
async_execute
Section titled “async_execute”method · orcapod.core.operators.index.Index.async_execute
async def async_execute( inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], *, input_pipeline_hashes: Sequence[ContentHash],) -> NoneProcess packets one at a time as they arrive (true streaming).
Each packet is processed independently via _process_one and
forwarded immediately to the output channel — no buffering required.
Parameters:
- inputs (
Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]]): Single-element sequence of readable channels. - output (
WritableChannel[tuple[TagProtocol, DataProtocol]]): Channel to send transformed packets to. - input_pipeline_hashes (
Sequence[ContentHash]): Not used: this operator keeps system-tag names as they are, so no column is named after an input's pipeline hash.