Skip to content
This site documents the development version of Orcapod (the main branch), which may differ from the latest release. For the releases, see PyPI and the changelog.

Function Pods

Function pods wrap Python functions to transform individual data in a stream.

class · orcapod.core.function_pod.FunctionPod

class FunctionPod(
data_function: DataFunctionProtocol,
pod_config: PodConfig | None = None,
ctx_arg_name: str | None = None,
**kwargs,
)

Bases: _FunctionPodBase

classmethod · orcapod.core.function_pod.FunctionPod.from_fn

def from_fn(
fn: Callable,
output_keys: list[str] | str,
*,
ctx_arg_name: str | None = None,
name: str | None = None,
version: str = 'v0.0',
pod_config: PodConfig | None = None,
label: str | None = None,
**kwargs: Any,
) -> 'FunctionPod'

Construct a FunctionPod directly from a callable.

When ctx_arg_name is set, the named parameter is excluded from the pod's exposed input_data_schema and auto-injected with a per-row InvocationContext at process_data time. The full function signature (including ctx_arg_name) is retained in the underlying PythonDataFunction for correct identity hashing.

Parameters:

  • fn (Callable): The user function.
  • output_keys (list[str] | str): Output column key(s).
  • ctx_arg_name (str | None) default None: If set, exclude this parameter from the exposed input schema and inject an InvocationContext per row under this name.
  • name (str | None) default None: Optional canonical function name override.
  • version (str) default 'v0.0': Version string (default "v0.0", as for every entry point).
  • pod_config (PodConfig | None) default None: Optional per-pod config.
  • label (str | None) default None: Optional display label.
  • **kwargs (Any) default {}: Forwarded to _FunctionPodBase.__init__.

Returns:

  • 'FunctionPod': A new FunctionPod.

property · orcapod.core.function_pod.FunctionPod.pod_config

Type: PodConfig

Per-pod executor configuration.

method · orcapod.core.function_pod.FunctionPod.process

def process(*streams: StreamProtocol, label: str | None = None) -> FunctionPodStream

Invoke the data processor on the input stream(s).

Parameters:

  • *streams (StreamProtocol) default (): Input streams to process.
  • label (str | None) default None: Optional label for tracking.

Returns:

  • FunctionPodStream: A FunctionPodStream wrapping the computation.

method · orcapod.core.function_pod.FunctionPod.to_config

def to_config() -> dict[str, Any]

Serialize this function pod to a JSON-compatible config dict.

Returns:

  • dict[str, Any]: A JSON-serializable dict containing the URI, data function config,
  • dict[str, Any]: and pod config for this function pod.

classmethod · orcapod.core.function_pod.FunctionPod.from_config

def from_config(
config: dict[str, Any],
*,
fallback_to_proxy: bool = False,
) -> 'FunctionPod'

Reconstruct a FunctionPod from a config dict.

Parameters:

  • config (dict[str, Any]): A dict as produced by to_config.
  • fallback_to_proxy (bool) default False: If True and the data function cannot be resolved, use a DataFunctionProxy instead of raising.

Returns:

  • 'FunctionPod': A new FunctionPod instance.

function · orcapod.core.function_pod.function_pod

def function_pod(
output_keys: str | Sequence[str] | None = None,
function_name: str | None = None,
version: str = 'v0.0',
label: str | None = None,
ctx_arg: str | None = None,
result_database: ArrowDatabaseProtocol | None = None,
pod_cache_database: ArrowDatabaseProtocol | None = None,
executor: DataFunctionExecutorProtocol | None = None,
post_run_hooks: Sequence[PostRunHook] | None = None,
**kwargs: Any,
) -> Callable[..., CallableWithPodProtocol]

Decorator that attaches a FunctionPod as a pod attribute.

Parameters:

  • output_keys (str | Sequence[str] | None) default None: Keys for the function output(s).
  • function_name (str | None) default None: Name of the function pod; defaults to func.__name__.
  • version (str) default 'v0.0': Version string for the data function.
  • label (str | None) default None: Optional label for tracking.
  • result_database (ArrowDatabaseProtocol | None) default None: Optional database for data-level caching (wraps the data function in CachedDataFunction).
  • pod_cache_database (ArrowDatabaseProtocol | None) default None: Optional database for pod-level caching (wraps the pod in CachedFunctionPod, which caches at the process_data level using input data content hash).
  • executor (DataFunctionExecutorProtocol | None) default None: Optional executor for running the data function.
  • post_run_hooks (Sequence[PostRunHook] | None) default None: Optional list of post-run hooks to register on the pod after construction. Each entry is either a plain callable (PostRunPayload) -> None or a HookConfig.
  • **kwargs (Any) default {}: Forwarded to PythonDataFunction.

Returns:

  • Callable[..., CallableWithPodProtocol]: A decorator that adds a pod attribute to the wrapped function.

class · orcapod.side_effects.SideEffectPod

class SideEffectPod(
fn: Callable,
config: SideEffectPodConfig | None = None,
tracker_manager: TrackerManagerProtocol | None = None,
name: str | None = None,
label: str | None = None,
data_context: Any = None,
ctx_arg_name: str = 'ctx',
)

Bases: TraceableBase

A pipeline node whose primary purpose is a side effect.

Wraps a (data: T, ctx: InvocationContext) -> None callable. InvocationContext is always constructed and passed — it is part of the function contract. Callees that do not need it may ignore it (name the parameter _ctx by convention).

Returns a pass-through stream. When drop_on_failure=True, only successfully-delivered rows flow downstream.

In standalone mode (no PipelineJob), executes row-by-row via SideEffectPodStream with no invocation logging.

In pipeline mode, promoted to SideEffectJobNode at compile time, which adds DB-backed invocation logging and completion tracking. A job delivers only when it runs (job.run() or the node's run()); every read of the node returns the rows its last run passed on and delivers nothing (ITL-1091).

Parameters:

  • fn (Callable): A callable (data, ctx: InvocationContext) -> None.
  • config (SideEffectPodConfig | None) default None: Pod-level configuration. Defaults to SideEffectPodConfig().
  • tracker_manager (TrackerManagerProtocol | None) default None: Optional tracker manager override.
  • name (str | None) default None: Optional canonical function name used in uri and the invocation-log table path. Defaults to fn.__name__.
  • label (str | None) default None: Optional display label (separate from name).
  • data_context (Any) default None: Optional data context override.

property · orcapod.side_effects.SideEffectPod.ctx_arg_name

Type: str

Parameter name injected with InvocationContext on each call.

property · orcapod.side_effects.SideEffectPod.pod_config

Type: SideEffectPodConfig

Pod-level configuration.

property · orcapod.side_effects.SideEffectPod.canonical_function_name

Type: str

Human-readable function identifier, defaults to fn.__name__.

method · orcapod.side_effects.SideEffectPod.computed_label

def computed_label() -> str | None

Use the callable's __name__ as the default label.

property · orcapod.side_effects.SideEffectPod.uri

Type: tuple[str, ...]

Canonical URI for this pod: ("side_effects", canonical_function_name).

method · orcapod.side_effects.SideEffectPod.argument_symmetry

def argument_symmetry(streams: Collection[StreamProtocol]) -> Any

Single ordered input — return as an ordered tuple.

method · orcapod.side_effects.SideEffectPod.output_schema

def output_schema(
*streams: StreamProtocol,
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> tuple[Schema, Schema]

Return the input stream's schema unchanged (pass-through).

Parameters:

  • *streams (StreamProtocol) default (): Exactly one input stream.
  • columns (ColumnConfig | dict[str, Any] | None) default None: Optional column config.
  • all_info (bool) default False: Include all metadata columns.

Returns:

  • tuple[Schema, Schema]: The input stream's (tag_schema, data_schema) unchanged.

method · orcapod.side_effects.SideEffectPod.process

def process(*streams: StreamProtocol, label: str | None = None) -> SideEffectPodStream

Invoke the side-effect pod on the input stream.

Registers a SideEffectInvocation with the tracker manager (if inside a with PipelineJob(): block), then returns a SideEffectPodStream for standalone / lazy execution.

Parameters:

  • *streams (StreamProtocol) default (): Exactly one input stream.
  • label (str | None) default None: Optional label for the compiled node.

Returns:

  • SideEffectPodStream: A SideEffectPodStream.

function · orcapod.side_effects.side_effect_pod

def side_effect_pod(
fn: Callable | None = None,
*,
config: SideEffectPodConfig | None = None,
name: str | None = None,
ctx_arg_name: str = 'ctx',
) -> SideEffectPod | Callable[[Callable], SideEffectPod]

Decorator that wraps a callable as a SideEffectPod.

Supports both bare (@side_effect_pod) and parameterised (@side_effect_pod(config=...)) usage.

Parameters:

  • fn (Callable | None) default None: The callable to wrap (when used as a bare decorator).
  • config (SideEffectPodConfig | None) default None: Optional SideEffectPodConfig to apply.
  • name (str | None) default None: Optional canonical function name override (see SideEffectPod).
  • ctx_arg_name (str) default 'ctx': Name of the parameter that receives InvocationContext. Defaults to "ctx".

Returns:

function · orcapod.side_effects.sink_pod

def sink_pod(
fn: Callable | None = None,
*,
config: SideEffectPodConfig | None = None,
name: str | None = None,
ctx_arg_name: str = 'ctx',
) -> SideEffectPod | Callable[[Callable], SideEffectPod]

Decorator preset: track_completion=True, drop_on_failure=True.

Caller-supplied config fields override the presets. Supports both bare (@sink_pod) and parameterised (@sink_pod(config=...)) usage.

Parameters:

  • fn (Callable | None) default None: The callable to wrap (bare usage).
  • config (SideEffectPodConfig | None) default None: Optional config override.
  • name (str | None) default None: Optional canonical function name override (see SideEffectPod).
  • ctx_arg_name (str) default 'ctx': Name of the parameter that receives InvocationContext. Defaults to "ctx".

Returns:

function · orcapod.side_effects.tap_pod

def tap_pod(
fn: Callable | None = None,
*,
config: SideEffectPodConfig | None = None,
name: str | None = None,
ctx_arg_name: str = 'ctx',
) -> SideEffectPod | Callable[[Callable], SideEffectPod]

Decorator preset: track_completion=False, drop_on_failure=False.

Caller-supplied config fields override the presets. Supports both bare (@tap_pod) and parameterised (@tap_pod(config=...)) usage.

Parameters:

  • fn (Callable | None) default None: The callable to wrap (bare usage).
  • config (SideEffectPodConfig | None) default None: Optional config override.
  • name (str | None) default None: Optional canonical function name override (see SideEffectPod).
  • ctx_arg_name (str) default 'ctx': Name of the parameter that receives InvocationContext. Defaults to "ctx".

Returns:

class · orcapod.core.data_function.PythonDataFunction

class PythonDataFunction(
function: Callable[..., Any],
output_keys: str | Sequence[str] | None = None,
function_name: str | None = None,
version: str = 'v0.0',
input_schema: SchemaLike | None = None,
output_schema: SchemaLike | Sequence[type] | None = None,
label: str | None = None,
data_context: str | DataContext | None = None,
config: OrcapodConfig | None = None,
executor: PythonFunctionExecutorProtocol | None = None,
)

Bases: DataFunctionBase[PythonFunctionExecutorProtocol]

property · orcapod.core.data_function.PythonDataFunction.data_function_type_id

Type: str

Unique function type identifier.

property · orcapod.core.data_function.PythonDataFunction.canonical_function_name

Type: str

Human-readable function identifier.

method · orcapod.core.data_function.PythonDataFunction.get_function_variation_data

def get_function_variation_data() -> dict[str, Any]

Raw data defining function variation, stored with every result row.

function_version is the full version string the function was declared with ("v3.7"); only its major part is in the table path.

method · orcapod.core.data_function.PythonDataFunction.get_function_variation_data_schema

def get_function_variation_data_schema() -> Schema

Schema for the data returned by get_function_variation_data.

method · orcapod.core.data_function.PythonDataFunction.get_execution_data

def get_execution_data() -> dict[str, Any]

Raw data defining execution context.

method · orcapod.core.data_function.PythonDataFunction.get_execution_data_schema

def get_execution_data_schema() -> Schema

Schema for the data returned by get_execution_data.

property · orcapod.core.data_function.PythonDataFunction.input_data_schema

Type: Schema

Schema describing the input data this function accepts.

property · orcapod.core.data_function.PythonDataFunction.output_data_schema

Type: Schema

Schema describing the output data this function produces.

method · orcapod.core.data_function.PythonDataFunction.is_active

def is_active() -> bool

Return whether the function is active (will process data).

method · orcapod.core.data_function.PythonDataFunction.set_active

def set_active(active: bool = True) -> None

Set the active state. If False, call returns None for every data.

property · orcapod.core.data_function.PythonDataFunction.is_async

Type: bool

Return whether the wrapped function is an async coroutine function.

method · orcapod.core.data_function.PythonDataFunction.call

def call(
data: DataProtocol,
*,
logger: DataExecutionLoggerProtocol | None = None,
**kwargs: Any,
) -> DataProtocol | None

Process a single data, routing through the executor if one is set.

When an executor implementing PythonFunctionExecutorProtocol is set, the raw callable and call dict are handed to execute_callable which captures I/O and records to the logger.

Parameters:

  • data (DataProtocol): The input data to process.
  • logger (DataExecutionLoggerProtocol | None) default None: Optional logger for I/O capture.
  • **kwargs (Any) default {}: Extra keyword arguments merged with data.as_dict() before invoking the function. Pods use this to supply auto-provided arguments (e.g. InvocationContext) that are not part of the data schema.

method · orcapod.core.data_function.PythonDataFunction.async_call

async def async_call(
data: DataProtocol,
*,
logger: DataExecutionLoggerProtocol | None = None,
**kwargs: Any,
) -> DataProtocol | None

Async counterpart of call.

method · orcapod.core.data_function.PythonDataFunction.direct_call

def direct_call(data: DataProtocol, **kwargs: Any) -> DataProtocol | None

Execute the function on data synchronously (no executor path).

On user-function failure the exception is re-raised. For async functions, the coroutine is driven to completion via asyncio.run() (or a helper thread when already inside an event loop).

Parameters:

  • data (DataProtocol): The input data to process.
  • **kwargs (Any) default {}: Extra keyword arguments merged with data.as_dict() before invoking the function.

method · orcapod.core.data_function.PythonDataFunction.direct_async_call

async def direct_async_call(data: DataProtocol, **kwargs: Any) -> DataProtocol | None

Execute the function on data asynchronously (no executor path).

Async functions are await-ed directly. Sync functions are offloaded to a thread pool via run_in_executor. On failure, the exception is re-raised.

Parameters:

  • data (DataProtocol): The input data to process.
  • **kwargs (Any) default {}: Extra keyword arguments merged with data.as_dict() before invoking the function.

method · orcapod.core.data_function.PythonDataFunction.to_config

def to_config() -> dict[str, Any]

Serialize this data function to a JSON-compatible config dict.

Returns:

  • dict[str, Any]: A dict with data_function_type_id and a nested config
  • dict[str, Any]: containing enough information to reconstruct this instance via
  • dict[str, Any]: from_config.

classmethod · orcapod.core.data_function.PythonDataFunction.from_config

def from_config(config: dict[str, Any]) -> 'PythonDataFunction'

Reconstruct a PythonDataFunction by importing the callable.

Parameters:

  • config (dict[str, Any]): A dict as produced by to_config.

Returns:

  • 'PythonDataFunction': A new PythonDataFunction wrapping the imported callable.

Raises:

  • ImportError: If the module specified in config cannot be imported.
  • AttributeError: If the callable name does not exist in the module.

class · orcapod.core.data_function.CachedDataFunction

class CachedDataFunction(
data_function: DataFunctionProtocol,
result_database: ArrowDatabaseProtocol,
record_path_prefix: tuple[str, ...] = (),
**kwargs,
)

Bases: DataFunctionWrapper

Wrapper around a DataFunctionProtocol that caches results for identical input data.

Uses a shared ResultCache for lookup/store/conflict-resolution logic (same mechanism as CachedFunctionPod).

property · orcapod.core.data_function.CachedDataFunction.result_database

Type: ArrowDatabaseProtocol

The database this function's results are cached in.

method · orcapod.core.data_function.CachedDataFunction.set_auto_flush

def set_auto_flush(on: bool = True) -> None

Set auto-flush behavior. If True, the database flushes after each record.

property · orcapod.core.data_function.CachedDataFunction.record_path

Type: tuple[str, ...]

Return the path to the cached records in the result database (versioned).

method · orcapod.core.data_function.CachedDataFunction.call

def call(
data: DataProtocol,
*,
logger: DataExecutionLoggerProtocol | None = None,
skip_cache_lookup: bool = False,
skip_cache_insert: bool = False,
**kwargs: Any,
) -> DataProtocol | None

Look up cache keyed on data; on miss, call and store result.

**kwargs are forwarded to the wrapped data function's call so that auto-provided arguments (e.g. InvocationContext) reach the underlying function. They are intentionally excluded from the cache key — such arguments are deterministic per row and pod, so the same input data always produces the same output.

Raises:

  • InputValidationError: If data is of another data context than this function's; the wrapped function has not been called then, whatever skip_cache_lookup and skip_cache_insert are.

method · orcapod.core.data_function.CachedDataFunction.async_call

async def async_call(
data: DataProtocol,
*,
logger: DataExecutionLoggerProtocol | None = None,
skip_cache_lookup: bool = False,
skip_cache_insert: bool = False,
**kwargs: Any,
) -> DataProtocol | None

Async counterpart of call with cache check and recording.

Raises:

  • InputValidationError: If data is of another data context than this function's; the wrapped function has not been called then.

method · orcapod.core.data_function.CachedDataFunction.get_cached_output_for_data

def get_cached_output_for_data(input_data: DataProtocol) -> DataProtocol | None

Retrieve the cached output data for input_data.

If multiple cached entries exist, the most recent (by timestamp) wins.

Returns:

  • DataProtocol | None: The cached output data, or None if no entry was found.

method · orcapod.core.data_function.CachedDataFunction.record_data

def record_data(
input_data: DataProtocol,
output_data: DataProtocol,
skip_duplicates: bool = False,
) -> DataProtocol

Record the output data against the input data in the result store.

method · orcapod.core.data_function.CachedDataFunction.get_all_cached_outputs

def get_all_cached_outputs(include_system_columns: bool = False) -> 'pa.Table | None'

Return all cached records from the result database for this function.

Parameters:

  • include_system_columns (bool) default False: If True, include system columns (e.g. record_id) in the result.

Returns:

  • 'pa.Table | None': A PyArrow table of cached results, or None if empty.

class · orcapod.core.data_function.DataFunctionBase

class DataFunctionBase(
version: str = 'v0.0',
label: str | None = None,
data_context: str | DataContext | None = None,
config: OrcapodConfig | None = None,
executor: DataFunctionExecutorProtocol | None = None,
)

Bases: TraceableBase, Generic[E]

Abstract base class for DataFunctionProtocol.

Type-parameterized with the executor protocol E. Concrete subclasses that bind E (e.g. class Foo(DataFunctionBase[SomeProto])) get automatic isinstance validation in set_executor at class definition time via __init_subclass__.

method · orcapod.core.data_function.DataFunctionBase.computed_label

def computed_label() -> str | None

Return the canonical function name as the label if no explicit label is given.

property · orcapod.core.data_function.DataFunctionBase.output_data_schema_hash

Type: str

Return the hash of the output data schema as a bare hex digest.

The digest is part of the function uri and so of table paths and source-info tokens: a hash inside a name carries no label. It is computed lazily on first access and cached for subsequent calls.

Returns:

  • str: The hex digest of the output data schema hash, without a label.

property · orcapod.core.data_function.DataFunctionBase.data_function_type_id

Type: str

Unique function type identifier (e.g. "python.function.v1").

property · orcapod.core.data_function.DataFunctionBase.canonical_function_name

Type: str

Human-readable function identifier.

property · orcapod.core.data_function.DataFunctionBase.input_data_schema

Type: Schema

Schema describing the input data this function accepts.

property · orcapod.core.data_function.DataFunctionBase.accepts_arbitrary_keywords

Type: bool

Whether the function takes input columns beyond input_data_schema.

False for every data function kind today: the parameters are a fixed set (PythonDataFunction rejects **kwargs). A kind that passes every input column to its function overrides this to return True; a function pod then never drops a column for it (ITL-807).

property · orcapod.core.data_function.DataFunctionBase.output_data_schema

Type: Schema

Schema describing the output data this function produces.

method · orcapod.core.data_function.DataFunctionBase.get_function_variation_data

def get_function_variation_data() -> dict[str, Any]

Raw data defining function variation

method · orcapod.core.data_function.DataFunctionBase.get_execution_data

def get_execution_data() -> dict[str, Any]

Raw data defining execution context

method · orcapod.core.data_function.DataFunctionBase.get_function_variation_data_schema

def get_function_variation_data_schema() -> Schema

Schema for the data returned by get_function_variation_data.

method · orcapod.core.data_function.DataFunctionBase.get_execution_data_schema

def get_execution_data_schema() -> Schema

Schema for the data returned by get_execution_data.

property · orcapod.core.data_function.DataFunctionBase.executor

Type: E | None

Return the executor used to run this data function, or None for direct execution.

method · orcapod.core.data_function.DataFunctionBase.set_executor

def set_executor(executor: DataFunctionExecutorProtocol | None) -> None

Set or clear the executor, validating type compatibility.

Performs two checks:

  1. The executor supports this function's data_function_type_id.
  2. If the subclass bound E via Generic[E], the executor is an instance of the resolved protocol (checked once at assignment time, not in the hot path).

Raises:

  • TypeError: If executor fails either compatibility check.

method · orcapod.core.data_function.DataFunctionBase.call

def call(
data: DataProtocol,
*,
logger: DataExecutionLoggerProtocol | None = None,
**kwargs: Any,
) -> DataProtocol | None

Process a single data.

Base implementation calls direct_call. Subclasses that support executors (e.g. PythonDataFunction) override to route through the executor's execute_callable.

**kwargs are forwarded to direct_call and merged into the function call dict alongside the data fields. Callers (e.g. a pod) use this to supply auto-provided arguments such as an InvocationContext that are not part of the data schema.

method · orcapod.core.data_function.DataFunctionBase.async_call

async def async_call(
data: DataProtocol,
*,
logger: DataExecutionLoggerProtocol | None = None,
**kwargs: Any,
) -> DataProtocol | None

Asynchronously process a single data.

Base implementation calls direct_async_call. Subclasses that support executors override to route through the executor.

**kwargs are forwarded to direct_async_call and merged into the function call dict alongside the data fields.

method · orcapod.core.data_function.DataFunctionBase.direct_call

def direct_call(data: DataProtocol, **kwargs: Any) -> DataProtocol | None

Execute the function's native computation on data.

This is the method executors invoke. It bypasses executor routing and runs the computation directly. On user-function failure the exception is re-raised. Subclasses must implement this.

**kwargs are merged with data.as_dict() before the underlying callable is invoked.

method · orcapod.core.data_function.DataFunctionBase.direct_async_call

async def direct_async_call(data: DataProtocol, **kwargs: Any) -> DataProtocol | None

Asynchronous counterpart of direct_call.