Skip to content
This site documents the development version of Orcapod (the main branch), which may differ from the latest release. For the releases, see PyPI and the changelog.

Getting Started

This guide walks you through the core workflow in Orcapod: creating a data source, inspecting its stream, applying a transformation with a function pod, and examining the results.

Install Orcapod with uv, the recommended way: start a project with uv init (in a new directory), add Orcapod to it, and run your scripts with uv run:

Terminal window
uv init
uv add git+https://github.com/metamorphic-labs/orcapod-python
uv run python my_pipeline.py

In an environment not managed by uv, pip install git+https://github.com/metamorphic-labs/orcapod-python installs it instead.

Until 0.4.0 is released, these commands install the development version from the repository: PyPI has no current version (0.3.0 was withdrawn, and a plain uv add orcapod would install 0.1.4). Once 0.4.0 is out, uv add orcapod (or pip install orcapod) installs it.

A source is the entry point for data in an Orcapod pipeline. The simplest way to get started is with DictSource, which accepts a list of dictionaries:

from orcapod.sources import DictSource
source = DictSource(
data=[
{"experiment": "exp_001", "temperature": 20.5, "pressure": 1.01},
{"experiment": "exp_002", "temperature": 22.3, "pressure": 0.98},
{"experiment": "exp_003", "temperature": 19.8, "pressure": 1.05},
],
tag_columns=["experiment"], tag_only_columns=True,
source_id="lab_results",
)

There are two important concepts here:

  • Tag columns (tag_columns) are the keys that identify each row -- like primary keys in a database or independent variables in an experiment. Here, experiment uniquely identifies each measurement.
  • Data columns are the actual data payload. In this example, temperature and pressure are the data columns.

By default a source's tag columns are also data columns: each is a data-tag, one column seen in both roles, so a function can read it. tag_only_columns=True keeps the tags plain (a tag only), which is what this example does, so the data columns are exactly the columns that are not tags. See Data-tags.

The source_id is a human-readable name used for provenance tracking.

In Orcapod, a source is a stream. You can inspect it immediately without any extra conversion step. In a notebook, put source on the last line of a cell to see a table of its first rows; Working in notebooks shows what that table and Orcapod's other notebook displays show.

Use output_schema() to see the tag and data column types:

tag_schema, data_schema = source.output_schema()
print(tag_schema)
# Schema({'experiment': str})
print(data_schema)
# Schema({'temperature': float, 'pressure': float})

Use keys() to get just the column names:

tag_keys, data_keys = source.keys()
print(tag_keys)
# ('experiment',)
print(data_keys)
# ('temperature', 'pressure')

A stream is iterable: a for loop walks through each (Tag, Data) pair. Iterating a stream is the same as calling its iter_data() method:

for tag, data in source:
print(f"Tag: {tag.as_dict()}, Data: {data.as_dict()}")
# Tag: {'experiment': 'exp_001'}, Data: {'temperature': 20.5, 'pressure': 1.01}
# Tag: {'experiment': 'exp_002'}, Data: {'temperature': 22.3, 'pressure': 0.98}
# Tag: {'experiment': 'exp_003'}, Data: {'temperature': 19.8, 'pressure': 1.05}

Use as_table() to get a PyArrow Table, which you can convert to pandas:

table = source.as_table()
print(table.to_pandas())
# experiment temperature pressure
# 0 exp_001 20.5 1.01
# 1 exp_002 22.3 0.98
# 2 exp_003 19.8 1.05

A function pod transforms data row by row. Use the @function_pod decorator to turn a plain Python function into a reusable, trackable transformation:

from orcapod import function_pod
@function_pod(output_keys=["temp_fahrenheit", "is_high_pressure"])
def analyze_conditions(temperature: float, pressure: float) -> tuple[float, bool]:
temp_f = temperature * 9.0 / 5.0 + 32.0
is_high = pressure > 1.0
return temp_f, is_high

A few things to note about function pods:

  • Parameter names match data column names. The function's parameter names (temperature, pressure) must match the data column names from the input stream exactly -- one parameter per data column. Data columns the function does not take are dropped when you call the pod; pod(stream, strict=True) and pod.process(stream) reject them instead (see Function Pods).
  • Type annotations define the schemas. Orcapod uses them to build the output schema and to validate compatibility when the pod is applied. A parameter without an annotation needs its type in the decorator's input_schema mapping instead.
  • output_keys names the output columns. Since the function returns a tuple of two values, output_keys must be a list of two names. For a function returning a single value, pass a single string (e.g., output_keys="result").
  • The decorator creates a .pod attribute. You call .pod(stream) to apply the function to a stream.

Now apply it:

result = analyze_conditions.pod(source)

The result is a new stream. Tags are preserved from the input; the data columns are replaced with the function's outputs.

The result stream supports the same inspection methods as the source:

tag_schema, data_schema = result.output_schema()
print(tag_schema)
# Schema({'experiment': str})
print(data_schema)
# Schema({'temp_fahrenheit': float, 'is_high_pressure': bool})

The tag schema is unchanged -- function pods never modify tags. The data schema now reflects the function's output types.

Iterate over the results:

for tag, data in result:
print(f"Tag: {tag.as_dict()}, Data: {data.as_dict()}")
# Tag: {'experiment': 'exp_001'}, Data: {'temp_fahrenheit': 68.9, 'is_high_pressure': True}
# Tag: {'experiment': 'exp_002'}, Data: {'temp_fahrenheit': 72.14, 'is_high_pressure': False}
# Tag: {'experiment': 'exp_003'}, Data: {'temp_fahrenheit': 67.64, 'is_high_pressure': True}

Or view it as a table:

print(result.as_table().to_pandas())
# experiment temp_fahrenheit is_high_pressure
# 0 exp_001 68.90 True
# 1 exp_002 72.14 False
# 2 exp_003 67.64 True

Here is the complete example in one block:

from orcapod import function_pod
from orcapod.sources import DictSource
# Create a source from raw data
source = DictSource(
data=[
{"experiment": "exp_001", "temperature": 20.5, "pressure": 1.01},
{"experiment": "exp_002", "temperature": 22.3, "pressure": 0.98},
{"experiment": "exp_003", "temperature": 19.8, "pressure": 1.05},
],
tag_columns=["experiment"], tag_only_columns=True,
source_id="lab_results",
)
# Define a transformation
@function_pod(output_keys=["temp_fahrenheit", "is_high_pressure"])
def analyze_conditions(temperature: float, pressure: float) -> tuple[float, bool]:
temp_f = temperature * 9.0 / 5.0 + 32.0
is_high = pressure > 1.0
return temp_f, is_high
# Apply and view results
result = analyze_conditions.pod(source)
print(result.as_table().to_pandas())
# experiment temp_fahrenheit is_high_pressure
# 0 exp_001 68.90 True
# 1 exp_002 72.14 False
# 2 exp_003 67.64 True

Everything above runs in memory: nothing is stored, so each new run computes everything again. To save results in a database and have later runs compute only what is new, record the same steps inside a PipelineJob. See Pipelines and Jobs, or work through the Tutorial, which builds up to it step by step.

Now that you have the basics, explore these topics:

  • Sources -- learn about the different source types and how provenance tracking works.
  • Streams -- understand the immutable (Tag, Data) stream model.
  • Working in notebooks -- what streams, run summaries and pipeline diagrams show at the end of a notebook cell.
  • Function Pods -- advanced function pod usage, including caching with databases.
  • Operators -- structural transforms like Join, Batch, and Filter that work on tags and stream structure without inspecting data content.
  • Identity & Hashing -- how Orcapod tracks content identity and pipeline structure for reproducibility.
  • Tutorial -- a self-paced course with exercises, on one small dataset.
  • Troubleshooting -- common errors, their causes and fixes.