Getting Started
This guide walks you through the core workflow in Orcapod: creating a data source, inspecting its stream, applying a transformation with a function pod, and examining the results.
Installing
Section titled “Installing”Install Orcapod with uv, the recommended way: start a project
with uv init (in a new directory), add Orcapod to it, and run your scripts with uv run:
uv inituv add git+https://github.com/metamorphic-labs/orcapod-pythonuv run python my_pipeline.pyIn an environment not managed by uv, pip install git+https://github.com/metamorphic-labs/orcapod-python installs it instead.
Until 0.4.0 is released, these commands install the development version from the repository:
PyPI has no current version (0.3.0 was withdrawn, and a plain uv add orcapod would install
0.1.4). Once 0.4.0 is out, uv add orcapod (or pip install orcapod) installs it.
Creating a source
Section titled “Creating a source”A source is the entry point for data in an Orcapod pipeline. The simplest way to
get started is with DictSource, which accepts a list of dictionaries:
from orcapod.sources import DictSource
source = DictSource( data=[ {"experiment": "exp_001", "temperature": 20.5, "pressure": 1.01}, {"experiment": "exp_002", "temperature": 22.3, "pressure": 0.98}, {"experiment": "exp_003", "temperature": 19.8, "pressure": 1.05}, ], tag_columns=["experiment"], tag_only_columns=True, source_id="lab_results",)There are two important concepts here:
- Tag columns (
tag_columns) are the keys that identify each row -- like primary keys in a database or independent variables in an experiment. Here,experimentuniquely identifies each measurement. - Data columns are the actual data payload. In this example,
temperatureandpressureare the data columns.
By default a source's tag columns are also data columns: each is a data-tag, one
column seen in both roles, so a function can read it. tag_only_columns=True keeps the tags
plain (a tag only), which is what this example does, so the data columns are exactly the
columns that are not tags. See Data-tags.
The source_id is a human-readable name used for provenance tracking.
Inspecting the stream
Section titled “Inspecting the stream”In Orcapod, a source is a stream. You can inspect it immediately without any extra
conversion step. In a notebook, put source on the last line of a cell to see a table of
its first rows; Working in notebooks shows what that table and
Orcapod's other notebook displays show.
Schema
Section titled “Schema”Use output_schema() to see the tag and data column types:
tag_schema, data_schema = source.output_schema()print(tag_schema)# Schema({'experiment': str})print(data_schema)# Schema({'temperature': float, 'pressure': float})Column names
Section titled “Column names”Use keys() to get just the column names:
tag_keys, data_keys = source.keys()print(tag_keys)# ('experiment',)print(data_keys)# ('temperature', 'pressure')Iterating over rows
Section titled “Iterating over rows”A stream is iterable: a for loop walks through each (Tag, Data) pair. Iterating a stream
is the same as calling its iter_data() method:
for tag, data in source: print(f"Tag: {tag.as_dict()}, Data: {data.as_dict()}")# Tag: {'experiment': 'exp_001'}, Data: {'temperature': 20.5, 'pressure': 1.01}# Tag: {'experiment': 'exp_002'}, Data: {'temperature': 22.3, 'pressure': 0.98}# Tag: {'experiment': 'exp_003'}, Data: {'temperature': 19.8, 'pressure': 1.05}Getting the full table
Section titled “Getting the full table”Use as_table() to get a PyArrow Table, which you can convert to pandas:
table = source.as_table()print(table.to_pandas())# experiment temperature pressure# 0 exp_001 20.5 1.01# 1 exp_002 22.3 0.98# 2 exp_003 19.8 1.05Applying a function pod
Section titled “Applying a function pod”A function pod transforms data row by row. Use the @function_pod decorator
to turn a plain Python function into a reusable, trackable transformation:
from orcapod import function_pod
@function_pod(output_keys=["temp_fahrenheit", "is_high_pressure"])def analyze_conditions(temperature: float, pressure: float) -> tuple[float, bool]: temp_f = temperature * 9.0 / 5.0 + 32.0 is_high = pressure > 1.0 return temp_f, is_highA few things to note about function pods:
- Parameter names match data column names. The function's parameter names
(
temperature,pressure) must match the data column names from the input stream exactly -- one parameter per data column. Data columns the function does not take are dropped when you call the pod;pod(stream, strict=True)andpod.process(stream)reject them instead (see Function Pods). - Type annotations define the schemas. Orcapod uses them to build the output schema and
to validate compatibility when the pod is applied. A parameter without an annotation needs
its type in the decorator's
input_schemamapping instead. output_keysnames the output columns. Since the function returns a tuple of two values,output_keysmust be a list of two names. For a function returning a single value, pass a single string (e.g.,output_keys="result").- The decorator creates a
.podattribute. You call.pod(stream)to apply the function to a stream.
Now apply it:
result = analyze_conditions.pod(source)The result is a new stream. Tags are preserved from the input; the data columns
are replaced with the function's outputs.
Inspecting the result
Section titled “Inspecting the result”The result stream supports the same inspection methods as the source:
tag_schema, data_schema = result.output_schema()print(tag_schema)# Schema({'experiment': str})print(data_schema)# Schema({'temp_fahrenheit': float, 'is_high_pressure': bool})The tag schema is unchanged -- function pods never modify tags. The data schema now reflects the function's output types.
Iterate over the results:
for tag, data in result: print(f"Tag: {tag.as_dict()}, Data: {data.as_dict()}")# Tag: {'experiment': 'exp_001'}, Data: {'temp_fahrenheit': 68.9, 'is_high_pressure': True}# Tag: {'experiment': 'exp_002'}, Data: {'temp_fahrenheit': 72.14, 'is_high_pressure': False}# Tag: {'experiment': 'exp_003'}, Data: {'temp_fahrenheit': 67.64, 'is_high_pressure': True}Or view it as a table:
print(result.as_table().to_pandas())# experiment temp_fahrenheit is_high_pressure# 0 exp_001 68.90 True# 1 exp_002 72.14 False# 2 exp_003 67.64 TruePutting it all together
Section titled “Putting it all together”Here is the complete example in one block:
from orcapod import function_podfrom orcapod.sources import DictSource
# Create a source from raw datasource = DictSource( data=[ {"experiment": "exp_001", "temperature": 20.5, "pressure": 1.01}, {"experiment": "exp_002", "temperature": 22.3, "pressure": 0.98}, {"experiment": "exp_003", "temperature": 19.8, "pressure": 1.05}, ], tag_columns=["experiment"], tag_only_columns=True, source_id="lab_results",)
# Define a transformation@function_pod(output_keys=["temp_fahrenheit", "is_high_pressure"])def analyze_conditions(temperature: float, pressure: float) -> tuple[float, bool]: temp_f = temperature * 9.0 / 5.0 + 32.0 is_high = pressure > 1.0 return temp_f, is_high
# Apply and view resultsresult = analyze_conditions.pod(source)print(result.as_table().to_pandas())# experiment temp_fahrenheit is_high_pressure# 0 exp_001 68.90 True# 1 exp_002 72.14 False# 2 exp_003 67.64 TrueKeeping results
Section titled “Keeping results”Everything above runs in memory: nothing is stored, so each new run computes everything again. To
save results in a database and have later runs compute only what is new, record the same
steps inside a PipelineJob. See Pipelines and Jobs, or work
through the Tutorial, which builds up to it step by step.
Next steps
Section titled “Next steps”Now that you have the basics, explore these topics:
- Sources -- learn about the different source types and how provenance tracking works.
- Streams -- understand the immutable (Tag, Data) stream model.
- Working in notebooks -- what streams, run summaries and pipeline diagrams show at the end of a notebook cell.
- Function Pods -- advanced function pod usage, including caching with databases.
- Operators -- structural transforms like Join, Batch, and Filter that work on tags and stream structure without inspecting data content.
- Identity & Hashing -- how Orcapod tracks content identity and pipeline structure for reproducibility.
- Tutorial -- a self-paced course with exercises, on one small dataset.
- Troubleshooting -- common errors, their causes and fixes.