Part 5: Keeping results
Everything in Parts 1 to 4 was computed in memory. Close Python and the results are gone; run the script again and every function runs again. That is fine for 64 rows. It is not fine when one function call takes a minute and you have ten thousand of them.
A PipelineJob runs the same steps against a database. Each result is stored together
with a record of the inputs it was computed from. The next time the pipeline runs, results
that are already stored are read back instead of being computed again.
You will:
- record a pipeline inside a
PipelineJoband run it - read results from the job's nodes
- keep the results on disk, and see a second run compute nothing
from workshop_data import make_datasetfrom orcapod import PipelineJob, function_podfrom orcapod.sources import CSVSource
data_dir = make_dataset()
sessions = CSVSource( str(data_dir / "sessions.csv"), tag_columns=["subject_id", "session"], tag_only_columns=True, source_id="sessions",)units = CSVSource( str(data_dir / "units.csv"), tag_columns=["subject_id", "session", "probe", "unit"], tag_only_columns=True, source_id="units",)
calls = [] # every function call adds an entry, so we can see when functions really run
@function_pod(output_keys="rate_hz")def firing_rate(n_spikes: int, duration_s: float) -> float: calls.append("firing_rate") return n_spikes / duration_s
@function_pod(output_keys="mean_rate_hz")def mean_rate(rate_hz: list[float]) -> float: calls.append("mean_rate") return sum(rate_hz) / len(rate_hz)These are the two functions from Part 4. The calls list is only there for teaching: it
lets us count how often each function was really called.
Record, then run
Section titled “Record, then run”A job needs a database to put its results in. Start with one that lives in memory. Then write the pipeline exactly as before, inside a with block:
from orcapod.databases import InMemoryArrowDatabase
database = InMemoryArrowDatabase()
job = PipelineJob(name="workshop", database=database)with job: rates = firing_rate.pod( units.select_data_columns(["n_spikes"]), sessions.select_data_columns(["duration_s"]), label="rates", ) mean_rate.pod( rates.group_by(["subject_id", "session"]).select_data_columns(["rate_hz"]), label="session_means", )
print(len(calls))# 0Nothing has been computed. Inside the with block the job only records which operators
and pods are applied to which streams. The one new thing in the code is label=: a name for
the step, which you will use to find its results.
run() executes what was recorded and writes every result to the database:
job.run()
print(len(calls))# 71print(calls.count("firing_rate"), calls.count("mean_rate"))# 63 8mean_rate ran 8 times, once per session. firing_rate ran 63 times for 64 units. That is
not a mistake: two units happen to have the same spike count and the same session duration.
Orcapod stores a result under the function and the values of its inputs, so the second of
those two units found its answer already stored.
Nodes: where the results are
Section titled “Nodes: where the results are”Each recorded step is a node of the job. job.nodes lists them by name:
print(sorted(job.nodes))# ['GroupBy', 'Join', 'SelectDataColumns_1', 'SelectDataColumns_2', 'SelectDataColumns_3', 'rates', 'session_means', 'sessions', 'units'](sorted is only there to make this page reproducible: list(job.nodes) gives the same
names, in an order that can differ from one Python session to the next.)
- A source is listed under its
source_id. - A function pod is listed under the
labelyou gave it. Without a label it gets the function's name. - Operators are named after their kind, numbered when there are several.
To see how the nodes connect, put job on the last line of a notebook cell: the job draws
itself as a diagram. Working in notebooks
has a picture of this job's diagram and what each display in a notebook shows.
Drag it to move it, zoom with the + and − buttons, and click a node to see its schema
(its tag and data columns) and how many rows the last run computed or read. Outside a notebook,
print(job.diagram()) gives the same graph as text, one node per line with the nodes it
reads from:
print(job.diagram())# Job 'workshop': 9 nodes# sessions (source · CSVSource · bound)# SelectDataColumns_1 (operator · current) <- sessions# units (source · CSVSource · bound)# SelectDataColumns_3 (operator · current) <- units# Join (operator · current) <- SelectDataColumns_1, SelectDataColumns_3# rates (function pod · firing_rate · current) <- Join# GroupBy (operator · current) <- rates# SelectDataColumns_2 (operator · current) <- GroupBy# session_means (function pod · mean_rate · current) <- SelectDataColumns_2The Join is there because firing_rate was given two streams: the pod joins them for you.
current means the node holds the rows of the run you just did; before job.run() every
node but the sources says not run. job.diagram(collapse=True) hides the steps that only
choose or join columns, leaving the sources, the function pods and the GroupBy (in the
notebook, the collapse checkbox above the diagram does the same):
print(job.diagram(collapse=True))# Job 'workshop': 5 nodes# sessions (source · CSVSource · bound)# units (source · CSVSource · bound)# rates (function pod · firing_rate · current) <- sessions, units# GroupBy (operator · current) <- rates# session_means (function pod · mean_rate · current) <- GroupByA node is a stream. Everything from Part 1 works on it:
means = job.nodes["session_means"]print(means.output_schema())# (Schema({'subject_id': str, 'session': str}), Schema({'mean_rate_hz': float}))print(means.as_pandas_df().sort_values(["subject_id", "session"], ignore_index=True))# subject_id session mean_rate_hz# 0 mouse_01 s1 6.033333# 1 mouse_01 s2 6.793750# 2 mouse_02 s1 5.845833# 3 mouse_02 s2 7.506250# 4 mouse_03 s1 4.906944# 5 mouse_03 s2 5.059375# 6 mouse_04 s1 5.136458# 7 mouse_04 s2 5.129167print(job.nodes["rates"].as_table().num_rows)# 64Run it again
Section titled “Run it again”Record the same pipeline in a new job over the same database. A small function saves retyping:
def build_job(database): job = PipelineJob(name="workshop", database=database) with job: rates = firing_rate.pod( units.select_data_columns(["n_spikes"]), sessions.select_data_columns(["duration_s"]), label="rates", ) mean_rate.pod( rates.group_by(["subject_id", "session"]).select_data_columns(["rate_hz"]), label="session_means", ) return job
job = build_job(database)job.run()
print(len(calls))# 71print(job.nodes["session_means"].as_table().num_rows)# 8Still 71. The second run called no function at all. Every result was found in the database, and the node holds the same 8 rows.
Keep results on disk
Section titled “Keep results on disk”InMemoryArrowDatabase disappears with the Python process. DeltaTableDatabase writes to a
directory and is what you use for real work:
import shutil
from orcapod.databases import DeltaTableDatabase
shutil.rmtree("workshop_db", ignore_errors=True) # start empty, also when you rerun this pagejob = build_job(DeltaTableDatabase("workshop_db"))job.run()print(len(calls))# 142This is a different, empty database, so everything was computed once more: 71 further calls.
(You will see many WARN deltalake_core::writer::stats lines while it writes. They are
harmless; see Part 0.)
Now do what a new Python session would do: make a new database object for the same directory, build the job, run it.
job = build_job(DeltaTableDatabase("workshop_db"))job.run()
print(len(calls))# 142means = job.nodes["session_means"].as_pandas_df()print(means.sort_values(["subject_id", "session"], ignore_index=True).head(2))# subject_id session mean_rate_hz# 0 mouse_01 s1 6.033333# 1 mouse_01 s2 6.793750No calls. The results came from the files in workshop_db/. Rows read back from a database
are not guaranteed to come in the order they were first computed in, which is why the sample
sorts them before printing. Sort whenever the order matters to you.
from pathlib import Path
print(sorted(p.name for p in Path("workshop_db").iterdir()))# ['workshop']print(sorted(p.name for p in Path("workshop_db/workshop").iterdir()))# ['_result', 'firing_rate', 'mean_rate']The first level is the job's name. Below it, _result holds each function's results, and
the directory named after each function records which inputs the job gave it. Two jobs with
different names over the same directory do not share results. You never need to read
these files yourself, and deleting the whole directory is safe: it only means the next run
computes everything again.
What to remember
Section titled “What to remember”PipelineJob(name=..., database=...), awithblock to record,job.run()to execute.- Give pods a
label; read results fromjob.nodes[label]afterrun(). - Results are stored per function and per input values. A second run over the same database computes only what is missing.
InMemoryArrowDatabasefor trying things out;DeltaTableDatabase("directory")to keep results between sessions.
The concept page Pipelines and jobs covers what this part left out: reusable pipeline blueprints, binding sources later, and saving a job to a file.
Exercises
Section titled “Exercises”1. Run a pipeline, quit Python, and run it again. Put the script below in a file
run_pipeline.py and run python run_pipeline.py twice. What does each run print?
from workshop_data import make_datasetfrom orcapod import PipelineJob, function_podfrom orcapod.databases import DeltaTableDatabasefrom orcapod.sources import CSVSource
data_dir = make_dataset()n_calls = 0
@function_pod(output_keys="depth_mm")def to_mm(depth_um: int) -> float: global n_calls n_calls += 1 return depth_um / 1000
job = PipelineJob(name="depths", database=DeltaTableDatabase("exercise_db"))with job: units = CSVSource( str(data_dir / "units.csv"), tag_columns=["subject_id", "session", "probe", "unit"], tag_only_columns=True, source_id="units", ) to_mm.pod(units.select_data_columns(["depth_um"]), label="depth_mm")job.run()
print("rows:", job.nodes["depth_mm"].as_table().num_rows)# rows: 64print("function calls in this run:", n_calls)# function calls in this run: 54Solution
The first run prints rows: 64 and function calls in this run: 54. Fifty-four, not
64, because some units share a depth and identical inputs are computed once.
The second run prints rows: 64 and function calls in this run: 0. It is a new Python
process, with a new job object and a new database object. The only thing it shares with
the first run is the exercise_db directory, and that is enough.
Delete exercise_db/ and run the script once more: 54 calls again.
2. In the main example, what would list(job.nodes) show if you removed both label=
arguments? Predict, then try it with a new job over an InMemoryArrowDatabase.
Solution
The two function nodes are listed under their function names.
unlabelled = PipelineJob(name="unlabelled", database=InMemoryArrowDatabase())with unlabelled: rates = firing_rate.pod( units.select_data_columns(["n_spikes"]), sessions.select_data_columns(["duration_s"]), ) mean_rate.pod( rates.group_by(["subject_id", "session"]).select_data_columns(["rate_hz"]) )print(sorted(unlabelled.nodes))# ['GroupBy', 'Join', 'SelectDataColumns_1', 'SelectDataColumns_2', 'SelectDataColumns_3', 'firing_rate', 'mean_rate', 'sessions', 'units']Labels become useful when the same function is applied in two places of one pipeline.
3. Create a job without a database, record one step, and call run(). Read the error and do
what it suggests.
Solution
no_database = PipelineJob(name="no_database")with no_database: firing_rate.pod( units.select_data_columns(["n_spikes"]), sessions.select_data_columns(["duration_s"]), label="rates", )
try: no_database.run()except ValueError as err: print(err)# PipelineJob.run() requires a database. Call job.bind(database=db) before run(). For transient (in-memory) storage use job.bind(database=InMemoryArrowDatabase()).
no_database.bind(database=InMemoryArrowDatabase())no_database.run()print(no_database.nodes["rates"].as_table().num_rows)# 64Next: Part 6: Re-runs and caching.