Part 8: Inspecting and tracing
A number in a figure raises questions. Which steps produced it? From which rows of which table? When was it computed, and with what? Because Orcapod runs every step itself, it keeps the answers. This part shows where to find them.
You will:
- walk backwards from a result to the sources it came from
- reveal the provenance columns that every stream carries
- read the record Orcapod keeps of each function call
- start a second pipeline from the stored results of the first
from workshop_data import make_datasetfrom orcapod import PipelineJob, function_podfrom orcapod.databases import InMemoryArrowDatabasefrom orcapod.sources import CSVSource
data_dir = make_dataset()database = InMemoryArrowDatabase()
@function_pod(output_keys="rate_hz")def firing_rate(n_spikes: int, duration_s: float) -> float: return n_spikes / duration_s
@function_pod(output_keys="mean_rate_hz")def mean_rate(rate_hz: list[float]) -> float: return sum(rate_hz) / len(rate_hz)
job = PipelineJob(name="workshop", database=database)with job: units = CSVSource( str(data_dir / "units.csv"), tag_columns=["subject_id", "session", "probe", "unit"], tag_only_columns=True, source_id="units", ) sessions = CSVSource( str(data_dir / "sessions.csv"), tag_columns=["subject_id", "session"], tag_only_columns=True, source_id="sessions", ) rates = firing_rate.pod( units.select_data_columns(["n_spikes"]), sessions.select_data_columns(["duration_s"]), label="rates", ) mean_rate.pod( rates.group_by(["subject_id", "session"]).select_data_columns(["rate_hz"]), label="session_means", )job.run()This is the full pipeline from Part 5.
Which steps produced this?
Section titled “Which steps produced this?”Every node knows the nodes it reads from, in node.upstreams, and its own name, in
node.label. Following upstreams until there are none left takes you from a result back
to the sources:
def print_lineage(node, depth=0): print(" " * depth + node.label) for upstream in node.upstreams: print_lineage(upstream, depth + 1)
print_lineage(job.nodes["session_means"])# session_means# SelectDataColumns_2# GroupBy# rates# Join# SelectDataColumns_3# units# SelectDataColumns_1# sessionsRead from the bottom up, this is the pipeline as you wrote it, including the join that the
two-stream call to firing_rate added for you. When a job has several steps of the same
kind, it numbers them (SelectDataColumns_1, ...); the numbers tell the steps apart but do
not follow the order you wrote them in.
The whole graph, rather than one node's lineage, is what the job draws when it is the last
line of a notebook cell (see Part 5).
Clicking a node there shows its columns and its row counts in the last run.
job.diagram(direction="TB") lays it out top to bottom (in the notebook, the vertical
checkbox above the diagram does the same), and print(job.diagram()) lists it as text.
Working in notebooks shows the diagram, the table preview of a
node and the run summary as they appear in a notebook.
Which rows did a value come from?
Section titled “Which rows did a value come from?”Every stream carries hidden columns that record where its rows and values came from. They
are left out by default. Ask for them with the columns argument of as_table.
Source-info columns. For each data column there is a companion column named
_source_<column>. On a source it holds the source's name, the row number and the column:
table = job.nodes["units"].as_table(columns={"source": True})print(table.column_names)# ['subject_id', 'session', 'probe', 'unit', 'region', 'depth_um', 'n_spikes', '_source_region', '_source_depth_um', '_source_n_spikes']print(table.column("_source_n_spikes").to_pylist()[5])# units::row_5::n_spikesRow 5 of the source you named units, column n_spikes. This is what the source_id you
have been passing since Part 1 is for.
System tags. Each source also adds two hidden tag columns to every row: one with the
source's name, one with an identifier of the row. They travel with the row through every
operator and function pod. When two streams are joined, the joined row carries the system
tags of both. So for any row of rates you can read off which sources it depends on:
import polars as pl
table = pl.from_arrow(job.nodes["rates"].as_table(columns={"system_tags": True}))source_columns = [c for c in table.columns if c.startswith("_tag_source_id")]print(len(source_columns))# 2print(table.select(source_columns).unique().rows())# [('units', 'sessions')]Two system-tag columns, because two sources feed rates; every row depends on one row of
sessions and one row of units. The column names end in a long hash that identifies the
source's position in the pipeline, which is why the sample selects them by prefix.
The companion _tag_record_id::... columns identify the individual source row. They hold
compact binary identifiers that Orcapod uses internally to tell rows apart; they are not
meant to be read by eye.
as_table(all_info=True) shows every hidden column at once.
How was it computed?
Section titled “How was it computed?”For each function, the database holds one record per computed result, with details of the
call. get_all_records returns them; ask for the meta columns to see the details:
records = pl.from_arrow(job.nodes["rates"].get_all_records(columns={"meta": True}))meta = [c for c in records.columns if c.startswith("__")]print(meta)# ['__data_id', '__input_data_hash', '__output_data_hash', '__data_context', '__computed', '__is_ephemeral', '__input_data_hash_right', '__pf_var_function_name', '__pf_var_function_version', '__pf_var_function_signature_hash', '__pf_var_function_content_hash', '__pf_var_git_hash', '__pf_exec_executor_type', '__pf_exec_executor_info', '__pf_exec_python_version', '__pf_exec_extra_info', '__pod_ts']The ones you are most likely to want:
| Column | Meaning |
|---|---|
__pf_var_function_name |
The function that produced the result |
__pod_ts |
When the result was computed |
__pf_exec_python_version |
The Python version it ran under |
__pf_var_git_hash |
The git commit of the code, when run inside a git repository |
__computed |
True if this row was computed, False if it reused the result of an identical input |
__input_data_hash |
The hash of the input values: the key the result is stored under (Part 6) |
print(records.select("__pf_var_function_name", "__pf_exec_executor_type").unique().rows())# [('firing_rate', 'local')]print(records["__computed"].sum(), "of", len(records))# 63 of 64print(records["__pod_ts"].dtype)# Datetime(time_unit='us', time_zone='UTC')63 of 64: the count from Part 5 again, now visible in the records. One row reused the result of another row with the same inputs.
as_table versus get_all_records
Section titled “as_table versus get_all_records”They answer different questions.
node.as_table()is the node's output for the inputs of this job's run: 64 rows here. Before the job has run in this Python session it has no rows at all, even when the database already holds results.node.get_all_records()is everything stored for this function in this job's part of the database, including results from earlier runs over other inputs. It does not need a run.
After the "two more units" run of Part 6, a job over the original file, once run, has 64
rows in as_table(), while get_all_records() has 66.
Starting a second pipeline from stored results
Section titled “Starting a second pipeline from stored results”A finished analysis is often the input of the next one. as_source() turns the stored
results of a node into a source:
session_means = job.nodes["session_means"].as_source()print(type(session_means).__name__)# DerivedSourceprint(session_means.keys())# (('subject_id', 'session'), ('mean_rate_hz',))A DerivedSource is a source like those of Part 1, so it can be joined, filtered and fed to
pods. Here the next stage flags sessions with a low mean rate:
@function_pod(output_keys="is_low")def is_low(mean_rate_hz: float) -> bool: return mean_rate_hz < 5.5
flags = is_low.pod(session_means).as_pandas_df()print(flags.sort_values(["subject_id", "session"], ignore_index=True))# subject_id session is_low# 0 mouse_01 s1 False# 1 mouse_01 s2 False# 2 mouse_02 s1 False# 3 mouse_02 s2 False# 4 mouse_03 s1 True# 5 mouse_03 s2 True# 6 mouse_04 s1 True# 7 mouse_04 s2 TrueThe next stage does not re-run the first. It reads what the first one stored. This is how
you split a large analysis into stages that different people, or different days, can own.
as_source() reads what the database holds, whether or not the node has run in this Python
session; for a node whose results were never stored, it has no rows. It reads every stored
result of the node, including those of earlier runs over other inputs, like
get_all_records() above; when you want only this run's output, use as_table().
What to remember
Section titled “What to remember”node.upstreamsandnode.labelwalk the pipeline backwards from any result.as_table(columns={"source": True})andcolumns={"system_tags": True}reveal where values and rows came from;all_info=Trueshows everything.get_all_records(columns={"meta": True})shows how and when each result was computed.as_table()is the output of this job's run;get_all_records()is everything stored.node.as_source()starts a new pipeline from stored results.
Exercises
Section titled “Exercises”1. Print the lineage of the rates node. How many sources does it depend on?
Solution
print_lineage(job.nodes["rates"])# rates# Join# SelectDataColumns_3# units# SelectDataColumns_1# sessionsTwo: units and sessions.
2. Which row of sessions.csv did the duration of mouse_03, session s2 come from?
Use the source-info column of the sessions node.
Solution
table = pl.from_arrow(job.nodes["sessions"].as_table(columns={"source": True}))row = table.filter((pl.col("subject_id") == "mouse_03") & (pl.col("session") == "s2"))print(row["_source_duration_s"].to_list())# ['sessions::row_5::duration_s']Row 5, counting data rows from 0: the sixth line after the header.
Note the plain Polars filter on a table you have already taken out of Orcapod. Inside a
pipeline you would write the same test as stream.polars_filter(...) (Part 3).
3. Using as_source(), compute the mean of the session means per animal (one row per
subject_id). Reuse group_by from Part 4.
Solution
@function_pod(output_keys="animal_mean_hz")def mean_of_means(mean_rate_hz: list[float]) -> float: return sum(mean_rate_hz) / len(mean_rate_hz)
means = job.nodes["session_means"].as_source()result = mean_of_means.pod(means.group_by(["subject_id"])).as_pandas_df()print(result.sort_values("subject_id", ignore_index=True))# subject_id animal_mean_hz# 0 mouse_01 6.413542# 1 mouse_02 6.676042# 2 mouse_03 4.983160# 3 mouse_04 5.132812Next: Part 9: When things go wrong.