Sources
Sources are the entry points for external data into an Orcapod pipeline. Every pipeline begins with one or more sources that load raw data -- from Python dicts, lists, CSV files, Delta Lake tables, or Pandas DataFrames -- and present it as an immutable stream of (Tag, Data) pairs. Sources also attach provenance metadata (source-info columns and system tag columns) so that every downstream value can be traced back to its origin.
Key classes
Section titled “Key classes”RootSource (abstract base)
Section titled “RootSource (abstract base)”All sources inherit from RootSource. A root source is a pure stream with no upstream
dependencies -- it sits at the root of the computational graph. Key properties:
source.producerreturnsNone(no upstream pod)source.upstreamsis always an empty tuplesource.source_idis a canonical name used for provenance tracking and the source registry
Concrete source types
Section titled “Concrete source types”All sources follow the same pattern: convert input data to a PyArrow Table, then pass it
through SourceStreamBuilder which handles enrichment (provenance columns, system tags,
hashing) and produces the final immutable stream.
| Source | Input type | Notes |
|---|---|---|
ArrowTableSource |
PyArrow Table |
Accepts an Arrow table directly |
DictSource |
list[dict] |
Each dict becomes one (Tag, Data) pair |
ListSource |
list[Any] |
Each element stored under the data column given by name; the tag defaults to element_index |
DataFrameSource |
Pandas or Polars DataFrame |
Converts via Arrow |
CSVSource |
File path (string) | Reads CSV into Arrow |
DeltaTableSource |
Directory path | Reads Delta Lake table |
orcapod.sources also provides database-backed sources (SQLiteTableSource,
PostgreSQLTableSource, SpiralDBTableSource) and PollingSource for data that arrives over
time; see the API reference. Their default source_id names the table
where it lives (sqlite_table:<db path>:main.<table>). DBTableSource, the base they share,
can also be used directly over any connector; a connector gives no address of its database,
so it then has no default and needs a source_id.
Two construction rules apply to every source and are the most common first errors:
- Every name in
tag_columnsmust be a column of the data, otherwise the source raisesValueError: tag_columns not found in table. - At least one data column must remain, otherwise the source raises
ValueError: No data columns found in the table. A data-tag is also a data column, so this happens only when every tag is kept plain (tag_only_columns) and no other column is left.
Data-tags and plain tags: tag_only_columns. The sources built from a table
(ArrowTableSource, DictSource, DataFrameSource, CSVSource, DeltaTableSource and the
database sources) take the keyword-only option tag_only_columns. By default every declared
tag is a data-tag: it is also the data column of the same name (see
Data-tags). tag_only_columns names the tags kept plain, a tag
only and never a data column, as every tag was in v0.3; tag_only_columns=True keeps all of
them plain. A source with plain tags and one with data-tags over one table have the same source id and record
ids, but different schemas and so different hashes.
from orcapod.sources import DictSource
with_data_tags = DictSource(data=[{"k": "a", "x": 1}], tag_columns=["k"])print(with_data_tags.keys())# (('k',), ('k', 'x'))plain = DictSource( data=[{"k": "a", "x": 1}], tag_columns=["k"], tag_only_columns=True)print(plain.keys())# (('k',), ('x',))Data with no rows has no columns to read, so DictSource(data=[], ...) raises
ValueError: Table is empty. To build a source with no rows, pass data_schema= so the
columns are known.
Arrow types of a source
Section titled “Arrow types of a source”Every source gives the large variants of string, binary and list types (large_string,
large_binary, large_list, recursively), whatever its input: an Arrow table of your own
(ArrowTableSource), an Arrow schema you pass (schema= or data_schema=), Python values,
a data frame, a CSV file or a database table. A string column you give comes back as
large_string. Only the layout changes: names, nullability, widths (int8 stays int8) and
logical types are kept, and the two variants hash identically, so no hash depends on which
one you gave. The Arrow list_view and large_list_view types are not supported: a source
refuses a column of either type.
SpiralDB tables
Section titled “SpiralDB tables”SpiralDBTableSource exposes a table in a SpiralDB project as a
source. Install the extra and log in once; the spiral command comes with the extra, and
pyspiral keeps the login in its per-user directory (~/Library/Application Support/pyspiral/
on macOS, ~/.config/pyspiral/ on Linux):
uv add "orcapod[spiraldb]" # or, without uv: pip install "orcapod[spiraldb]"uv run spiral loginLog in before you build a source. Without a saved login, pyspiral starts a sign-in in the browser and waits until it is completed.
To find the tables of a project and their key columns:
from orcapod.databases import SpiralDBConnector
with SpiralDBConnector("edi-sample-v1-855632") as spiral: print(spiral.get_table_names()) print(spiral.get_pk_columns("session"))The table's key columns become the tag columns, and every other column is a data column:
from orcapod.sources import SpiralDBTableSource
sessions = SpiralDBTableSource("edi-sample-v1-855632", "session")experiments = SpiralDBTableSource("edi-sample-v1-855632", "experiment")print(sessions.output_schema())sessions.as_df() # also as_polars_df(), as_pandas_df(), as_lazy_frame(), as_table()Things to know about Spiral tables:
- Nullability comes from the table's nullability record. See
Nullability of Spiral columns: a recorded non-nullable
column has the type
T, every other columnT | None. - Types are kept. Arrow types come through unchanged (
uint32staysuint32), except that strings, binaries and lists become their large variants. Joining two Spiral tables on their shared key columns works; joining with another source requires the same Arrow types on both sides (uint32againstint64is an error, not a conversion). - Not supported: columns of type
map, which the Arrow hasher does not hash. Building the source raisesValueErrornaming each such column and its type; drop or convert those columns first.fixed_size_listcolumns are supported. Also not supported: time zones (SpiralDB stores timestamps without one) and Orcapod logical types such asPath(they come back as their storage type). - Source identity. A Spiral source is identified by where its table is, never by its
rows. The default
source_idisspiraldb_table:<project id>.<dataset>.<table>@<Spiral table id>, for examplespiraldb_table:edi-tutorial-v1-841226.default.session@table_Jo7J0lpkLcOM. The table id is SpiralDB's own, so tables with the same name in different projects stay apart, and a table that is dropped and created again under the same name is a different source: nothing recorded for the old table is reused. A renamed table also gets a newsource_id. - Record identity. A row is identified by its primary key: its record ID is the
canonical JSON of its key columns (
{"session_id":"s1"}). A row keeps its record ID when other rows are added, and has the same one whichever read returned it.
Reading a Spiral table
Section titled “Reading a Spiral table”Building a SpiralDBTableSource reads no rows. It reads the table's schema, key columns and
table id, which takes about a second whatever the table's size. output_schema(), keys(),
content_hash() and pipeline_hash() read no rows either. Rows are read when you ask for
them, and every read scans the table again:
iter_data()and a pipeline job hand the rows on in batches of at mostbatch_rowsrows (default 8,192). SpiralDB sends a scan in batches that follow its stored fragments, whose sizes cannot be chosen; one of them is held while its rows are handed on.as_table(),as_df(),as_polars_df(),as_pandas_df()andas_lazy_frame()read the rows into memory and return them all.
To read part of a table, say so when you build the source. SpiralDB applies both restrictions before any row is transferred:
first = SpiralDBTableSource("edi-tutorial-v1-841226", "spike_event", limit=1000)unit = SpiralDBTableSource( "edi-tutorial-v1-841226", "spike_event", where={"unit_id": 106})print(first.row_count_estimate) # rows in the whole table, without reading anylimit=nreads the firstnrows in key order.where={column: value, ...}reads the rows in which every named column equals its value.asof=reads the table as of an earlier version, and keeps reading it. Without it, a source reads the version the table had when the source was built, in every read, until a run finds a newer one (see What a source does at a run):head(),as_table()and the preview outside a run never move it.
The row cap. One read returns at most max_rows rows (default 100,000). A read that
would return more raises SourceTooLargeError, which gives the row count and the ways out:
limit=, where=, or a larger max_rows (max_rows=None removes the cap). When nothing
but limit= bounds the read, the source decides before reading any row, from the row count
SpiralDB records for the table. That count can include deleted rows that are not yet
compacted, so a table near the cap can be refused although a scan would return fewer rows;
the error says so. The cap is there
because a pipeline job keeps every row it processes in memory, about 5 KB per row for each
node, so 100,000 rows cost about 0.5 GB per node. A table of millions of rows can be iterated
with max_rows=None, but it cannot yet be run through a job on a laptop.
Looking at a table in a notebook. A Spiral source shown at the end of a notebook cell
(its head(), see Looking at data) reads only its first 10
rows, in one scan to which the row cap does not apply, so a table of any size can be looked
at; head(n) reads n. The preview shows the rows it read in tag order (it asks for
sort_by_tags, as does head(columns={"sort_by_tags": True}) or all_info=True): the rows
read, sorted, not the first n of the table in tag order. The header line says what is
shown, for example SpiralDBTableSource[spike_event] — 10 rows read, sorted by tags, of about 63,349 (the table's recorded row count, read without scanning), all 4 rows, sorted by tags when the source reads no more than that, or 10 matching rows read, sorted by tags (where unit_id=106); the table has about 63,349 rows with where=. The unsorted head()
says first 10 rows of about 63,349. Tag columns come first, and the line above the table
names them. To see more rows, use as_df() with a limit= or where= source.
Restrictions are not part of identity. where, limit and asof choose rows of the
same table, so two sources on one table have the same source_id and content_hash
whatever their restrictions, and a row's result is computed once and shared. One
consequence: with a shared store, reading a job node's stored records back
(get_all_records(), node.as_source()) returns every row that any job with the same
pipeline has read from that table, not only the rows of the latest restriction. The rows
sent downstream during a run are only the rows that run read.
Running again. A run first checks the source: when the table has a newer version (rows
added since the source was built or last checked), the source reads that version from then
on, and the job's nodes below it run. The added rows are new keys, the only rows computed;
the rows already seen are reused. With nothing added, a second run computes nothing. A
source given asof= keeps its version: build a new source to read a later one.
Key columns as data. A function pod never receives a tag, and in a time-series table the
time is part of the key, so it is a tag. tags_as_data also offers named key columns as data
columns, copied as each batch is read (so the source still reads no rows when built, and
limit=, where=, max_rows and the notebook preview work as before). The tag keeps its
name; you name the copy, which comes after the table's data columns, with the key column's
type, never null:
spikes = SpiralDBTableSource( "edi-tutorial-v1-841226", "spike_event", tags_as_data={"timestamp": "spike_time_us"})unit_key = [ "spike_extraction_hash", "subject_id", "session_start_datetime", "recording_index", "probe_index", "unit_id",]per_unit = spikes.group_by(unit_key)print(per_unit.keys())# (('spike_extraction_hash', 'subject_id', 'session_start_datetime', 'recording_index', 'probe_index', 'unit_id'), ('spike_amplitude_uv', 'spike_time_us'))per_unit has one row per unit, and its spike_time_us list is in time order: group_by
orders the members of a group by the tags outside the group key, and on the full unit key
the only such tag is timestamp. The timestamp tag itself is dropped, as group_by drops
every tag outside the key; its copy spike_time_us is data and stays. group_by(["unit_id"]) alone sorts by
spike_extraction_hash, subject, session, recording, probe, then timestamp, so time order
holds only within one session and probe; group on the full unit key.
The copies are data columns, so they are part of the schema, the content_hash and the
pipeline_hash; the mapping itself is part of the content_hash too, because two mappings
can give the same schema while the copy holds different values. The source_id and every
row's record id are the same as without the option. Compared with
DerivedSource(spikes, tags_as_data=...), which reads the whole table as soon as its schema
or rows are asked for (building a job asks for the schema), the option keeps a Spiral source,
whose schema is known without a scan.
Known limitations.
- Each row's key is hashed with one call, about 0.2 ms per row (about 18 s per 100,000 rows), in addition to about 0.1 ms per row to build the row.
- A job keeps its rows in memory (see the row cap above).
- Only equality on column values can be pushed down; there is no key range, no column
selection (use
SelectDataColumnsafter the source), and no polling for new rows within one run. - A table that has only its key columns (for example one never written to) has no data column and cannot be a source.
Nullability of Spiral columns
Section titled “Nullability of Spiral columns”SpiralDB reports every column as nullable, key columns included, and every field of a struct column too, whatever nullability the data was written with; only list items keep a non-null flag. Orcapod therefore keeps a nullability record for each table: the list of the table's non-nullable columns and nested fields. The nullability record is an entry in the table's key-value metadata, next to the data; it is not a row of the table.
- A table with a nullability record (written by
SpiralDBConnector, by aConnectorArrowDatabaseon SpiralDB, or given one withwrite_nullability_record, as the tables of the workshop project are): the columns it lists have the typeT, so a function pod can takex: float. Every batch read is checked: a null in a listed column raisesNullInNonNullableColumnErrornaming the table and the column, so a wrong nullability record never hides a null. A nullability record that cannot be read in full (unknown version, missing part, a column the table does not have) raisesNullabilityRecordError. - A table without a nullability record (for example the EDI sample tables): every column
has the type
T | None(str | None,datetime.datetime | None, ...). A function pod receiving such a column must acceptNone: annotate the parameterx: float | None, notx: float, or the pod refuses the stream. Such a table reads exactly as it did before nullability records existed: same schema, same hashes. - Joins are type strict. A key column that is
Tin a table with a nullability record andT | Nonein a table without one is an error, not a conversion.
SpiralDBConnector writes the nullability record on a table's first write, before the rows.
It refuses a write that holds a null in a column its schema declares non-nullable, nested
fields included (SpiralDB would store a null written to a non-nullable struct field as 0),
and a later write whose non-nullable columns differ from the nullability record. It never adds
a nullability record to a table that already holds rows, and never changes one.
SpiralDB cannot store a null struct column. pyspiral refuses one (Spiral does not support null column groups) unless the writer passes push_down_nulls=True, and then the null struct
reads back as a struct whose fields are all null: a different value with a different hash. (Both
were seen for a top-level struct column; nested structs were not tested.) SpiralDBConnector
refuses a write holding a null struct at any depth (a struct column, a struct field of a
struct, a struct inside a list) with UnsupportedValueError naming the table and the column,
and writes nothing. Null lists, null list items and null struct fields are stored as written.
Writes are not coordinated between writers: SpiralDB has no lock for them. If two writers make
the first write to the same new table at once and one of them fails, the table can be left
without a nullability record although rows were written. It then reads as a table without one
(every column T | None); a null is never hidden. To repair it, check that the non-nullable
columns hold no null and call write_nullability_record as below.
Other writers give a table its nullability record with write_nullability_record, passing the
Arrow schema the data was written with. It reads no data, so check first that no non-nullable
column holds a null. The same call replaces a wrong nullability record.
import pyarrow as paimport spiral
from orcapod.databases import read_nullability_record, write_nullability_record
table = spiral.Spiral().project("my-project-123456").table("default.session")schema = pa.schema( [ pa.field("subject_id", pa.string(), nullable=False), pa.field("weight", pa.float64(), nullable=False), pa.field("booth", pa.string(), nullable=True), ])write_nullability_record(table, schema) # True if it was written, False if already currentprint(read_nullability_record(table.get_metadata()))# frozenset({('subject_id',), ('weight',)})Adding or changing a nullability record changes the table's identity in Orcapod. The
nullability is part of the schema, and the schema is part of a source's pipeline_hash and
content_hash. For the session table above:
from orcapod.sources import SpiralDBTableSource
# Before write_nullability_record:SpiralDBTableSource("my-project-123456", "session").output_schema()# (Schema({'subject_id': str | None}), Schema({'weight': float | None, 'booth': str | None}))
# After it:SpiralDBTableSource("my-project-123456", "session").output_schema()# (Schema({'subject_id': str}), Schema({'weight': float, 'booth': str | None}))The two sources have different pipeline_hash and content_hash values, and so do the nodes
built on them. A pipeline built on the table before it had a nullability record keeps its
stored results under its old identity: after the change, those nodes compute again under new
identities, and the earlier results are not reused (nothing stored is changed or deleted).
Pods annotated weight: float | None keep working, since they also accept float. So the
practical rule is: write the nullability record when the table is created, before anyone
builds pipelines on it.
The nullability record is stored under the metadata key __orcapod_nullability__ (format
orcapod.nullability, version 1), split over further keys when a wide table needs more than
SpiralDB's 8 KiB per metadata entry. Other SpiralDB clients ignore it and still see every
column as nullable.
DerivedSource
Section titled “DerivedSource”A DerivedSource is a source whose rows come from something Orcapod already has: another
source, or a node of a job. It has two uses.
Reading a job's results. A DerivedSource reads the stored results of a node of a
PipelineJob that has run
and presents them as a new source. You get one by calling as_source() on the node, for
example job.nodes["bmi"].as_source(). This is useful for chaining pipeline stages: run a
job, then use one of its outputs as the input of a new pipeline. The as_source() of a node
downstream of a join cannot yet be bound as a source of a new PipelineJob: its rows carry
the system tags of every joined source, and binding refuses it with SourceSpecMismatchError
(support is planned, ITL-1127).
Using a tag value as an input to a computation. Tags identify a row and data is what a
function computes on. A data-tag (the default for a declared tag) is both: a function
reads it through its data role, under the same name (see
Data-tags). A plain tag is a tag only, and a function pod
never receives it. When a plain tag's value is also needed as data (a session's start time
identifies the session, and it is also what you subtract to get an elapsed time), or you want
the value as a data column under another name, pass tags_as_data to get a source that keeps
the tag and also offers its value as a data column under a name you choose:
from datetime import datetime, timezone
from orcapod import function_podfrom orcapod.sources import DerivedSource, DictSource
start_1 = datetime(2026, 1, 5, 9, 0, tzinfo=timezone.utc)start_2 = datetime(2026, 1, 6, 9, 30, tzinfo=timezone.utc)
experiments = DictSource( data=[ {"subject_id": "m1", "session_start": start_1, "experiment_index": 0, "experiment_start": datetime(2026, 1, 5, 9, 1, tzinfo=timezone.utc)}, {"subject_id": "m1", "session_start": start_1, "experiment_index": 1, "experiment_start": datetime(2026, 1, 5, 9, 15, tzinfo=timezone.utc)}, {"subject_id": "m1", "session_start": start_2, "experiment_index": 0, "experiment_start": datetime(2026, 1, 6, 9, 32, tzinfo=timezone.utc)}, ], tag_columns=["subject_id", "session_start", "experiment_index"], tag_only_columns=True, source_id="experiments",)
with_start = DerivedSource( experiments, tags_as_data={"session_start": "session_start_value"})
tag_keys, data_keys = with_start.keys()print(tag_keys)# ('subject_id', 'session_start', 'experiment_index')print(data_keys)# ('experiment_start', 'session_start_value')
@function_pod(output_keys="onset_s")def onset_s(session_start_value: datetime, experiment_start: datetime) -> float: return (experiment_start - session_start_value).total_seconds()
onsets = onset_s.pod(with_start)print(onsets.as_table().select(["subject_id", "experiment_index", "onset_s"]).to_pandas())# subject_id experiment_index onset_s# 0 m1 0 60.0# 1 m1 1 900.0# 2 m1 0 120.0The result still has session_start as a tag, so it joins with other tables on that key as
before. Things to know:
- The mapping is
{tag column: new data column}. The new name must not already be a column and must not start with_; a wrong name raisesInputValidationErrorwhen the source is built. - The copy has exactly the tag's type, including whether it may be null. If the tag may be
null, annotate the function parameter as optional (
datetime | None). - The copy equals the tag when the source is read. After that it is ordinary data: a function
may replace it and
map_datamay rename it, while the tag stays as it was. - It works on any source (
DerivedSource(source, tags_as_data=...)) and on a job node (job.nodes["bmi"].as_source(tags_as_data=...)). - It is a new source, not a step in the origin's pipeline:
upstreamsis empty, and it has provenance columns of its own, like the result of handing the same rows toArrowTableSource. Use it in place of the origin. Its defaultsource_idistags_as_data:followed by a labelled hash built from the origin's identity and the mapping, so it changes when either changes; passsource_id=to choose a name. The mapping is part of the source'scontent_hasheither way, so two mappings never share one. - A saved job reloads a derived source whenever its origin can be reloaded (a CSV file, a Delta table, a database table). Over an in-memory source it is saved but not reloaded, like the in-memory source itself.
print(with_start.upstreams)# ()print(with_start.source_id)# tags_as_data:...:...Only a source can do this. An operator rearranges rows and columns, and every data value it emits was already a data value of its input; turning a tag value into a data value creates new data, which is what a source is for.
What a source does at a run
Section titled “What a source does at a run”A job node that has run holds its rows until it is stale, and staleness comes from modified
times only (see Running one node). So every run (job.run(),
run_async(), a job node's run(), a load-only run too) first asks each source feeding it,
once, whether its data changed: source.check_for_new_data() (developer, November 14, 2026,
ITL-1102). A source that finds new data takes it in and moves its modified time to that
moment, so every node below it runs; a source that finds nothing changes nothing, so a job
whose sources are unchanged computes nothing. head() and the notebook preview never check,
and neither does a source's own run() outside a job (source.run() does what
source.as_table() does); call source.check_for_new_data() to check a source by hand.
| Source | What the check does |
|---|---|
ArrowTableSource, DictSource, ListSource, DataFrameSource |
nothing: data held in memory cannot change; build a new source and bind it to run on new data |
PollingSource |
polls once (and fetches when the poll reports new data); its modified time moves only when the fetch returned rows, to the moment it found them (a cursor's modified_at is informational) |
DeltaTableSource |
reads the Delta log only; when the table's version moved, reads the table again and, if its rows changed (row order counts only without key columns, see below; a version that only compacted files changes nothing), holds the new rows |
SpiralDBTableSource |
holds no rows (every read scans the pinned version); reads the table's current version (metadata only: no row, and max_rows does not apply) and pins it when it is newer, so the next read returns the new rows; an asof you gave never moves; record ids are the canonical JSON of the key columns, so only new or changed rows are new entries (a version that moves without a change, if SpiralDB moves it on compaction, re-runs the nodes below and reuses every stored result) |
CSVSource |
reads the file's modification time and size; when either changed, reads the file again and holds the new rows if they changed (row order counts only without key columns) |
SQLiteTableSource, PostgreSQLTableSource, DBTableSource |
reads the whole table again and holds the new rows if they changed (with key columns in any order, so an unordered SELECT is not a change) |
CachedSource |
checks the source it wraps; when that source's modified time moved since the cache last took in its rows (also when another source over it made the check), compares the rows' record hashes with the rows it presents (before its first read, with the cache: a read; nothing is written) and moves only when it will present new rows, (a row removed from the wrapped source stays cached); the check guarantees nothing for two cached sources that share one cache (the same cache_path or table): give each its own cache (ITL-1143) |
DerivedSource |
over a source, checks it; reads the origin's rows again when the origin's modified time moved since this source read them (a source that took in new data, a job node that ran again); holds the new rows if they changed; one that holds no rows (over a job node that had not run) moves when its origin's time moves |
SourceProxy |
checks the source bound to it; its modified time follows that source's, and binding or unbinding moves it |
Rows read again are compared with the rows read before, as read from the table or file
(every column, a system_tag_columns column too), value by value, with a
NaN equal to a NaN (and -0.0 equal to 0.0), so a table holding a NaN is not a change at every
check. A check that raises part way changes nothing, so the next run checks again.
Row order follows the source's record id rule (developer, November 14, 2026). Without key columns a row's record id is its position: a change of row order is a change, so the source takes the rows in their new order and every row gets the id of its new position at once (the nodes below run again and reuse every stored function result, because results are stored by the hash of the row's data). With key columns, only content counts: the same rows in another order are no change. A source that cannot tell which rule applies counts order. The model is deliberately simple; a fuller design is ITL-1103 / ITL-1104.
A database table has no cheap and exact sign of a change, so each check of a SQLite or PostgreSQL source reads the whole table, one full read per run, as building the source did. For a large table, build the source over a narrower table or view, or keep the data in a store that records versions: a Delta check whose version did not move reads only the Delta log (one whose version moved reads the table again, as above), and a SpiralDB check reads only the table's metadata.
A table or file whose columns changed (a CSV file's types are inferred again on each read,
so a null in a column that had none is a change of type), a table that is gone or empty, or a
SpiralDB table dropped and created again, fails the check: the source's schema and table are
part of its identity, so build a new source to read it. A column that loses its nulls keeps
its nullable type (where nullability is inferred from the rows: a CSV file read without
schema=, a database whose connector gives no column information), so it is not a
change of columns.
A check that raises does not fail the run. The run issues one SourceCheckFailedWarning per
source, naming the source and the error; the source keeps its data and modified time; the
run proceeds with the data the source already holds, and its RunSummary lists the source
in unchecked_sources ("sources not checked for new data (see the warning): ..."). A job
node's run() that then runs nothing (the node holds a current run) makes no summary and
keeps its last_run: the warning is the report. A later failure to read the source raises
as it always has. A PollingSource reads by polling: a run that has to read it (a node
below it runs) polls it again, and a poll that fails again raises, as it did before the
check existed; a run whose nodes are all current proceeds. Reading a polling source's held
rows after a failed check is ITL-1139.
A source that takes in new data keeps its record id rule. Without key columns, a
row's record id comes from its position, so a row that moves (a reorder or rows inserted
before it) gets another id and is recorded as a new entry; its function result is still reused, because
results are stored by the hash of the row's data. Give a source that will take in new data
key columns that are unique per row (ITL-1104). A PollingSource requires them: its
rows arrive in batches, so a key identifies a polled row, and a batch that repeats a key
already polled is refused. Further revisions of what each kind checks are collected in
ITL-1103.
How sources add provenance columns
Section titled “How sources add provenance columns”When you create a source, Orcapod automatically adds two kinds of hidden columns to track data lineage:
Source-info columns (prefix _source_) store a provenance token for each data column.
For example, a data column weight gets a companion _source_weight column. These tokens
identify which source originally produced each value.
System tag columns track which source and which source row contributed each row. Every
source adds two of them, _tag_source_id::<hash> and _tag_record_id::<hash>, where <hash>
is the source's pipeline hash. These columns are used internally during
joins to maintain provenance through multi-stream operations.
Each row carries two system tags: the source's source_id and the row's record ID, the row
identifier itself (ITL-1154). By default the record ID of a row of an in-memory, CSV or Delta
source is its position, row_<n>, so a row that moves to another position under the same
source_id gets a different record ID; a database or SpiralDB table names its rows by their
primary key, as canonical JSON ({"trial":17}). Downstream pipeline entries
and sink_pod completion records are keyed on the system tags together with a hash of the
row's data, so the moved row is treated as a new entry: a function node records it again,
and its stored table then shows two entries with identical tag values but distinct record
IDs, and a sink_pod delivers it again. Because every key includes the hash of the row's own
data, no result is ever paired with the wrong row; a function's stored result for that data
is reused rather than recomputed.
Every source accepts record_id_columns, one or several key columns whose values name the
row instead ({"unit":"u1","time":1700000000}), so the record ID follows the row. Both
system tags must be stable: when you omit source_id an in-memory source is named after a
hash of its table, as <source kind>:<algorithm label>:<hex> (for example
dict:arrow_v1:000002fe…), which changes when any row changes or is reordered. Whenever row
order or content can vary between runs, pass an explicit, stable source_id and name
key columns that are unique per row; this avoids both the duplicate entries and the repeated
deliveries.
These columns are hidden by default. You can reveal them using ColumnConfig:
from orcapod.sources import DictSource
source = DictSource( data=[{"subject_id": "mouse_01", "age": 12, "weight": 25.3}], tag_columns=["subject_id"], tag_only_columns=True, source_id="mice",)
# Show source-info columnstable = source.as_table(columns={"source": True})print(table.column_names)# ['subject_id', 'age', 'weight', '_source_age', '_source_weight']print(table.column("_source_weight").to_pylist())# ['mice::row_0::weight']
# Show system tag columnstag_schema, data_schema = source.output_schema(columns={"system_tags": True})print(list(tag_schema))# ['subject_id', '_tag_source_id::...', '_tag_record_id::...']
# Show everythingtable = source.as_table(all_info=True)print(table.column_names)# ['subject_id', 'age', 'weight', '_tag_source_id::...', '_tag_record_id::...', '_content_hash', '_context_key', '_source_age', '_source_weight']Pass source_id to give the source a readable name in these provenance tokens. Without it,
Orcapod derives an id from the source's content hash.
Code example
Section titled “Code example”Create a DictSource, inspect its schema, and iterate over its stream:
from orcapod.sources import DictSource
source = DictSource( data=[ {"subject_id": "mouse_01", "age": 12, "weight": 25.3}, {"subject_id": "mouse_02", "age": 8, "weight": 22.1}, {"subject_id": "mouse_03", "age": 15, "weight": 27.8}, ], tag_columns=["subject_id"], tag_only_columns=True,)
# Inspect the schematag_schema, data_schema = source.output_schema()print("Tag schema:", dict(tag_schema))# Tag schema: {'subject_id': <class 'str'>}print("Data schema:", dict(data_schema))# Data schema: {'age': <class 'int'>, 'weight': <class 'float'>}
# Get column namestag_keys, data_keys = source.keys()print("Tag keys:", tag_keys)# Tag keys: ('subject_id',)print("Data keys:", data_keys)# Data keys: ('age', 'weight')
# Iterate over (Tag, Data) pairsfor tag, data in source: print(f" Tag: {tag.as_dict()}, Data: {data.as_dict()}")# Tag: {'subject_id': 'mouse_01'}, Data: {'age': 12, 'weight': 25.3}# Tag: {'subject_id': 'mouse_02'}, Data: {'age': 8, 'weight': 22.1}# Tag: {'subject_id': 'mouse_03'}, Data: {'age': 15, 'weight': 27.8}
# Convert to a PyArrow tabletable = source.as_table()print(table.to_pandas())# subject_id age weight# 0 mouse_01 12 25.3# 1 mouse_02 8 22.1# 2 mouse_03 15 27.8How it connects to other concepts
Section titled “How it connects to other concepts”- Sources produce Streams -- immutable sequences of (Tag, Data) pairs
- Streams flow into Operators for structural transforms (joins, filters, column selection)
- Streams flow into Function Pods for value-level transforms
- Every source has a
content_hash()andpipeline_hash()-- see Identity & Hashing for how these work