Skip to content
This site documents the development version of Orcapod (the main branch), which may differ from the latest release. For the releases, see PyPI and the changelog.

Hashing Reference

This document is a developer-facing reference for every hash computation site in the orcapod-python framework. For conceptual background on the two identity chains (content_hash and pipeline_hash), see Identity & Hashing. Every hash listed here that Orcapod stores or embeds in a name is pinned as a golden value; how released values are protected is described in Golden Hashes.


The table below summarises all 15 hash computation sites across the six usage groups documented in the sections that follow. (Site 6b is numbered alongside site 6 rather than appended, because it is a variant of the same system tag mechanism.)

# Site Algorithm Output format One-line guarantee
1 content_hash() SemanticAwarePythonHasher → JSON + SHA-256 ContentHash Unique per semantic content of the object; data-inclusive
2 pipeline_hash() SemanticAwarePythonHasher → JSON + SHA-256 ContentHash Unique per pipeline topology + schema; data-exclusive
3 Schema hash (system tag column naming) SemanticAwarePythonHasher.hash_object((tag_schema, data_schema)).to_hex(n) Truncated hex str Unique per (tag_schema, data_schema) pair; embedded in system tag column names
4 Default source_id of an in-memory source "<source kind>:" + StarfixArrowHasher.hash_table(table).to_string() str, <source kind>:<algorithm label>:<hex> Unique per source kind and raw table content; used as the source's name when none is given
5 Per-row record_id (system tag value) the row identifier itself: row_<n>, or the canonical JSON of the key columns (key_row_identifier) str (large_string) Names the row within its source; stable as long as its position or its key is
6 Join system tag suffix stream.pipeline_hash().to_hex(n) + :{idx} appended to column name Column name suffix Unique per (input topology, canonical join position); encodes full join lineage in column name
6b Aggregating operator system tag fold SHA-256 of the canonical JSON list of member values, after the operation's label, for both system tags "<fold label>:<hex>" str, plus a column name suffix Deterministic per ordered member set; keeps system tags scalar through a many-to-one reduction
7 compute_base_entry_id() StarfixArrowHasher.hash_table(system_tags + INPUT_DATA_HASH_COL) bytes (b"method:digest") Unique per (node, tag lineage, input_data content) across all recomputation attempts
8 compute_pipeline_entry_id() StarfixArrowHasher.hash_table(system_tags + INPUT_DATA_HASH_COL + recomputation_index) bytes (b"method:digest") Unique per (node, tag lineage, input_data content, recomputation attempt)
9 Side-effect record_id StarfixArrowHasher.hash_table(system_tags + INPUT_DATA_HASH_COL + recomputation_index=0) bytes Unique per (tag lineage, input_data content); pod-version scoped via table path (uri + pipeline_hash())
10 invocation_hash f"{serialize(pipeline_hash_ch)}::{serialize(record_id_hash_ch)}" str Unique per (pod topology, tag lineage, input_data content); pod-version scoped via table path
11 Output schema hash SemanticAwarePythonHasher.hash_object(output_data_schema).to_hex() str (bare hex digest) Unique per output schema definition; embedded in function URI
12 run_id uuid.uuid4().hex[:16] 16-char hex str Non-deterministic; unique per pipeline execution
13 snapshot_hash hashlib.sha256(sorted labelled leaf content hashes joined by newline).hexdigest()[:16] 16-char hex str Unique per (DAG leaf topology + data state) at run time
14 datagram_uuid uuid7() normalised to stdlib uuid.UUID uuid.UUID Unique per datagram instance; time-ordered; not a content hash

A ContentHash pairs a label (method) with a digest. One rule holds for every hash Orcapod computes (developer, November 15, 2026; ITL-1154, revising ITL-945):

A hash always carries its label, except where it is written into a name, whose grammar fixes which algorithm produced it.

# Rule
1 A label names the algorithm that produced the digest and nothing else. "Algorithm" is the whole hashing procedure: the canonical form, any preparation, the encoding and the digest function.
2 The format is <algorithm>_v<N> with an integer N; a label never contains :. In v0.3 every label is _v1. The file content hash keeps sha256, the name of the digest function applied to the file's bytes: the one exception.
3 A label changes only when its algorithm would give an existing input a different digest. A change that leaves existing digests alone keeps the label.
4 A hash fed into another hash carries its label. A nested hash in a structure is {"__type__": "hash", "label": "<label>", "digest": "<hex>"}; a value-hashed cell in a table is <identifier>::<label>:<digest>; the entry-id preimage, the operator-cache row, the pandas value hash and a directory's file entries take b"<label>:" + digest; the run snapshot takes "<label>:<hex>". So two algorithms that happen to produce the same digest never feed the same input (a File whose bytes are some value's canonical JSON, and that value's semantic_v1 hash), and renaming a label moves every hash it feeds. (This reverses ITL-945's rule 6, "a label never enters a hash computation".)
5 A hash stored as a value in its own column or field carries its label: "<label>:<hex>" as text, b"<label>:" + digest in a binary column.
6 A hash inside a name is the bare hex digest: path components, column names, the function uri, operator identity, source-node slot names, the run URI. The name's grammar fixes which algorithm produced the digest. A name that is later fed into a hash (the function uri in a pod's identity, system-tag column names in an entry-id table) is still a name: its digest moves whenever what it was computed from moves.
7 Source ids. A source id is the source's name; a given one is used exactly as given. A default names its kind first, <kind>:<rest>: <kind>:<label>:<hex> when built from a hash (dict:arrow_v1:<hex>, derived:semantic_v1:<hex>, tags_as_data:semantic_v1:<hex>), <kind>:<location> when built from where the data lives (csv:<path>). A value folded by an operation is <operation label>:<hex> (batch_v1:<hex>, group_v1:<hex>). See "Default source ids and row identifiers".
8 Record ids. A record id is the row identifier itself, as text: row_<n> (the position) or the canonical JSON of the key columns ({"unit":"u1","time":1700000000}). It is not a hash and holds no source id; the source id is the other system tag. A folded record id is <operation label>:<hex>, like a folded source id. A source-info token is <source id>::<record id>::<column>. A record id holds key values, which may contain ::; a source id and a column name never do (both are refused), so a token is split at its first and its last :: only.
9 A random identifier (uuid7, uuid4: datagram and result ids, run ids) is not a hash and carries no label.
10 Handler routes. A value handler describes its value as tagged data (Encoded), hashed by semantic_v1: bytes / bytearray as {"__type__": "bytes", "hex": ...}, uuid.UUID as {"__type__": "uuid", "hex": ...}. It returns a labelled hash of its own only when the identity needs a dedicated algorithm: external content (File, Directory) or a library value's canonical form (numpy.ndarray, torch, pandas, Polars, pa.Table, a Polars expression).
11 Column value handlers. A logical type's column is hashed through a value handler only when its stored form is not its identity. A UUID column's 16 stored bytes are its identity, so starfix hashes it as stored (UUIDHandler.COLUMN_VALUE_HANDLER = False); a bare uuid.UUID keeps its tagged value hash.
12 The implementer of a logical type chooses the label of its value hash, in the same format.
13 Each label is defined in exactly one place, in code beside its algorithm. The data context file names the hasher class and does not repeat the label.

Grammar restrictions. The encodings above decode in one way only because: the registry refuses ; in an extension name, so the identifier text <extension name>;<hex of the metadata> splits at its one ;; the extension metadata is written as hex, so nothing after the ; is a :; and a label contains no :, so <label>:<digest> splits at its first :. The container cell keeps its element count prefix, so it is self-contained and does not depend on the cell boundary. tests/test_hashing/test_hash_grammar.py fails if any of these stops holding.

The labels of v0.3:

Label Names Defined in
semantic_v1 the structure hash: a Python structure expanded to tagged JSON, SHA-256 of its canonical text SEMANTIC_HASH_LABEL, hashing/semantic_hashing/semantic_hasher.py
arrow_v1 the Arrow table and schema hash (§9) ARROW_HASH_LABEL, hashing/arrow_hashers.py
arrow_ordered_v1 the Arrow table hash with column and struct-field order included (§9), fed into the pandas value hash ARROW_ORDERED_HASH_LABEL, hashing/arrow_hashers.py
directory_v1 the directory hash: a SHA-256 Merkle tree over the entries, file leaves taken with their label DIRECTORY_HASH_LABEL, hashing/directory_hashers.py
ndarray_v1 the array value hash (§8) NumpyArrayHandler.HASH_LABEL, hashing/semantic_hashing/builtin_handlers.py
pandas_dataframe_v1 the pandas.DataFrame value hash (§10) PANDAS_DATAFRAME_HASH_LABEL, hashing/pandas_value_hash.py
pandas_series_v1 the pandas.Series value hash (§10) PANDAS_SERIES_HASH_LABEL, hashing/pandas_value_hash.py
torch_tensor_v1 the torch tensor value hash (§8): the element digest, then requires_grad, the lazy conjugate and negative bits, the strides and the class TORCH_TENSOR_HASH_LABEL, logical_types/torch_type.py
polars_expr_v1 the Polars expression hash (§1c) POLARS_EXPR_HASH_LABEL, hashing/semantic_hashing/polars_expr.py
polars_dataframe_v1 the Polars DataFrame value hash (§1e) PolarsDataFrameHandler.HASH_LABEL, hashing/semantic_hashing/builtin_handlers.py
polars_series_v1 the Polars Series value hash (§1e) PolarsSeriesHandler.HASH_LABEL, hashing/semantic_hashing/builtin_handlers.py
batch_v1 the folded source id and record id of Batch (site 6b) BATCH_FOLD_LABEL, core/operators/batch.py
group_v1 the folded source id and record id of GroupBy (site 6b), and the merged source-info token of a data-tag group key GROUP_BY_FOLD_LABEL, core/operators/group_by.py
join_v1 the merged source-info token of a key that is a data-tag on several inputs of a Join (site 6b) JOIN_FOLD_LABEL, core/operators/join.py
sha256 the file content hash: SHA-256 of the file's bytes the file hasher's algorithm

Where a hash appears, and in which form:

Form Where
Labelled value __input_data_hash, __output_data_hash, __node_content_hash, __pipeline_base_entry_id, the pipeline table's record id, the side-effect log's record_id and record_id_hash, the record id (_record_hash) of operator-node and cached-source tables, the _content_hash column, the hashes in a saved job file, InvocationContext.invocation_hash
Labelled, inside a hash input a nested hash in a structure: {"__type__": "hash", "label": "<label>", "digest": "<hex>"}; a value-hashed cell in a table: b"<extension name>;<hex of the extension metadata>::<label>:" + digest (the logical type's whole identifier, which the hashed field also keeps as its extension name and metadata, so the type is part of the hash of a column with no value too); the input data hash in the entry-id preimage (§3); the node content hash in the operator-node record hash; the sink hashes of the run snapshot (site 13); the ordered Arrow digest in a pandas value hash (§10); a file's hash in a directory entry
Bare digest in a name the function uri (its second component), the path components schema:<pipeline digest>, instance:<content digest> and node:<content digest>, system-tag column names, the operator uri, the function uri at the start of a function output's source-info token, source-node slot names, the run URI (<job>@<snapshot>)
Labelled hash inside a source id every default source id built from a hash: <source kind>:<label>:<hex> for an in-memory source (arrow_table:, dict:, list:, data_frame:, then the labelled table hash) and a DerivedSource (derived:, then the node's labelled content hash; also what node.as_source() returns), defined once by hash_source_id in core/sources/source_ids.py; tags_as_data:<label>:<hex> of a tags_as_data derivation (DerivedSource._default_source_id); <fold label>:<hex> of a folded source id (fold_system_tag_values)

A function node's table in v0.3, with the values it stores:

from orcapod import PipelineJob, function_pod
from orcapod.databases import InMemoryArrowDatabase
from orcapod.sources import DictSource
from orcapod.types import ContentHash
@function_pod(output_keys="doubled")
def double(x: int) -> int:
return 2 * x
rule_store = InMemoryArrowDatabase()
rule_job = PipelineJob(name="my_pipeline", database=rule_store)
with rule_job:
numbers = DictSource(
data=[{"k": "a", "x": 1}], tag_columns=["k"], tag_only_columns=True, source_id="numbers"
)
_ = double.pod(numbers, label="double")
rule_job.run()
# A hash kept as a value carries its label.
rule_node = rule_job.nodes["double"]
print(rule_node.pipeline_hash())
# semantic_v1:c4294cdcdc37c18c76b88f83a50a21e71293c77dc25a9104c7b8d710b0041235
# A hash inside a name is the bare digest: the function uri and the table path.
print(double.pod.uri)
# ('double', '32567b339b2e50ee4f435e6234441a0415db766db953431439e81cf471c94945', 'v0', 'python.function.v0')
pipeline_table = next(p for p in rule_store.list_sources() if "_result" not in p)
print("/".join(pipeline_table))
# my_pipeline/double/32567b339b2e50ee4f435e6234441a0415db766db953431439e81cf471c94945/v0/python.function.v0/schema:c4294cdcdc37c18c76b88f83a50a21e71293c77dc25a9104c7b8d710b0041235/pdb_v2
# Stored columns hold labelled hashes; the system-tag column name holds a bare digest.
stored = rule_store.get_all_records(pipeline_table)
print(ContentHash.from_prefixed_digest(stored["__input_data_hash"][0].as_py()))
# arrow_v1:000002fcbe7eec1821a57679d34084a583d8282798563aa706ecb9c0c24382580de4df
print(ContentHash.from_prefixed_digest(stored["__node_content_hash"][0].as_py()))
# semantic_v1:43529fee65d63d64303db070024f9dbae24ba6f9c3e9ec0ebc12c79f5dde6ddb
print([name for name in stored.column_names if name.startswith("_tag_source_id")])
# ['_tag_source_id::3bd6e739b7e15bd3402717af3bafd487d480f3b382028cb22c2250fcb6a90e64']

The path is <job>/<function name>/<output-schema digest>/v<major>/<wrapper kind>/schema:<pipeline digest>/<table layout>. The path holds bare digests, but those digests were computed from labelled nested hashes: renaming a label moves every hash it feeds, and with it the paths and keys built from them.


A source id is the source's name (rule 7). A source_id the user gives is used exactly as given, by every source. With none, a source builds a default that names its kind first: the class's SOURCE_KIND, which is also its config type key (one vocabulary). Every default is built in core/sources/source_ids.py: hash_source_id for a default built from a hash, location_source_id for one built from where the data lives. A default never contains ::, the separator of a source-info value: a location that holds :: has no default, and the source refuses it until a source_id is given; a given source id that holds :: is refused too, and so is a column name that holds it (developer, November 15, 2026: "every source ought to have some default naming convention for the source id that reflects the category, but we should make it totally overridable"; November 16, 2026, ITL-1154).

Source Kind Default source id
ArrowTableSource arrow_table arrow_table:arrow_v1:<hex> of the table
DictSource dict dict:arrow_v1:<hex> of the table
ListSource list list:arrow_v1:<hex> of the table
DataFrameSource data_frame data_frame:arrow_v1:<hex> of the table
DerivedSource (a node's records; node.as_source()) derived derived:semantic_v1:<hex> of the node's content hash
DerivedSource with tags_as_data tags_as_data (an operation) tags_as_data:semantic_v1:<hex> of the derivation
CSVSource csv csv:<file path as passed>
DeltaTableSource delta_table delta_table:<path or URI of the table folder as passed>
SQLiteTableSource sqlite_table sqlite_table:<db path as passed>:main.<table> (SQLite names a table <schema>.<table>; the main database's schema is main)
PostgreSQLTableSource postgresql_table postgresql_table:<database>.<schema>.<table> (the three-part name, from current_database() and current_schema())
DBTableSource (any other connector) db_table none: source_id is required. The connector protocol exposes no database address, and the table name alone does not say which database the table is in (developer decision, November 16, 2026)
SpiralDBTableSource spiraldb_table spiraldb_table:<project>.<dataset>.<table>@<table id> (Spiral's own table identifier, then SpiralDB's table id; SPIRALDB_SOURCE_ID_HAS_TABLE_ID)
PollingSource polling_source polling_source:semantic_v1:<hex> of impl.identity() (never its repr)
CachedSource cached the wrapped source's id
folded source id of Batch / GroupBy (a system-tag value, not a source) batch_v1:<hex> / group_v1:<hex>

A location default names the data where it lives, so two sources of one kind share it only when they read the same data. The generic DBTableSource knows no database, so it has no default and refuses to be built without a source_id. A location whose parts cannot be told apart has no default (joined_location): a SQLite table name holding :, a PostgreSQL or SpiralDB name holding ., a SpiralDB address or table id holding @. Such a source needs a source_id.

A row's record id is its row identifier (site 5), chosen by record_id_columns (a column name or several, in key order; every source accepts it):

Row identifier When
row_<n> the row's position: the default of in-memory, CSV and Delta sources, or record_id_columns=()
{"<column>":<value>,...} key columns: those named, or by default the primary key of a database table (SQLite, PostgreSQL, another connector) and of a SpiralDB table. Canonical JSON in key order, no spaces; 17 and "17" differ

A key value must be a string, an integer, a boolean, a timestamp (developer, November 16, 2026), written as ISO-8601 text from its stored integer (a naive timestamp without an offset, a zoned one in UTC with Z, the fraction digits fixed by the unit), or a date, YYYY-MM-DD. Any other type (floats, bytes, times of day, UUIDs, containers, logical types) is refused when the source is built, and so is a null key value and a key repeated in two rows. A database table without a primary key is refused unless record_id_columns names a key or asks for the position (()); a SpiralDB table refuses (); a PollingSource needs record_id_columns, since a position names a row only within its batch (ITL-1158).

Name a source (source_id) and a key (record_id_columns) when its results must stay reusable as its data changes: a default in-memory source id is the hash of the whole table, and a position moves when rows are inserted or reordered.

Every place Orcapod produces a digest, and every place it reads one, with its class (ITL-1154). The classes follow the one-line rule above:

Class Form
into a hash carries its label
stored value carries its label ("<label>:<hex>" text, b"<label>:" + digest binary)
in a name bare hex; the name's grammar fixes the algorithm
id from a recipe no label (uuid7, uuid4; a record id is a row identifier, not a hash)
display only logs, observers, hooks, error text, in-memory keys and ordering: never stored, never hashed

A new hash site is added here, with its class. tests/test_hashing/test_hash_site_inventory.py runs a trace (tests/test_hashing/hash_site_trace.py) that records every library call site producing a digest (ContentHash, hashlib, uuid5, starfix) or reading one (.digest, to_hex, to_string, to_prefixed_digest, str(), comparison, hash()), and fails when a call site is missing from the "Call sites" column below. The same plugin runs over a whole test session with -p tests.test_hashing.hash_site_trace.

Location What is hashed Algorithm / label Used as Call sites
structure hasher a Python value as a tagged JSON tree SHA-256 of canonical JSON, semantic_v1 every content_hash() / pipeline_hash(), schema, type and function hash hashing/semantic_hashing/semantic_hasher.py:SemanticAwarePythonHasher._hash_to_content_hash
Arrow hasher a table after the semantic visitor starfix, arrow_v1; hash_table_ordered: arrow_ordered_v1; schema: arrow_v1 row data hashes, entry ids, row keys, default source ids, table values hashing/arrow_hashers.py:StarfixArrowHasher.hash_table, hashing/arrow_hashers.py:StarfixArrowHasher.hash_table_ordered, hashing/arrow_hashers.py:StarfixArrowHasher.hash_schema, hashing/arrow_hashers.py:StarfixArrowHasher._digest
file hasher a file's bytes the configured digest function, label = its name (sha256 in the data context; others: ITL-1155) File value hash hashing/hash_utils.py:hash_file, hashing/hash_cachers.py:SqliteHashCacher.get, hashing/hash_cachers.py:SqliteHashCacher.put
directory hasher a directory tree Merkle tree of SHA-256 entries, directory_v1; file leaves taken with their label Directory value hash hashing/directory_hashers.py:BasicDirectoryHasher.hash_directory, hashing/directory_hashers.py:_hash_dir
array value hash an array: element digest, then order, byte order, struct layout SHA-256, ndarray_v1 (the element digest is a step of the same algorithm) numpy.ndarray value hash logical_types/numpy_type.py:array_value_digest, logical_types/array_format.py:array_digest_v1
tensor value hash element digest, flags, strides, class SHA-256, torch_tensor_v1 torch.Tensor value hash logical_types/torch_type.py:tensor_digest
pandas value hash descriptor, \0, b"arrow_ordered_v1:" + D SHA-256, pandas_dataframe_v1 / pandas_series_v1 pandas value hash hashing/pandas_value_hash.py:pandas_frame_content_hash, hashing/pandas_value_hash.py:pandas_series_content_hash, hashing/labelled_hash.py:labelled_content_hash, hashing/semantic_hashing/builtin_handlers.py:PandasDataFrameHandler.digest, hashing/semantic_hashing/builtin_handlers.py:PandasSeriesHandler.digest
Polars value hash descriptor, \0, starfix digest called directly (a step of this algorithm) SHA-256, polars_dataframe_v1 / polars_series_v1 Polars value hash logical_types/polars_type.py:_digest, logical_types/polars_type.py:_values_digest
Polars expression hash the expression's canonical tree SHA-256 of canonical JSON, polars_expr_v1 PolarsFilter identity hashing/semantic_hashing/polars_expr.py:polars_expr_content_hash
value handlers with an ignore list wrap the value digests above the type's label top level, nested, Arrow cell logical_types/ignore_list.py:IgnoreListHandler.handle, logical_types/ignore_list.py:IgnoreListHandler.hash_ignoring
parse a stored hash "<label>:<hex>" or b"<label>:" + digest the stored label saved jobs and pipelines, proxies, a node's saved run hash, a cached output hash types.py:ContentHash.from_string, types.py:ContentHash.from_prefixed_digest
folds of Batch / GroupBy the canonical JSON list of a group's member values, for each system tag SHA-256 hex after the operation's label: batch_v1 (batch.py), group_v1 (group_by.py); one helper folded source id and record id utils/arrow_utils.py:fold_system_tag_values
run snapshot sorted labelled sink hashes SHA-256 hex, 16 digits the run URI <job>@<snapshot> (a name) pipeline/job.py:PipelineJob._prepare_run, pipeline/job.py:PipelineJob._prepare_run.<locals>.<genexpr>
user-class identity a class's identity record SHA-256 hex prefix extension name name@<fingerprint> (a name) logical_types/user_class_identity.py:identity_string
database registry key a database config, canonical JSON SHA-256 hex, 8 digits db_<hex> key in a saved job (a name) pipeline/serialization.py:DatabaseRegistry._make_key
SpiralDB nullability record the nullability path list SHA-256 hex chunk key and checksum of a stored record (a name) databases/spiraldb_nullability.py:_encode, databases/spiraldb_nullability.py:read_nullability_record
synthesized struct class name the field signature MD5 hex, 8 digits a Python class name (a name) (not reached by a traced test) semantic_types/universal_converter.py
random ids nothing uuid4, uuid7 run ids, datagram and record uuids, observer ids, correlation keys (not hashes)
unused code Arrow tables SHA-256, SHA3, BLAKE2b no caller; removal is ITL-1156 (not reached) hashing/arrow_utils.py, hashing/arrow_serialization.py
Location Hash consumed Into Form Call sites
nested hash in a structure any nested hash: types, schemas, upstream streams, value hashes the enclosing structure hash {"__type__": "hash", "label": ..., "digest": ...} hashing/semantic_hashing/semantic_hasher.py:_nested_hash
union and generic alias arguments argument type hashes, sorted by digest, then label (ordering only) the type hash, via the nested form as above hashing/semantic_hashing/builtin_handlers.py:_sorted_hashes, hashing/semantic_hashing/builtin_handlers.py:_sorted_hashes.<locals>.<lambda>
logical-type cell, scalar and container a value-hashed cell's value hash the Arrow table hash <identifier>::<label>:<digest>; a container cell: count, then per element a marker, a u64 length and the element token hashing/visitors.py:SemanticHashingVisitor.visit_extension
entry-id preimage the input row's data hash entry ids, pipeline record id, side-effect log key b"<label>:" + digest core/nodes/function_node.py:_build_record_id_preimage
operator-cache row the node content hash _record_hash b"<label>:" + digest core/nodes/operator_node.py:OperatorJobNode._store_output_stream
run snapshot sink nodes' content hashes the snapshot "<label>:<hex>" pipeline/job.py:PipelineJob._prepare_run.<locals>.<genexpr>
pandas value hash the ordered Arrow digest the pandas value hash b"arrow_ordered_v1:" + digest hashing/pandas_value_hash.py:pandas_frame_content_hash, hashing/pandas_value_hash.py:pandas_series_content_hash
directory entry a file's hash the directory hash b"<label>:" + digest hashing/directory_hashers.py:_hash_dir
ListSource tag function (content mode) the semantic hash of the function's source text the source's identity the ContentHash, nested core/sources/list_source.py:ListSource._hash_tag_function
ignore-list value handlers the pandas digest the handler wraps under its own label the handler's ContentHash its own procedure's digest hashing/semantic_hashing/builtin_handlers.py:PandasDataFrameHandler.digest, hashing/semantic_hashing/builtin_handlers.py:PandasSeriesHandler.digest
Value Call sites
__input_data_hash, __output_data_hash core/result_cache.py:ResultCache.store, core/result_cache.py:ResultCache.lookup, core/nodes/function_node.py:FunctionJobNode._add_pipeline_record
__node_content_hash core/nodes/function_node.py:FunctionJobNode._add_pipeline_record, core/nodes/function_node.py:FunctionJobNode._fetch_joined_records, core/nodes/function_node.py:FunctionJobNode._stored_record_count, core/nodes/operator_node.py:OperatorJobNode._store_output_stream, core/nodes/operator_node.py:OperatorJobNode._filter_by_content_hash
__pipeline_base_entry_id, the pipeline record id core/nodes/function_node.py:FunctionJobNode.compute_base_entry_id, core/nodes/function_node.py:FunctionJobNode.compute_pipeline_entry_id
_record_hash of operator-cache and cached-source tables core/nodes/operator_node.py:OperatorJobNode._store_output_stream, core/sources/cached_source.py:CachedSource._live_records
side-effect log record_id, record_id_hash side_effects.py:_execute_side_effect_row
the _content_hash column core/streams/arrow_table_stream.py:ArrowTableStream._assemble_table.<locals>.hashes, core/function_pod.py:FunctionPodStream.as_table.<locals>.all_content_hashes, core/function_pod.py:FunctionPodStream.head.<locals>.shown_content_hashes, core/function_pod.py:FunctionPodStream.head.<locals>.<lambda>, core/nodes/function_node.py:FunctionNodeBase.as_table.<locals>.all_content_hashes, core/nodes/function_node.py:FunctionNodeBase._head_of_pairs.<locals>.<lambda>, side_effects.py:_table_of_rows
__pf_var_function_signature_hash, __pf_var_function_content_hash core/data_function.py:PythonDataFunction.__init__
saved pipeline and job files pipeline/graph.py:Pipeline.save, pipeline/job.py:PipelineJob.save, pipeline/job.py:PipelineJob.as_pipeline, core/sources/base.py:RootSource._identity_config
cached file hashes hashing/hash_cachers.py:SqliteHashCacher.put
InvocationContext.invocation_hash (<label>:<hex>::<label>:<hex>; its redesign is a side-effect follow-up) invocation.py:_serialize_component
Name Call sites
function uri (output-schema digest) core/data_function.py:DataFunctionBase.output_data_schema_hash
operator uri core/operators/static_output_pod.py:StaticOutputOperatorPod.uri
schema: / instance: / node: path components core/nodes/function_node.py:FunctionNodeBase.node_identity_path, core/nodes/operator_node.py:OperatorNodeBase.node_identity_path, side_effects.py:SideEffectJobNode.attach_databases, core/sources/cached_source.py:CachedSource._unversioned_cache_path
source-node slot names core/nodes/source_node.py:SourceNode.from_stream, core/nodes/source_node.py:SourceJobNode.from_stream
source system-tag column name (schema digest) utils/schema_utils.py:compute_schema_hash
system-tag suffixes of Join, MergeJoin, Batch, GroupBy core/operators/join.py:Join._compute_system_tag_suffixes, core/operators/join.py:Join._predict_system_tag_schema, core/operators/join.py:Join.static_process, core/operators/merge_join.py:MergeJoin._predict_system_tag_schema, core/operators/merge_join.py:MergeJoin.binary_static_process, core/operators/batch.py:Batch.unary_output_schema, core/operators/batch.py:Batch.unary_static_process, core/operators/group_by.py:GroupBy.unary_output_schema, core/operators/group_by.py:GroupBy.unary_static_process
a saved hash shown as a digest core/nodes/function_node.py:_bare_digest
source ids (<kind>:<label>:<hex>, rule 7) core/sources/source_ids.py:hash_source_id, core/sources/derived_source.py:DerivedSource._default_source_id

A name can itself be fed into a hash (the function uri in a pod's identity, system-tag column names in an entry-id table, a source id in a source's identity and record ids). It stays a name: its grammar fixes the algorithm, and it moves whenever what it was computed from moves.

A record id is the row identifier (rule 8), built from row positions or key values by stream_builder.row_identifiers, never from a digest, so it is not a hash site.

Use Call sites
observer events (node_hash) core/nodes/function_node.py:FunctionJobNode.execute, core/nodes/function_node.py:FunctionJobNode.async_execute, core/nodes/function_node.py:FunctionJobNode._execute_load_only, core/nodes/function_node.py:FunctionJobNode._async_execute_load_only, core/nodes/function_node.py:FunctionJobNode._async_execute_cache_only, core/nodes/operator_node.py:OperatorJobNode.execute, core/nodes/operator_node.py:OperatorJobNode.async_execute, core/nodes/source_node.py:SourceNodeBase.execute, core/nodes/source_node.py:SourceNodeBase.async_execute
hooks (pod_hash) core/function_pod.py:_FunctionPodBase._build_post_run_payload
in-memory graph keys and sort keys pipeline/base.py:AbstractPipelineBase.compile, pipeline/base.py:AbstractPipelineBase.compile.<locals>.<lambda>, pipeline/base.py:AbstractPipelineBase._record_invocation, pipeline/base.py:AbstractPipelineBase.to_invocations, pipeline/base.py:AbstractPipelineBase.from_invocations, pipeline/job.py:PipelineJob.compile
canonical input order (sort by digest). Inputs with equal pipeline hashes (developer decision, November 16, 2026, ITL-1154: kept as it is): system-tag column names follow the stable pipeline-hash sort; each row's values are sorted by (source id, record id) by sort_system_tag_values, so the result does not depend on argument order; the operator's identity holds its inputs as a frozenset core/operators/join.py:Join.order_input_streams.<locals>.<lambda>, core/operators/merge_join.py:MergeJoin._canonical_order.<locals>.<lambda>
in-memory identity and repr core/base.py:ContentIdentifiableBase.__hash__, hashing/semantic_hashing/content_identifiable_mixin.py:ContentIdentifiableMixin.__hash__, hashing/semantic_hashing/content_identifiable_mixin.py:ContentIdentifiableMixin.__eq__, hashing/semantic_hashing/content_identifiable_mixin.py:ContentIdentifiableMixin.__repr__
comparing a stored hash with the computed one; error text core/nodes/operator_node.py:OperatorJobNode._read_run_records, core/sources/source_proxy.py:SourceProxy.bind, core/data_function_proxy.py:DataFunctionProxy.bind, pipeline/job.py:PipelineJob.bind, errors.py:EphemeralResultMissingError.__init__

The pipeline below exercises hash sites 1--14; it contains no aggregating operator, so site 6b is not covered. source_id values are set explicitly so every content hash is reproducible across runs. Run the script to regenerate the values in this section:

uv run python scripts/hash_audit_example.py
from orcapod.core.function_pod import function_pod
from orcapod.core.nodes.function_node import FunctionJobNode
from orcapod.core.operators.join import Join
from orcapod.core.sources.dict_source import DictSource
from orcapod.databases.in_memory_databases import InMemoryArrowDatabase
from orcapod.side_effects import SideEffectPod, InvocationContext
scores = DictSource(
[{"student": "alice", "math": 90}, {"student": "bob", "math": 75}],
tag_columns=["student"], tag_only_columns=True,
source_id="scores_v1",
)
attendance = DictSource(
[{"student": "alice", "days": 180}, {"student": "bob", "days": 160}],
tag_columns=["student"], tag_only_columns=True,
source_id="attendance_v1",
)
joined = Join()(scores, attendance) # sites 3–6
@function_pod(output_keys="grade")
def grade(math: int, days: int) -> float:
return math * 0.7 + days / 180.0 * 30.0
pod = grade.pod
node = FunctionJobNode(pod, joined, pipeline_database=InMemoryArrowDatabase())
captured: list[InvocationContext] = []
def _log_fn(ctx: InvocationContext, **kwargs) -> None:
captured.append(ctx)
side_pod = SideEffectPod(_log_fn, ctx_arg_name="ctx")
Object content_hash() pipeline_hash()
scores semantic_v1:80462569849407b33efc33ecc0381cd0858643c31d9349a350eb5fbc71e9208f semantic_v1:a19744e791fb1f4c5431daad4380327d8c45f52d4f9353ce944b287265097650
attendance semantic_v1:48795b2977ca4fb5cadc4d8f57436a593555a36d69d1b39126b8a6f177847106 semantic_v1:eb2bdc439e8a48f8246b2ea8d52227418571f43c998be577b5a1c0794ca4d749
joined semantic_v1:792d218f5fcbceab6d8012bf35ef49c663c04b4e04e8c06d34e5879150a04964 semantic_v1:a308714c9fb11116c31a493dd1a09be8b43654b8b8e07acbfbd9fbfb72998571
pod semantic_v1:781abfcf2b8637daa3393e0e94e4b9b5b9709c7869cf44049819a2946f9d18ec semantic_v1:781abfcf2b8637daa3393e0e94e4b9b5b9709c7869cf44049819a2946f9d18ec
node semantic_v1:e51bb0b27c83974fa4bd5f92d2144036e4ccb71f40448bec72f0edced01391d7 semantic_v1:0ae6bcf82d5a5e52b4ba8de2bc389335d448ad5dc2703ab8e3149199b77c2ba0

pod.content_hash() == pod.pipeline_hash() because grade has no ctx_arg_name, so identity_structure() and pipeline_identity_structure() return the same value.

scores.pipeline_hash() ≠ attendance.pipeline_hash() even though both sources have tag_columns=["student"] — their data schemas differ ({math: int} vs {days: int}).

Site Item Value
3 Schema hash (scores) a19744e791fb1f4c5431daad4380327d8c45f52d4f9353ce944b287265097650
4 scores.source_id "scores_v1" (explicit; no Arrow hash computed)
5 alice record_id row_0 (its position: scores names no key columns)
5 bob record_id row_1

System tag column names on scores (sites 3 + 5):

_tag_source_id::a19744e791fb1f4c5431daad4380327d8c45f52d4f9353ce944b287265097650
_tag_record_id::a19744e791fb1f4c5431daad4380327d8c45f52d4f9353ce944b287265097650

Post-join system tag column names (site 6):

_tag_source_id::a19744e791fb1f4c5431daad4380327d8c45f52d4f9353ce944b287265097650::a19744e791fb1f4c5431daad4380327d8c45f52d4f9353ce944b287265097650:0
_tag_record_id::a19744e791fb1f4c5431daad4380327d8c45f52d4f9353ce944b287265097650::a19744e791fb1f4c5431daad4380327d8c45f52d4f9353ce944b287265097650:0
_tag_source_id::eb2bdc439e8a48f8246b2ea8d52227418571f43c998be577b5a1c0794ca4d749::eb2bdc439e8a48f8246b2ea8d52227418571f43c998be577b5a1c0794ca4d749:1
_tag_record_id::eb2bdc439e8a48f8246b2ea8d52227418571f43c998be577b5a1c0794ca4d749::eb2bdc439e8a48f8246b2ea8d52227418571f43c998be577b5a1c0794ca4d749:1

The suffix ::a19744e...:0 on scores columns encodes the canonical join position (0); the suffix ::eb2bdc4...:1 on attendance columns encodes position 1. The pipeline hash in each suffix is the stream's pipeline_hash() (sites 1b and 6).

Site Row Value (hex)
7 alice base_entry_id 6172726f775f76313a0000029a2d37584820ff55fe7d7094fd4969a5700bfac520ef992bec3e14a6ddecba84
7 bob base_entry_id 6172726f775f76313a000002610e9096dcfead882f81c5017fa3c3db2d98c9b631766d3fae6d275014527915
8 alice pipeline_entry_id (idx=0) 6172726f775f76313a00000262d1dd80d0993a00228fdc099ac6b2100cc9b758be2bfa3af26e28ff9cfd0c75
8 bob pipeline_entry_id (idx=0) 6172726f775f76313a0000021991499aae99e538c06f12bfe1335a72017ddce904291eb35463437a51122247

The hex prefix 6172726f775f7631 decodes to the ASCII string arrow_v1 — the method portion of the b"{method}:{digest}" format described in §3.

§4 — Side-Effect Record ID & Invocation Hash

Section titled “§4 — Side-Effect Record ID & Invocation Hash”
Site Row Value
10 alice invocation_hash semantic_v1:fd7ca26128a414fd0c569adfa78e3bea72e72cd47d173ad3380636209a175b2a::arrow_v1:00000262d1dd80d0993a00228fdc099ac6b2100cc9b758be2bfa3af26e28ff9cfd0c75
10 bob invocation_hash semantic_v1:fd7ca26128a414fd0c569adfa78e3bea72e72cd47d173ad3380636209a175b2a::arrow_v1:0000021991499aae99e538c06f12bfe1335a72017ddce904291eb35463437a51122247

Both rows share the same semantic_v1:fd7ca261... component — this is the SideEffectPodStream.pipeline_hash(), which depends only on the pod and its upstream topology, not on row content. The arrow_v1:... component is the per-row record_id from site 9, which differs per row.

Site Item Value
11 data_function.uri ('grade', '6dfe7b63d41fec2df9e9fe6e7581428678c88866f4927b45dd48a44df73dfe3d', 'v0', 'python.function.v0')
11 output_schema_hash (uri[1]) 6dfe7b63d41fec2df9e9fe6e7581428678c88866f4927b45dd48a44df73dfe3d (bare digest: it is part of a name)
Site Item Value
12 run_id (non-deterministic — changes each run)
13 snapshot_hash ec336c684fa5c875
14 datagram_uuid (time-ordered UUID v7 — changes each run)

snapshot_hash = "ec336c684fa5c875" is deterministic: the pipeline has one leaf node whose content_hash().to_string() is "semantic_v1:195343e1e04294d53c252fba6b24a048688a0bfaf8236df866a9e881d6ab2d8e". hashlib.sha256(leaf_hash.encode()).hexdigest()[:16] = "ec336c684fa5c875". The hash enters with its label.


Every class that inherits from TraceableBase carries both a content_hash() and a pipeline_hash(). Both use SemanticAwarePythonHasher, which recursively expands the object's identity structure into a JSON-serialisable representation and takes its SHA-256 digest.

  1. Call identity_structure() (for content hash) or pipeline_identity_structure() (for pipeline hash) on the object.
  2. Recursively expand the structure: ContentIdentifiable objects are replaced by their hash via a resolver callback (see below); containers (list, dict, tuple, set) are serialised to nested JSON. A nested hash (of a type, a schema, a pod, a stream, a value with its own hasher) enters as {"__type__": "hash", "label": "<label>", "digest": "<hex digest>"}: with its label (see The hash label rule).
  3. JSON-serialise the expanded structure and compute hashlib.sha256(json_bytes).digest() → ContentHash.

How containers expand (canonical JSON: sorted object keys, no whitespace, UTF-8):

Container Expanded form What counts
list JSON array order
tuple {"__type__": "tuple", "items": [...]} order
set {"__type__": "set", "items": [...]}, items sorted by their canonical JSON text membership only; never iteration order or PYTHONHASHSEED (ITL-942: before v0.3, items were sorted by str(), so {1, "1"} hashed differently per process)
frozenset {"__type__": "frozenset", "items": [...]}, items sorted as for set membership, and that it is a frozenset: a frozenset and a set of the same items differ (ITL-1154)
dict with only str keys, none of them "__type__" JSON object keys and values, not insertion order
any other dict {"__type__": "dict", "items": [[key, value], ...]}, pairs sorted by canonical JSON key types: {1: "x"} and {"1": "x"} differ (ITL-942: before v0.3, keys went through str() and could collapse)
namedtuple {"__type__": "namedtuple", "name": ..., "fields": [[name, value], ...]} class name, field names and values, in declared order (ITL-942: before v0.3 fields were sorted by name, so Point(x=1, y=2) and a Point declared (y, x) holding y=2, x=1 shared a hash)
a type handler's Encoded(tag, fields) {"__type__": tag, **fields} the handler's tag and fields

Only the hasher writes "__type__". A JSON object with a "__type__" key comes either from the hasher's own encodings (tags tuple, set, frozenset, dict, namedtuple, hash) or from a type handler that returned an Encoded value (generic_alias, union, bytes, uuid). A mapping that has a "__type__" key is hashed in the pair-list form, so a user value such as {"__type__": "tuple", "items": [1, 2]} never shares the hash of (1, 2) (ITL-942). For authors of a PythonTypeHandlerProtocol: return orcapod.hashing.semantic_hashing.Encoded(tag, fields) to produce a tagged object; a plain dict with a "__type__" key is hashed as an ordinary mapping. Encoded rejects the hasher's own tags and a "__type__" field, and is trusted only as a handler's return value: an Encoded inside a user value has no handler and is refused by the strict hasher.

Resolver pattern: The resolver callback is threaded through the entire recursive hash computation to ensure one consistent context per call:

  • content_resolver (used by content_hash()): routes every ContentIdentifiable leaf through leaf.content_hash(hasher).
  • pipeline_resolver (used by pipeline_hash()): routes PipelineElementProtocol objects through leaf.pipeline_hash(hasher); routes all other ContentIdentifiable objects (e.g. raw data values) through leaf.content_hash(hasher).

Both hashes are cached by hasher_id on each object, so repeated calls are free.


1a. content_hash() — data-inclusive identity

Section titled “1a. content_hash() — data-inclusive identity”

The table below shows what each class returns from identity_structure().

Class identity_structure() return value Notes
Datagram self._ensure_data_table() — the raw Arrow table Dispatched to ArrowTableHandler → StarfixArrowHasher
Tag User tag columns only (raw Arrow table) System tag columns (_tag::*) excluded — they are provenance metadata, not tag content
Data Data columns only (raw Arrow table) Source info columns (_source_*) excluded — they are provenance metadata, not data content
EmptyData Raises EmptyDataAccessError content_hash() is overridden to return a stored cached_content_hash directly
DataFunctionBase self.uri — (canonical_function_name, output_schema_hash, major_version, data_function_type_id) output_schema_hash is site 11; see §5
FunctionPod self.data_function (bare object) when no ctx arg; (self.data_function, self._ctx_arg_name) when ctx arg present ctx_arg_name is included when present so a ctx-aware pod has a distinct content_hash from a plain pod using the same data function
ArrowTableStream (producer, argument_symmetry(upstreams)) Falls back to (class_name, as_table(all_info=True), tag_columns) when no producer or producer is not a PipelineElementProtocol
RootSource (ArrowTableSource and every other root source without its own rule) (class_name, (tag_schema, data_schema), source_id, key_columns) source_id is the explicit user-supplied value; when not provided it defaults to <source kind>:<algorithm label>:<hex> of the Arrow hash of the raw table (site 4). key_columns is the resolved tuple of record-id key columns in key order (record_id_columns, or a database or SpiralDB table's primary key), () for the position: two sources that differ only in how they name their rows are two sources (developer decision, November 16, 2026, ITL-1154). Data-inclusive base case of the Merkle chain.
DerivedSource Origin node's content_hash Inherits its generating node's identity
DerivedSource with tags_as_data (class_name, output_schema, source_id, key_columns, ("tags_as_data", sorted items)); the default source_id is tags_as_data:<label>:<hex> of the hash of (("DerivedSource", sorted tags_as_data items), (origin,)), the origin entering through its content_hash; record_id_columns, when given, is appended to the parameters as ("record_id_columns", (names...)) The rule of every root source plus the mapping, which enters the content_hash directly (two mappings can give one schema) and so also with an explicit source_id; the origin enters through the default source_id only, which is the content hash an operator node doing the derivation would have
ListSource (class_name, name, elements, tag function hash, key_columns) The elements are the source's identity; key_columns as for every root source
PollingSource (impl.identity(), tag_columns, key_columns) The implementer's identity; key_columns (always given for a polling source) as for every root source
SpiralDBTableSource (class_name, output_schema, source_id, key_columns), then the sorted tags_as_data items when given A restriction (where, limit, asof) chooses rows of the same table and adds nothing
CachedSource the wrapped source's identity
Operators (unary) (operator_class_name, upstream_stream) Stream reference resolved via content_resolver
Operators (binary/N-ary) (operator_class_name, argument_symmetry(streams)) frozenset for commutative (Join, MergeJoin); tuple for ordered (SemiJoin)
SelectTagColumns, SelectDataColumns, DropTagColumns, DropDataColumns, SelectColumns, DropColumns (operator_class_name, sorted unique column names as a list, strict) The operators act on a set of names: a list, a tuple, a set, a single string, a repeated or a reordered name list give one hash (ITL-942)
MapTags, MapData, MapColumns (operator_class_name, name_map, drop_unmapped, strict) An operator's identity is its full configuration, strict included, as for the column selection operators (ITL-1160)

Inputs of a commutative operator are a set (kept by decision, ITL-942). A pipeline hash covers the structure of the pipeline, not only its schemas. MergeJoin()(m1, m1), a stream joined with itself, is a different pipeline from MergeJoin()(m1, m2) and has a different pipeline hash even when both inputs share one schema. Inputs are an unordered set compared by identity, so input order never matters, and two inputs with the same source identity (same class, schemas and source_id) count as one input: MergeJoin()(m1, m1b) hashes like MergeJoin()(m1, m1) when m1b has the source_id of m1.

Inputs with equal pipeline hashes tie, and argument order never breaks the tie (developer decision, November 16, 2026, ITL-1154). The system-tag column names follow the stable pipeline-hash sort, and each row's system-tag values at the tied positions are sorted by (source id, record id) (arrow_utils.sort_system_tag_values), so MergeJoin()(a, b) and MergeJoin()(b, a) over two sources of one schema give the same rows, the same column names and the same hashes.

Known exclusions from identity_structure():

Class Excluded Reason
Tag System tag columns (_tag::*) Provenance metadata, not content
Data Source info columns (_source_*) Provenance metadata, not content
EmptyData All data No payload; content_hash() overridden to return stored hash

1b. pipeline_hash() — schema and topology only

Section titled “1b. pipeline_hash() — schema and topology only”

The pipeline hash uses the same SemanticAwarePythonHasher with the pipeline_resolver. The key difference is what each class returns from pipeline_identity_structure().

Class pipeline_identity_structure() return value Notes
RootSource (tag_schema, data_schema) Base case — schemas only, no data content
DerivedSource (tag_schema, data_schema) Acts as a new root in the pipeline Merkle chain
DataFunctionBase self.uri (same as identity_structure()) Function identity is already schema-only
FunctionPod self.data_function only (excludes ctx_arg_name) A ctx-aware pod and a plain pod over the same data function share the same pipeline_hash value, but have different uri values (ctx_arg_name is set → "side_effect_function" is prepended to the URI) and therefore write to different DB table paths
ArrowTableStream (producer, argument_symmetry(upstreams pipeline hashes)) pipeline_resolver routes upstreams through pipeline_hash(). Falls back to (tag_schema, data_schema) when producer is absent or not a PipelineElementProtocol.
Operators (operator_class_name, argument_symmetry(upstream pipeline hashes)) Same structure as content identity but using pipeline hashes of upstreams
SideEffectPodStream (pod, argument_symmetry(upstreams)) Same as identity_structure()

Note: FunctionPod.ctx_arg_name IS included in content_hash() (via identity_structure()). It is excluded only from pipeline_identity_structure(). A ctx-aware pod receives a system-injected InvocationContext argument — a deliberate deviation from the pure data-in / data-out contract — so it is kept distinct from a plain pod at every level: different content_hash, different uri (prefix "side_effect_function"), and different DB table path. The shared pipeline_hash value has no practical effect on scoping once the uri differs.


PolarsFilter.identity_structure() is ("PolarsFilter", predicates, constraints), with predicates always a tuple. Keyword constraints hash as plain values. A pl.Expr predicate hashes through PolarsExprHandler to the leaf polars_expr_v1:<hex>.

Polars' own serialization of an expression (Expr.meta.serialize) changed layout between 1.20, 1.30 and 1.36 and is documented as unstable across versions, and str(expr) is lossy (pl.lit(0.1, dtype=pl.Float32) and pl.lit(0.1, dtype=pl.Float64) print alike), so neither is hashed. Orcapod parses the JSON serialization with a strict parser into a tree in its own grammar and hashes that:

ContentHash("polars_expr_v1", sha256(canonical JSON of the tree))
canonical JSON = keys sorted, separators (",", ":"), UTF-8
node := {"col": <name>} | {"lit": <literal>}
| {"binary": [<op>, <node>, <node>]}
| {"fn": [<fname>, <options>, [<node>, ...]]}
| {"cast": [<dtype>, "strict" | "non_strict", <node>]}
op := eq ne eq_missing ne_missing lt le gt ge and or xor add sub mul truediv
floordiv mod
fname := not is_null is_not_null is_nan is_not_nan negate all_horizontal
any_horizontal str.starts_with str.ends_with (options {})
| is_in {"nulls_equal": bool} | is_between {"closed": both|left|right|none}
| str.contains {"literal": bool, "strict": bool}
literal := {"dyn_int": "<decimal>"} | {"dyn_float": "<float.hex()>"}
| {"null": null} | {"boolean": bool} | {"string": str}
| {"int8" ... "uint64": "<decimal>"} | {"float32" | "float64": "<float.hex()>"}
| {"decimal": ["<unscaled>", precision, scale]} | {"binary": "<hex>"}
| {"date": days} | {"time": ns}
| {"datetime": [value, unit, time zone or null]} | {"duration": [value, unit]}
| {"list": [<dtype>, [<value or null>, ...]]} (no lists of decimals or bytes)
dtype := the dtype names of the Polars value descriptor ("int64", "string",
{"datetime": [unit, tz]}, {"list": <dtype>}, ...)

The module docstring of orcapod.hashing.semantic_hashing.polars_expr is the full definition. Consequences:

  • pl.col("x") > 0, pl.col("x").gt(0), pl.col("x") > pl.lit(0) and 0 < pl.col("x") hash equal. > 0 (a Python int, typed by Polars against the column), > 0.0 and > pl.lit(0, dtype=pl.Int8) hash differently. The expression is hashed as written: a & b and b & a differ, and so does the order of the predicates of one filter.
  • Unsupported expressions raise ValueError naming the expression: Python functions (map_elements, map_batches, whose serialized form differs between processes), selectors over several columns (pl.col("x", "y"), pl.all(), regex columns, pl.exclude, cs.*, dtype columns), aggregations and windows, alias, when/then/otherwise, NaN and infinite literals (the JSON form cannot tell them apart), pl.Series literals, and any node or layout the parser does not know. A Polars release that changes its serialization therefore fails loudly rather than moving a hash; the parser is then taught the new layout and maps it onto the same tree.
  • Supporting a new construct adds to the grammar without changing any existing hash. Only a change to the encoding of a supported construct needs a new label (polars_expr_v2).

The canonical trees of a corpus of expressions are pinned in tests/test_hashing/test_polars_expr_hashing.py, and their hashes in the golden suite (Golden Hashes).

1d. Function values and parameter defaults

Section titled “1d. Function values and parameter defaults”

A function used as a value (inside an identity structure, or passed to semantic_hasher.hash_object) is hashed from one structured description of its signature:

{"module": <module>, "name": <name>,
"parameters": [{"name": ..., "kind": ..., "annotation": ..., "default": ...}, ...],
"returns": <return annotation>}
Member of a parameter entry Value Absent when
name the parameter name never
kind POSITIONAL_ONLY, POSITIONAL_OR_KEYWORD, VAR_POSITIONAL, KEYWORD_ONLY or VAR_KEYWORD never
annotation the canonical text of the annotation (union members sorted, logical types by their identifier) the parameter has no annotation
default the semantic hasher's hash of the default value, as a nested hash the parameter has no default

Parameters are listed in declared order. The function body is not part of this hash.

A default value is hashed like every other hashed value (ITL-953): by the semantic hasher, never by its repr(). Equal defaults give equal hashes in every process and under every PYTHONHASHSEED; a default of None differs from having no default; and because a default enters as a nested hash, no literal default can spell out another default's entry. Before v0.3 the default entered as repr(value), so a frozenset default hashed differently per process and an object() default embedded a memory address. Before v0.3 the parameter kind was also not hashed: f(a, *, b), f(a, /, b) and f(a, b) shared a hash.

A default the hasher cannot hash is an error. There is no fallback to repr() or to the type name, either of which would let two different defaults share a hash. Hashing the function raises TypeError naming the parameter and the function. Since a data function computes its signature hash when it is built, a function pod whose function has such a default fails at construction (@function_pod, FunctionPod.from_fn, PythonDataFunction).

Default value Hashed as
None, bool, int, float, str the JSON primitive (a subclass such as an IntEnum member or numpy.float64 hashes as the number it is)
list, tuple, set, frozenset, dict, namedtuple the container forms of the algorithm above
bytes, uuid.UUID, a function, a type, numpy.ndarray, pandas objects, File, Directory, any object with identity_structure() its registered handler
a plain enum.Enum member, a dataclass or pydantic instance, pathlib.Path, datetime, date, Decimal, complex, a numpy scalar other than numpy.float64, any other object not hashable: TypeError

Remedies: use a default the hasher can hash (for example a str or None, building the object inside the function); remove the default and supply the value as an input column; or register a PythonTypeHandlerProtocol for the type in the data context's handler registry. Raising is the v0.3 behaviour (developer decision, November 11, 2026); hashing a pod parameter's default as the column value it would be, which would cover Path, datetime, date and numpy scalars, is the follow-up ITL-971.

Where the hash is used. As the hash of a function that appears as a value, and as the function_signature_hash stored with every result row (the __pf_var_function_signature_hash column), which is this same hash of the pod's function. A function pod's identity does not use it: the pod's uri, its pipeline_hash(), its content_hash() and its table paths depend on the function's name, version and output schema (see Data Function URI Hash), so changing a default moves no path.

This hash is provisional in v0.3 (see section 8).

A polars.DataFrame or polars.Series is a value-hashed logical type (PolarsDataFrameHandler, PolarsSeriesHandler; the algorithm is defined in orcapod.logical_types.polars_type). Inside a table its cell token is, like every value-hashed type's (see the hash label rule), b"orcapod.polars.dataframe;<hex of the extension metadata>::polars_dataframe_v1:" + digest: the logical type's identifier, then the hash with its label. The extension metadata is {"ignore":[],"version":1}: the version and the list of properties the hash ignores, empty by default.

Storage and hashing are separate. The value is stored with Polars' own write_ipc; the stored bytes are never hashed (they depend on the Polars version that wrote them). The hash is computed from the value's content:

sha256(descriptor + b"\x00" + starfix.ArrowDigester.hash_table(
values, ordered_columns=True, ordered_struct_fields=True))

under the label polars_dataframe_v1 for a frame and polars_series_v1 for a Series (a Series is hashed as its one-column frame, with its own descriptor).

values is the frame exported to Arrow with compat_level=oldest, with top-level Null columns left out, Array cast to large_list, nested Null cast to an all-null bool and dictionaries decoded, for hashing only. starfix runs in its ordered mode, so column order and struct field order at every depth are part of the digest, and it tells a null struct from a struct whose fields are all null. The descriptor carries what the Arrow values cannot (Enum categories, a Categorical's Categories, Array size, Null columns, the row count, a Series' name), and states names, order and dtypes again in Orcapod's own terms. descriptor is canonical JSON (keys sorted, compact separators, UTF-8) in a grammar Orcapod owns, never Polars' repr:

frame: {"kind": "orcapod.polars.dataframe", "height": <rows>,
"columns": [[<name>, <dtype>], ...]} in column order
series: {"kind": "orcapod.polars.series", "name": <name>, "height": <rows>,
"dtype": <dtype>}
<dtype> := "int8" | "int16" | "int32" | "int64" | "uint8" | "uint16" | "uint32" | "uint64"
| "float16" | "float32" | "float64" | "boolean" | "string" | "binary"
| "date" | "time" | "null"
| {"categorical": [<name>, <namespace>, <physical>]} its Categories
| {"enum": [<category>, ...]} every declared category, in order
| {"decimal": [<precision>, <scale>]}
| {"datetime": [<unit>, <time zone or null>]}
| {"duration": <unit>}
| {"list": <dtype>}
| {"array": [<dtype>, <size>]} a multi-dimensional array nests
| {"struct": [[<field>, <dtype>], ...]} in field order

The rule: everything a pod can observe through the public Polars API as the frame's logical content is identity. So two frames hash equal exactly when they have the same row count, the same column names in the same order, the same dtype for each column (struct fields and their order at every depth, Enum categories and their order, a Categorical's Categories name, namespace and physical width, Decimal precision and scale, Datetime unit and time zone, Duration unit and Array shape included) and the same values (a null struct differs from a struct whose fields are all null; a null differs from NaN, and 0.0 from -0.0). A pod that returns df.columns[0] therefore never receives another frame's cached result. An Enum's full enumeration of choices is part of its dtype: all its declared categories in declared order, including categories no row holds, so Enum(["a", "b"]), Enum(["b", "a"]) and Enum(["a", "b", "z"]) hash differently over the same values.

Physical properties are not identity: chunking (n_chunks()) and rechunking, slice offsets, the bytes under null slots, sortedness flags (Series.flags, which storage does not keep: a frame read back has them cleared), physical Categorical codes (to_physical()) and the contents of the process-wide category registry, which depend on what the process saw before, the Arrow layout Polars exports (string views, large or small offsets) and memory size. Supporting a new dtype adds to the grammar without changing any existing hash; only a change that would give an existing value a different digest needs new labels (polars_dataframe_v2, polars_series_v2). A dtype outside the grammar raises ValueError naming the column.

One kind of value cannot be hashed: starfix addresses a column or struct field by the path of its names joined with /, so a value in which two names form the same path (a column "a/b" beside a struct column a with a field b, or the same between a struct's fields) raises ValueError naming the path. A name containing / is fine as long as no other name forms the same path. The Arrow table hash (§9) has the same limit.

The hash is strict by default. An ignore list, part of the logical type's identifier, can leave out column_order and struct_field_order for a frame, and series_name and struct_field_order for a Series. A data context declares its list once, under hash_ignore in contexts/logical_types.json ("orcapod.polars.dataframe": [] by default); two lists are two logical types, so their columns never share a hash, and a stored column whose list differs from the data context's is refused on read. Ignoring column order lists the descriptor's columns sorted by name and runs starfix with ordered_columns=False; ignoring struct field order does the same for every struct's fields with ordered_struct_fields=False; ignoring a Series' name writes "" for "name" in the descriptor and hashes its values as a column named "".

A pandas.DataFrame is hashed differently: with the pandas_dataframe_v1 hash (§10), which counts column order. A Polars frame and the equal pandas frame never share a hash.


A dict and a set have no order, so equal values must hash equally whatever order they were built in. Arrow stores both as lists: dict[K, V] as large_list<struct<key, value>> and a set[T] of a primitive as the native set container set[<element Arrow type>] over large_list<T> (ITL-611; see Logical Type Identifiers). Orcapod puts them in canonical order at the point where a Python value becomes Arrow, so the stored list, and therefore the row hash, does not depend on insertion order (dicts) or on PYTHONHASHSEED (sets of str or bytes).

from orcapod.sources import DictSource
def dict_row_hash(value):
source = DictSource(
data=[{"k": "r", "m": value}],
tag_columns=["k"], tag_only_columns=True,
data_schema={"k": str, "m": dict[str, int]},
)
_, data = next(iter(source.iter_data()))
return data.content_hash()
print(dict_row_hash({"a": 1, "b": 2}) == dict_row_hash({"b": 2, "a": 1}))
# True

Where the order is applied. In the type converter (get_python_to_arrow_converter), which every Python-to-Arrow path uses: sources built from Python values, datagrams, function pod outputs (sync and async), and dicts nested in lists, dicts or dataclass and pydantic fields. A datagram built from Python values also holds its dict values (including dicts inside lists, dicts and Optional) as key-ordered copies, including values set later with update or with_columns, so a newly computed pod output and the same output read back from the cache iterate in the same order (decided November 12, 2026). The caller's dict is never modified; dicts inside dataclass or pydantic objects keep their own order in memory. Their stored dict values and hashes are canonical, but a fresh user-class instance can iterate a nested dict in insertion order while a read-back instance iterates it in key order. An order-sensitive pod can therefore observe different orders for equal hashed inputs; reconstructing user-class instances is outside this fix.

The order is defined on the converted (storage) key or element, in semantic_types/canonical_order.py:

Key or element type Order
None (for K | None) first
bool False before True
int, float, Decimal numeric
str Unicode code point, never locale
bytes unsigned bytewise
date, datetime, time, timedelta natural order; datetimes by instant (they are timezone aware after conversion)

A NaN key or element (a Python float, a NumPy floating scalar or a Decimal) has no order and raises CanonicalOrderError (a ValueError) naming the column, as do keys or elements that cannot be compared and two keys or elements that convert to the same stored value (decided November 12, 2026). A dict or set whose nested unordered containers are all already in this order is stored byte-for-byte as before this rule existed, so its hash did not change.

Signed zero. A float is hashed by its stored bytes, and 0.0 and -0.0 have different bytes, so they hash differently wherever they appear: in a plain float column and as a dict key or set element alike, although Python compares them equal ({0.0: 1} == {-0.0: 1}). Orcapod does not rewrite -0.0 to 0.0: pod code can tell them apart (math.copysign, repr, division), so the finer identity can cost a recompute but never serves a wrong result. Within one dict or set the two compare equal, so they cannot both be present.

Arrow tables are hashed exactly as stored. StarfixArrowHasher never sorts list contents. A list<struct<key, value>> column that arrives already in Arrow form (an ArrowTableSource, a Delta table, rows read back from a database) is read by Python as a dict, but at the Arrow level it is an ordered list and its stored order is part of its hash. A list[T] value is never sorted: its order is part of its identity. (Tables written by 0.2.0, which stored dicts in insertion order, are not read by v0.3 at all.)

Two orderings, two layers. This section is about the order a dict or set is stored in, which is also the order it is read back in. The semantic hasher (section 1a) separately orders the hash input it builds for a Python dict or set object, by each item's canonical JSON text (ITL-942); that order is never stored and never seen. The two rules are different on purpose (JSON text puts 10 before 2; storage order is numeric) and neither is defined in terms of the other: changing the hasher's order would move semantic hashes of affected inputs, and changing storage order would move row hashes and what a pod reads back. set[T] of a logical type (set[Path], set[UUID]) is ordered by ListLogicalType, not by this rule.

The set's identity is part of the row hash (ITL-611). A set of a primitive is stored under its native container type, and the Arrow hasher hashes an extension column's identifier (extension name and metadata) with its stored values. So {1, 2, 3} as set[int] and [1, 2, 3] as list[int] are two different rows, although both store the list [1, 2, 3]. The elements of a set of a primitive are hashed as stored, never replaced by their value hashes: only a container whose innermost element is a logical type is (a set[bytes] hashes like its stored large_list<large_binary> plus the identifier, although the semantic hasher has a handler for bytes). Before ITL-611 a set of a primitive was a plain large_list and hashed exactly like the list row; such rows were written only by development builds of v0.3.


When a source is built by SourceStreamBuilder, four hash-related operations happen in sequence before the stream is returned:

  1. Site 3 — Schema hash: computed from (tag_schema, data_schema); embedded in system tag column names.
  2. Site 4 — Default source_id: <source kind>:<algorithm label>:<hex> of the raw table hash if no explicit source_id is given, for an in-memory source (arrow_table, dict, list, data_frame); a file or database source's default names its location instead (csv:<path>, delta_table:<path>, and so on; see "Default source ids and row identifiers").
  3. Site 5 — Per-row record_id: the row identifier (row_<n> or the key's canonical JSON), stored per row as text.
  4. Two system tag columns, _tag_source_id::{schema_hash} and _tag_record_id::{schema_hash}, are appended to every row.

tag_only_columns (the tags kept plain; True for all of them) enters these sites only through the schema: a data-tag is also a data column, so it is in the data schema and the schema hash, while the default source_id (the table hash of an in-memory source, or the location of a file or database source) and the record ids are those of the same source with every tag plain. Every golden fixture that builds a source with tag columns passes tag_only_columns=True, so the golden values of data context v1 pin plain tags, unchanged now that tags are data-tags by default.

At Join time (site 6), all existing system tag columns are renamed by appending a topology suffix, encoding the full join lineage into each column name.


Field Value
Inputs (tag_schema, data_schema) — Python Schema objects mapping column names to Python types
Algorithm SemanticAwarePythonHasher.hash_object((tag_schema, data_schema)).to_hex(schema_n_char) where schema_n_char = OrcapodConfig.hashing.schema_n_char
Output format Truncated hex str (length = schema_n_char; None = full digest)
Uniqueness guarantee Unique per (tag_schema, data_schema) pair; two sources with identical schemas produce identically-named system tag columns, enabling consistent system tag lookup across sources of the same schema
Known exclusions No data content; no source identity; purely structural

Field Value
Inputs Full raw Arrow table (all columns, before any system tag injection)
Algorithm hash_source_id(<source kind>, StarfixArrowHasher.hash_table(table), hexdigits=path_n_char) — the table hash is starfix hash format 2: the 3-byte format number 00 00 02 followed by a SHA-256 digest (starfix 0.5, byte-identical to the Rust crate)
Output format str: <source kind>:<algorithm label>:<hex>, for example dict:arrow_v1:000002fe…. The source kind is the class's config type key (SOURCE_KIND: arrow_table, dict, list, data_frame); the label is the one the hash carries. The kind and the label are never shortened; hashing.path_n_char, when set, shortens the hex only. The parts hold no :, so the id never contains the :: that separates the parts of a source-info value
Uniqueness guarantee Unique per source kind and raw table content; changes if any cell value changes or the rows are reordered
Known exclusions Used only as a fallback when no explicit source_id is provided. A user-supplied source_id bypasses this computation entirely.

Site 5 — Per-row record_id (system tag value)

Section titled “Site 5 — Per-row record_id (system tag value)”
Field Value
Inputs The row's position, or the values of the key columns (record_id_columns, or a database table's primary key)
Algorithm row_<n>, or the canonical JSON object of the key columns in key order with no spaces (stream_builder.key_row_identifier)
Output format str, stored in a large_string Arrow column; the column keeps that type through tags, operators and stored tables. Not a hash; it holds no source id
Uniqueness guarantee Unique within the source: a key repeated in two rows, and a null key value, are refused when the source is built
Known exclusions A positional identifier changes when rows are inserted before the row or reordered; a key-based one follows the row

The record id names the row, and the source-info token names the row and its column. A key value may contain ::, so the record id may too; a source id and a column name cannot, so the token is split at its first :: and its last :: only, never at every ::.

subjects = DictSource(
data=[{"subject": "s1", "age": 3, "weight": 20.5}],
tag_columns=["subject"],
source_id="subjects",
record_id_columns="subject",
)
first_row = subjects.as_table(
columns={"system_tags": True, "source": True}
).to_pylist()[0]
print(
next(
value for name, value in first_row.items() if name.startswith("_tag_record_id")
)
)
# {"subject":"s1"}
print(first_row["_source_age"])
# subjects::{"subject":"s1"}::age

Field Value
Inputs stream.pipeline_hash() of each canonically-ordered input stream + its 0-based canonical position index idx
Algorithm Streams are first sorted by stream.pipeline_hash().to_hex() for determinism. For each input at canonical position idx, every existing system tag column name has {BLOCK_SEPARATOR}{stream.pipeline_hash().to_hex(system_tag_n_char)}:{idx} appended via arrow_utils.append_to_system_tags(). (system_tag_n_char from OrcapodConfig.hashing; default None = full digest.)
Output format Column name suffix; no separate value is stored
Uniqueness guarantee Each post-join system tag column name uniquely identifies (original schema, input topology, canonical join position); no collision even when joining streams with identical schemas
Known exclusions SemiJoin passes system tags through unchanged. Aggregating operators (Batch, GroupBy) use the fold described in site 6b instead.

Every input keeps its own system tag columns, so the count reflects the join depth: a stream split into two branches that are joined again carries each of its system tag columns twice, and k such stages stacked carry each one 2 ** k times (ITL-1114). The table of no rows that head() and the notebook preview of a stream that has not run are built from carries the system tag columns too, so those calls are as wide as the joins make them (ITL-1132).


Site 6b — Aggregating operator system tag fold

Section titled “Site 6b — Aggregating operator system tag fold”
Field Value
Inputs The ordered list of a group's member values for one system tag column, and the operation's label
Algorithm arrow_utils.fold_system_tag_values(values, label=...), one helper for both system tags: SHA-256 of the canonical JSON array of the members in order (json.dumps([None if v is None else str(v) for v in values], separators=(",", ":"), ensure_ascii=False), UTF-8), as 64 hex characters after the operation's label: batch_v1:<hex> for Batch (BATCH_FOLD_LABEL, in batch.py), group_v1:<hex> for GroupBy (GROUP_BY_FOLD_LABEL, in group_by.py). The label names the operation; each operation's procedure may change on its own (ITL-1154). The array marks where each member ends, so ["ab", "c"] and ["a", "bc"] fold differently, and a None member (null) differs from "" (ITL-942). The folded column name then gains {BLOCK_SEPARATOR}{stream.pipeline_hash().to_hex(system_tag_n_char)} via arrow_utils.append_to_system_tags().
Output format "<fold label>:<64 hex>" str for the source id and the record id alike: scalar, matching the input column's type
Uniqueness guarantee Deterministic per ordered member set, and stable across processes (both primitives are SHA-based). Batch and GroupBy both order their members deterministically before folding, so an unchanged member set always folds to the same digest
Known exclusions Member order is significant; a permuted member set folds differently. This is why GroupBy sorts members by their non-key tag values with record_id as a tiebreaker rather than relying on upstream emission order

Merged source-info tokens (data-tags, ITL-1017). A value merged from several equal elements has one source-info token, folded by the same helper under the merging operation's label: group_v1:<hex> for the key of a GroupBy, over the members' tokens in member order (the procedure group_v1 names for its system tags), and join_v1:<hex> for a key that is a data-tag on several inputs of a Join, over the members' tokens sorted (a missing token last), because a join meets its inputs in an order that is not canonical. The token holds no ::, so it never reads as a row's token. Source info enters no content or pipeline hash; an operator node's record hash, which includes the source-info columns (ITL-1019), takes a merged token as it takes any token.

Why fold rather than list-wrap. A many-to-one operator turns user tag and data columns into list[T], but system tag columns must stay scalar: _build_record_id_preimage (core/nodes/function_node.py) hashes those columns directly to derive record identity. Folding rather than dropping them preserves the Merkle link, so an upstream recompute that yields identical data still invalidates downstream records. Because the digest becomes a cache key, the fold must never use hash() or a set-based construction — a per-process value would look correct in a single-process test and miss the cache on every new driver run.

Key insight for entry IDs: Because compute_base_entry_id() (§3) calls tag.as_table(columns={"system_tags": True}), its preimage captures the full set of chained system tag column names. After a two-way join, the preimage contains four system tag columns — two per input — each with the join topology embedded in its name. The entry ID therefore implicitly commits to the complete join provenance graph without any join-awareness in the entry ID computation itself.


FunctionJobNode uses two related Arrow-based hashes as database primary keys. Both are computed by StarfixArrowHasher over a single-row preimage table.

The base entry ID (site 7) is stable across all recomputation attempts for the same logical input — it is used as an in-memory cache key and for Phase 1 DB lookups. The pipeline entry ID (site 8) adds a recomputation_index column and is the actual primary key stored in the pipeline DB.

The preimage for both sites is built by _build_record_id_preimage(tag, input_data):

preimage = tag.as_table(columns={"system_tags": True})
.append_column(INPUT_DATA_HASH_COL,
pa.array([input_data.content_hash().to_prefixed_digest()],
type=pa.large_binary()))

The input data hash enters the preimage with its label, b"<label>:<digest>", the form the stored __input_data_hash column keeps: a hash fed into another hash carries its label.

Because tag.as_table(columns={"system_tags": True}) includes all chained system tag columns (see §2, site 6), the preimage implicitly captures the full join provenance of the row.


Field Value
Inputs All system tag columns from tag.as_table(columns={"system_tags": True}) (including join-chained columns) + INPUT_DATA_HASH_COL (input_data.content_hash().to_prefixed_digest() as pa.large_binary(): the digest with its label)
Algorithm StarfixArrowHasher.hash_table(preimage).to_prefixed_digest() — single-row Arrow table hash
Output format bytes in b"{method}:{digest}" format
Uniqueness guarantee Unique per (node, tag lineage, input_data content) across all recomputation attempts; used as the in-memory cache key and Phase 1 DB filter
Known exclusions NODE_CONTENT_HASH_COL excluded — the node's content hash is fully determined by the DB table path (scoped by uri + pipeline_hash()). Recomputation index excluded by design — use site 8 for a versioned key.

Field Value
Inputs Same preimage as site 7 + _PIPELINE_RECOMPUTATION_INDEX_COL (value: recomputation_index, type pa.int32(); default 0)
Algorithm StarfixArrowHasher.hash_table(preimage).to_prefixed_digest()
Output format bytes in b"{method}:{digest}" format
Uniqueness guarantee Unique per (node, tag lineage, input_data content, recomputation attempt) — the primary key for all rows in the pipeline DB
Known exclusions At recomputation_index=0 this produces a hash that differs from the pre-ITL-508 implementation (the index column is now part of the preimage); existing pipeline DB records were intentionally invalidated when ITL-508 landed

4. Side-Effect Record ID & Invocation Hash

Section titled “4. Side-Effect Record ID & Invocation Hash”

Side-effect pods (and FunctionPod instances with a ctx_arg_name) use a parallel but distinct record key scheme. The row-level preimage structure mirrors §3; pod-version scoping is handled at the delivery log's table-path level via pipeline_hash() and uri, not at the preimage level — consistent with how FunctionJobNode scopes function cache entries in §3.

The invocation_hash string is composed from two ContentHash components and exposed to the pod function as an idempotency key via InvocationContext.invocation_hash.


Field Value
Inputs System tags + INPUT_DATA_HASH_COL (input_data.content_hash().to_prefixed_digest() as pa.large_binary(): the digest with its label) + _SIDE_EFFECT_RECOMPUTATION_INDEX_COL (fixed 0, pa.int32())
Algorithm StarfixArrowHasher.hash_table(preimage)
Output format ContentHash; .to_prefixed_digest() → bytes when stored in the delivery log
Uniqueness guarantee Unique per (tag lineage, input_data content) at recomputation_index=0. Pod-version scoping is provided by the delivery log's table path (derived from pipeline_hash() and uri), not by the preimage.
Known exclusions NODE_CONTENT_HASH_COL is not included — pod-version identity is captured at the table-path level, consistent with how sites 7–8 scope function cache entries. Recomputation index is always 0; side-effect delivery records are not versioned across recomputation attempts.

Field Value
Inputs pipeline_hash_ch — the pod's own pipeline_hash() as ContentHash + record_id_hash_ch — the ContentHash from site 9
Algorithm f"{serialize(pipeline_hash_ch)}::{serialize(record_id_hash_ch)}" where each component is serialised as f"{method}:{hex_or_base64_digest}" via InvocationHashConfig (default: hex, full digest)
Output format str of the form "{method}:{digest}::{method}:{digest}". This string keeps its labels in v0.3; the library feeds it into no hash computation
Uniqueness guarantee Unique per (pod topology, tag lineage, input_data content); exposed to pod functions as an idempotency key. Pod-version scoping is provided by the pipeline_hash()-derived table path, not by the preimage.
Known exclusions When track_completion=True (default): run_id is excluded — hash is run-independent for idempotency. When track_completion=False and pipeline_run_id is set: run_id is appended as a third :: component so that each run produces a distinct hash.

DataFunctionBase.uri is a tuple used as the canonical identity of a data function:

uri = (canonical_function_name, output_schema_hash, major_version, data_function_type_id)

The output_schema_hash component (site 11) is the only hash in the URI; the other components are plain strings. major_version comes from the version argument, which defaults to "v0.0" (major v0) for every entry point: @function_pod, FunctionPod.from_fn, side_effect_function_pod and PythonDataFunction (ITL-942: before v0.3, from_fn and side_effect_function_pod defaulted to "v1.0").


Field Value
Inputs output_data_schema — a Schema mapping output column names to Python types
Algorithm SemanticAwarePythonHasher.hash_object(output_data_schema).to_hex()
Output format str: the bare hex digest, 64 characters, no label (the hash is part of a name: the function uri, table paths and source-info tokens)
Uniqueness guarantee Unique per output schema definition; changing any output column name or type changes this hash and therefore the function's entire URI, content_hash, and pipeline_hash
Known exclusions Input schema not included (changing input schema alone does not change the URI); function code not included (tracked separately via major_version); data_function_type_id not included (plain string component)

PipelineJob.run() generates three identifiers at execution time. None of these are used in any data-level preimage; they serve logging, observability, and result inspection.


Field Value
Inputs None (random)
Algorithm uuid.uuid4().hex[:16]
Output format 16-char hex str
Uniqueness guarantee Non-deterministic; unique per execution with overwhelming probability
Known exclusions Does not reflect pipeline structure, data content, or any input. Two runs with identical pipelines and data produce different run_id values.

Field Value
Inputs Sorted content_hash().to_string() labelled hashes of all DAG leaf nodes (nodes with no downstream successors in the execution DAG)
Algorithm hashlib.sha256("\n".join(sorted_leaf_hashes).encode()).hexdigest()[:16]
Output format 16-char hex str; embedded in pipeline_uri as {pipeline_name}@{snapshot_hash}
Uniqueness guarantee Unique per (leaf node topology + data state) at run time; changes if any leaf node's schema, function code, or source data changes
Known exclusions Covers only leaf (sink) nodes — intermediate nodes not included. Truncated to 16 chars (collision-resistant in practice but not cryptographically guaranteed at this length).

Field Value
Inputs Current wall-clock time (monotonic within a process)
Algorithm uuid_utils.uuid7() normalised to stdlib uuid.UUID via uuid.UUID(bytes=uuid7().bytes)
Output format uuid.UUID
Uniqueness guarantee Unique per datagram instance; time-ordered (monotonically increasing within a process)
Known exclusions Not a content hash. Two datagrams with identical content have different UUIDs. Not used in any hash preimage. Serves as an object identity token, not a content fingerprint.

Provisional. The value hashes of the library value types, numpy.ndarray (this section), torch.Tensor (below), polars DataFrame and Series (1e) and pandas DataFrame and Series, and their logical type identifiers, are provisional until the library value type audit (after 0.4.0rc1) and may change then. Using one issues ProvisionalHashWarning once per type per process (see Provisional value hashes). The types are declared in the provisional list of orcapod/contexts/logical_types.json, by extension name; that list enters no hash, identifier or stored path.

Value hash labels. A type whose values are hashed by a dedicated handler labels its hash <type>_v<N>, for example ndarray_v1 or torch_tensor_v1. The label is the ContentHash.method; the type's implementer chooses it. The bytes produced under an existing label never change. A new definition gets a new label, so a hash always says which definition produced it. The label is not part of any hash input (see The hash label rule).

A numpy.ndarray is stored as a .npy file and hashed by value (ITL-981). The hash is defined on the array, never on the stored bytes. The semantic hasher's NumpyArrayHandler returns ContentHash(method="ndarray_v1", digest=D). What is stored and what reads back is described in NumPy Arrays.

D is computed in two steps. Notation: u8 is one byte, u64le and i64le are 8-byte little-endian unsigned and signed integers, and lp(x) = u64le(len(utf8(x))) || utf8(x).

Step 1, the element digest E. Take the array in canonical form: C order, little-endian, structured fields packed in declared order. With dtype tag T (below), shape (d0, ..., dn-1) and the canonical element bytes B:

E = SHA-256( lp(T) || u64le(n) || i64le(d0) || ... || i64le(dn-1) || u64le(len(B)) || B )

E covers the dtype, the shape and every element bit, and nothing else. It is shared with other array types: a torch.Tensor computes the same E, see Torch tensor value hash below.

Step 2, the value hash D.

D = SHA-256( lp("orcapod.numpy.ndarray") || E || u8(memory_order)
|| lp(byte_order) || lp(struct_layout) )
Field Value Canonical value (written when the property is ignored)
leading text the type's extension name, orcapod.numpy.ndarray
memory_order 1 when the array is Fortran-contiguous and not C-contiguous, else 0 0
byte_order one character per scalar field of the dtype, depth first in declared field order, a subarray field counting once: < little-endian, > big-endian, | for a type without byte order (bool, 8-bit integers, S, V). NumPy's native marker = is resolved to the machine's order. float32 gives < the same text with every > replaced by <
struct_layout JSON without whitespace of L(dtype): null for an unstructured dtype, otherwise the list [itemsize, [offset_1, L(field_1)], [offset_2, L(field_2)], ...] in declared field order; a subarray field uses its base dtype. float32 gives null L of the packed dtype

Every field is always present and length-prefixed or of fixed width, so two different inputs never share a preimage.

The ignore list. The three properties memory_order, byte_order and struct_layout count by default. A data context can list any of them as ignored for the type. An ignored property's field is still written, with its canonical value, so arrays that differ only in ignored properties have equal D, and with all three ignored D equals the default D of the canonical array. The list itself is not in the preimage. It is in the type's extension metadata, {"ignore":[...],"version":1} (names sorted, keys sorted, no whitespace), and so in the logical type identifier: two lists are two logical types. A bare array is hashed with the data context's list; a cell is hashed with the list of its own column.

Cells. Inside a table hash, an array cell becomes the token b"orcapod.numpy.ndarray;7b2269676e6f7265223a5b5d2c2276657273696f6e223a317d::" + D: the logical type's identifier (the extension name, ;, the extension metadata {"ignore":[],"version":1} in hex), then the digest, without the label. The cell is decoded to an array first; the stored .npy bytes are never hashed.

Variation Same D by default?
Dtype (including width, units, string width), shape, any element bit, structured field names or field order no
C against Fortran memory order no (memory_order)
Little- against big-endian byte order no (byte_order)
Structured field offsets or item size (aligned against packed) no (struct_layout)
Step slices, reversed or broadcast views against their C-contiguous copy yes
np.matrix, np.recarray, np.memmap against the plain array yes
The writeable flag, dtype metadata, the align flag, padding bytes yes

The last three rows are what the .npy storage does not keep: a value read back from a store must hash like the value that was written.

The tag names the element type without a byte order. For a structured dtype, the tag is a JSON list with one [name, tag] entry per field, or [name, tag, [dims]] for a subarray field, in declared field order. It is written with no whitespace and with non-ASCII characters escaped (json.dumps(..., separators=(",", ":"), ensure_ascii=True)).

NumPy Tag Canonical element bytes
bool bool one byte, 0 or 1
int8 ... int64, uint8 ... uint64 int8 ... uint64 two's complement or unsigned, little-endian
float16, float32, float64 same names IEEE 754, little-endian
complex64, complex128 same names real part, then imaginary part
datetime64[ns], datetime64[10ms], ... datetime64[ns], datetime64[10ms], ... int64, little-endian (NaT is the minimum)
timedelta64[s], ... timedelta64[s], ... int64, little-endian
U5 ucs4[5] five UCS-4 code units, each uint32 little-endian, NUL-padded
S3 bytes[3] three raw bytes, NUL-padded
V4 void[4] four raw bytes
[("b", "f8"), ("a", "i4", (2,))] [["b","float64"],["a","int32",[2]]] fields packed in tag order

The implementation below uses only the standard library, and the vectors are pinned:

import hashlib
import struct
def lp(text: str) -> bytes:
data = text.encode("utf-8")
return struct.pack("<Q", len(data)) + data
def element_digest(tag: str, shape: tuple[int, ...], data: bytes) -> bytes:
p = lp(tag) + struct.pack("<Q", len(shape))
p += b"".join(struct.pack("<q", d) for d in shape)
p += struct.pack("<Q", len(data)) + data
return hashlib.sha256(p).digest()
def ndarray_hash(
tag: str,
shape: tuple[int, ...],
data: bytes,
memory_order: int = 0,
byte_order: str = "<",
struct_layout: str = "null",
) -> str:
p = lp("orcapod.numpy.ndarray") + element_digest(tag, shape, data)
p += bytes([memory_order]) + lp(byte_order) + lp(struct_layout)
return hashlib.sha256(p).hexdigest()
six = struct.pack("<6f", 0, 1, 2, 3, 4, 5)
print(element_digest("float32", (2, 3), six).hex())
# 318f6e503e8064e89ce6039453fce709f8e01f60c501af2b63bbbe9cf8fbf90e
print(ndarray_hash("float32", (2, 3), six))
# bef92ae814ed78857164d1aea02e5590ff1ab7f00179f4c75d17aec7cae27317
print(ndarray_hash("float32", (2, 3), six, memory_order=1))
# bb8d317f7bd44a4832eeb7468867b2137621fb539b084de988426dd5928af130
print(ndarray_hash("float32", (2, 3), six, byte_order=">"))
# 1d5b6c2eb0ceaafd3758356256d651c5edf9ca0efa4deecf12c76c4e15bdb637
print(ndarray_hash("float64", (), struct.pack("<d", 2.5)))
# 642ad052de6aa53ffd059e5cd9a2eeeb5519caa3d3eb12efcf9e0c3fdbeab17b
print(
ndarray_hash(
'[["a","int8"],["b","float64"]]',
(1,),
struct.pack("<bd", 1, 0.5),
byte_order="|<",
struct_layout="[16,[0,null],[8,null]]",
)
)
# a47c34b71f1d8c812f52d3a3b78ef6dbafb3b9adb320123b885cbacf72fa50b8

The same values through Orcapod:

import numpy as np
from orcapod.contexts import get_default_context
hasher = get_default_context().semantic_hasher
a = np.arange(6, dtype=np.float32).reshape(2, 3)
print(hasher.hash_object(a).to_string())
# ndarray_v1:bef92ae814ed78857164d1aea02e5590ff1ab7f00179f4c75d17aec7cae27317
print(hasher.hash_object(np.asfortranarray(a)).to_string())
# ndarray_v1:bb8d317f7bd44a4832eeb7468867b2137621fb539b084de988426dd5928af130
print(hasher.hash_object(a.astype(">f4")).to_string())
# ndarray_v1:1d5b6c2eb0ceaafd3758356256d651c5edf9ca0efa4deecf12c76c4e15bdb637
print(hasher.hash_object(np.array(2.5)).to_string())
# ndarray_v1:642ad052de6aa53ffd059e5cd9a2eeeb5519caa3d3eb12efcf9e0c3fdbeab17b
aligned = np.dtype([("a", "i1"), ("b", "<f8")], align=True)
print(hasher.hash_object(np.array([(1, 0.5)], dtype=aligned)).to_string())
# ndarray_v1:a47c34b71f1d8c812f52d3a3b78ef6dbafb3b9adb320123b885cbacf72fa50b8

A torch.Tensor value (since v0.3, ITL-871) is hashed by TorchTensorHandler, which returns ContentHash(method="torch_tensor_v1", digest=D). The hash is defined on the tensor, never on its stored bytes (a cell is a torch.save archive, whose bytes PyTorch does not promise to keep across versions). It has two steps.

Step 1 is the element digest E of the array value hash above, over (T, shape, B), where T is the PyTorch dtype name (float32, bfloat16), and B holds the logical elements (lazy conjugate and negative bits applied) in C order, little-endian. The dtypes NumPy does not have give their raw bits: bfloat16 the 16-bit patterns, complex32 the real then the imaginary float16 of each element, and the float8 tags (float8_e4m3fn, float8_e5m2, float8_e4m3fnuz, float8_e5m2fnuz, float8_e8m0fnu) one byte per element.

Step 2, the value hash:

lp(x) = u64le(len(utf8(x))) || utf8(x)
P = lp("orcapod.torch.tensor")
|| E (32 bytes)
|| u8(requires_grad) || u8(conj_bit) || u8(neg_bit)
|| u64le(n) || i64le(s0) || ... || i64le(sn-1)
|| lp(class name)
D = SHA-256(P)

s0 ... sn-1 are the strides, in elements, and the class name is torch.Tensor or torch.nn.parameter.Parameter. The leading text is the type's extension name; it keeps a tensor apart from every other array type, so a tensor and an ndarray with equal elements never share a hash, also inside an untyped Python structure.

The five properties of step 2 are read from the tensor as it is stored (Arrays and Tensors): a tensor that does not cover its buffer is stored as clone(memory_format=torch.preserve_format), so its strides are those of the clone and its lazy bits are 0. This is what makes hash(read(write(v))) == hash(v).

Each of the five properties has an option name: requires_grad, memory_layout, conj_bit, neg_bit, subclass. A name in the type's ignore list (the ignore array of the extension metadata, {"ignore":[...],"version":1}) writes that property's slot with its canonical value instead of the real one: 0, the strides of a C-contiguous tensor of that shape (the last is 1, each earlier one is the next stride times the next dimension's size, a size of 0 counting as 1), 0, 0, torch.Tensor. The slot is always present and the list itself is not in P. So values that differ only in ignored properties hash the same, and with all five ignored D equals the strict D of the plain contiguous tensor. A bare value is hashed with the list of the data context; a stored cell with the list of its own column. The cell token is <identifier text> || "::" || label || ":" || D, where the identifier text (the extension name, ;, the metadata in hex) holds the list: for the default, b"orcapod.torch.tensor;7b2269676e6f7265223a5b5d2c2276657273696f6e223a317d::torch_tensor_v1:" + D.

Variation Same D by default?
Dtype, shape, any element bit no
requires_grad no
Strides of a tensor that covers its buffer (transposed, channels_last) no
Lazy conjugate or negative bit of a tensor that covers its buffer no
torch.Tensor against torch.nn.Parameter no
Device (CPU, CUDA, MPS) yes
Autograd history (grad_fn, .grad) yes
A view of a larger buffer against its clone(memory_format=torch.preserve_format) yes

A reference implementation with the standard library only, with lp and element_digest from the array reference above:

def torch_tensor_hash(
element: bytes,
requires_grad: bool,
conj_bit: bool,
neg_bit: bool,
strides: tuple[int, ...],
class_name: str,
) -> str:
p = lp("orcapod.torch.tensor") + element
p += bytes([requires_grad, conj_bit, neg_bit]) + struct.pack("<Q", len(strides))
p += b"".join(struct.pack("<q", s) for s in strides) + lp(class_name)
return hashlib.sha256(p).hexdigest()
# torch.arange(6, dtype=torch.float32).reshape(2, 3)
elements = element_digest("float32", (2, 3), struct.pack("<6f", 0, 1, 2, 3, 4, 5))
print(torch_tensor_hash(elements, False, False, False, (3, 1), "torch.Tensor"))
# 118005088af13c5566b5bc6e798fd5906520e34da424811f6ba128ea2fbd8fcb

The same value through Orcapod (PyTorch is optional, so this sample needs it installed):

import torch
print(hasher.hash_object(torch.arange(6, dtype=torch.float32).reshape(2, 3)).to_string())
# torch_tensor_v1:118005088af13c5566b5bc6e798fd5906520e34da424811f6ba128ea2fbd8fcb

StarfixArrowHasher.hash_schema hashes each field under the same rule as hash_table. A logical type whose columns are hashed through a value handler (File, Directory, numpy.ndarray, torch, the pandas and Polars types, and list or set of them) is hashed as a large_binary field that carries the type's identifier; its storage type is not part of the hash. A logical type whose stored form is its identity (Path, uuid.UUID, dataclasses and other user classes) is hashed as its storage type with the identifier, because its stored values are what a table hash covers. Orcapod's own schema hash (site 3), type hash and pipeline hash do not use this method: they hash Python types, where a logical type contributes its identifier text only.

Every Arrow-derived hash in the sites above (row data hashes, record_ids built from a default source_id, base and pipeline entry ids, operator and cached-source record hashes, pa.Table values) comes from StarfixArrowHasher.hash_table, which returns ContentHash(method="arrow_v1", digest=D). D is 35 bytes: the starfix hash format number 00 00 02 followed by a SHA-256 digest, computed by starfix 0.5 (hash format 2) and byte-identical to the Rust starfix crate. The byte layout is specified in starfix's docs/byte-layout-spec.md.

Label. arrow_v1 names this procedure: the replacement of extension-typed cells by their value digests, then the starfix digest. It moves to arrow_v2 only when an existing table would get a different digest. Its number is not kept equal to the starfix format number.

Label starfix Format prefix Orcapod
arrow_v0.1 0.4 00 00 01 up to v0.2
arrow_v1 0.5 00 00 02 v0.3

What is part of the hash. Column names, Arrow types (with large and non-large layouts of strings, binaries and lists hashing alike), nullability at every depth (top-level columns, struct fields and list items, whether or not the column holds data), every value, row order, list element order, list lengths at every depth, null structs (distinct from structs whose fields are all null) and the row count. Column order and struct field order are not part of the hash: this is the hash of Orcapod's own tables (streams, datagrams, rows), which ignore order on purpose. A value whose own equality depends on order is hashed with StarfixArrowHasher.hash_table_ordered instead, which is the same procedure with starfix's ordered_columns and ordered_struct_fields options on, so that column order and struct field order at every depth are part of the digest. It is a procedure of its own, labelled arrow_ordered_v1; pandas values take it with that label (§10). How a table is split into chunks or slices is not part of it either. A fixed_size_list<T>[n] column hashes differently from a large_list<T> column with the same values. Extension columns are hashed as their storage type plus their extension name and metadata, including pyarrow's canonical extension types (arrow.uuid, arrow.fixed_shape_tensor); only ARROW:extension:* metadata is hashed, all other metadata is ignored. List item names (item, element) are not part of the hash.

The hashed type is the declared type. Before starfix sees a table, Orcapod replaces the values of logical types whose columns are hashed through a value handler (File, Directory, numpy.ndarray, ...) by their cell tokens, <identifier>::<label>:<digest>, typed large_binary. A UUID or Path column is hashed as stored. The type each column is hashed under is derived from its declared type alone (SemanticHashingVisitor.hashed_type): every struct and list field keeps its name, nullability and metadata. A column therefore hashes under one type whether it has data, only nulls or no rows (ITL-942: before v0.3 the type was rebuilt from the first non-null value, which dropped nested nullability for a column with data).

What cannot be hashed. starfix raises ValueError, naming the column path and type, for: data of null, map, interval, string_view and binary_view columns (whatever the row count; their schemas hash); union and run-end-encoded types; two schema nodes sharing a path (duplicate column names, or a name containing / that aliases a struct child, such as a column "a/b" next to a struct column a with a child b); and a non-nullable field holding a null. A source whose table has such a column fails when it is constructed.

A label names the algorithm that produced one hash. It does not say which generation of Orcapod's hashing a stored row belongs to. That is the data context: one key that changes whenever any existing hash value can change (the hashers, the identity definitions, the record-id recipes, the libraries behind them). Orcapod v0.3 writes std:v1:default; everything stored before v0.3 is the earlier generation, std:v0.1:default. The manifest of the current data context is orcapod/contexts/data/v1.json. It is frozen at the release of the library version that contains it and never edited afterwards, except that its golden history gains revisions as golden values are added and released (Golden Hashes). It names the hashers, the definitions that live in code, the libraries the hashes depend on and what changed from the previous data context. Logical types are not part of it: they are registered in orcapod/contexts/logical_types.json and versioned by their own identifier.

Every row Orcapod stores names its data context in the column __data_context:

Table Layout (last path component)
Function-node pipeline table pdb_v2
Function result table rdb_v2
Operator-node cache table odb_v1
Cached-source table sdb_v1
Side-effect invocation log tdb_v2

The layout component is always the last component of a table's path, also when a CachedSource is given an explicit cache_path.

Rows of the current data context are the current entries. Key lookups match rows of the current data context only, and every read of a whole table (a node's as_table(), iter_data(), get_all_records(), as_source(), a cached function's get_all_cached_outputs(), a stored operator node, a CachedSource) leaves rows of another data context out. Those rows are never changed or deleted, and a read that leaves rows out logs one warning per table with their number (logger orcapod.contexts.stored).

Tables written before v0.3 have no __data_context column and sit at an older layout (pdb_v1, rdb_v1, tdb_v1; operator-node and cached-source tables, and the function tables of Orcapod v0.1, had no layout component). Orcapod does not read them as results and never modifies them. When it finds one it logs one warning with the number of rows:

Found 3 rows in stored table '...' written by an earlier version of Orcapod (older table
layout, data context std:v0.1:default). They are not used and are left unchanged; they can
be migrated later.

How they are found. The path of a table holds hashes (of the function's output schema, of the pipeline), and a later version of Orcapod can compute other hashes, so an older table is not only looked for at the path the current version computes. The beginning of a path holds no hash: the job, then the name of the function (<job>/<function>/... and <job>/_result/<function>/...; side_effect_function/<function> for a function pod that takes the invocation context; side_effects/<pod> for a side-effect pod). A node lists the tables stored under that beginning (list_table_paths of the database) and tells old from current by the last path component, the layout: pdb_v2, rdb_v2, tdb_v2, odb_v1 and sdb_v1 are current, every other table under the name was written before v0.3. Tables found this way are not read, so they are reported without a number of rows, once per node:

Found 2 tables written by an earlier version of Orcapod under 'my_job/double' (older table
layout, data context std:v0.1:default): 'my_job/double/.../pdb_v1', ... They are not used
and are left unchanged; they can be migrated later. Tables of another function or node with
the same name are included.

Operator caches and cached sources have no name in their path that survives a change of hashes. A job run lists the tables under the job's name once and reports one total of the older tables that no function or side-effect node of the job reports itself: operator caches, cached sources stored under the job, and tables of nodes the job no longer has.

Store 'my_job' holds 2 more tables written by an earlier version of Orcapod (operator
caches, cached sources, or nodes not in this job): '...'. They are not used and are left
unchanged; they can be migrated later.

Two consequences of finding tables by name: a table of another function or pod with the same name in the same job, and a table of another node of the same function, count as well. A store that cannot list its tables (TableListingNotSupportedError) is not searched this way; tables at the path the current version computes are still found.

Nodes that perform side effects are the exception. A completion is looked up by its key, so completions stored under another data context would not be found and every side effect would be performed again. There are two kinds: a side-effect pod (@side_effect_pod, @sink_pod, @tap_pod), whose completions are in its invocation log, and a function pod that takes the invocation context (@function_pod(ctx_arg=...)), whose completions are in its pipeline and result tables. Either raises SideEffectLogDataContextError, naming the tables, before doing anything when it finds a table at an older layout anywhere under its name, or completions of another data context in its current tables. Because every older table under the name counts, an unrelated pod of the same name in the same job raises the error too; the error names every table so that you can tell. When the database cannot list its tables, the node logs one warning saying that it could not look for older completions, and proceeds.

A node stores rows for inputs of its own data context only: an input of another data context raises InputValidationError naming both keys before anything is written. This holds for nodes, caches and cached sources used directly; inside a PipelineJob every input is wrapped in a source node of the default data context, so the refusal is not reached there. (What should eventually happen instead is tracked in ITL-951.) std:v0.1:default itself is not shipped with v0.3 and does not resolve; a pipeline or job file saved by v0.2 loads under the current data context with a warning when its schema types are supported. Saved files with name-only logical extension types are rejected: current saved schema text requires the full logical type identifier. See Saving and loading a job.

stored_data_contexts(database, record_path) reads a stored table and counts its rows by data context. A table without the column reports all its rows under std:v0.1:default.

from orcapod import PipelineJob, function_pod
from orcapod.contexts import stored_data_contexts
from orcapod.databases import InMemoryArrowDatabase
from orcapod.sources import DictSource
@function_pod(output_keys="doubled")
def double_for_contexts(x: int) -> int:
return 2 * x
context_store = InMemoryArrowDatabase()
context_job = PipelineJob(name="contexts_demo", database=context_store)
with context_job:
rows = DictSource(data=[{"k": "a", "x": 1}, {"k": "b", "x": 2}], tag_columns=["k"], tag_only_columns=True)
_ = double_for_contexts.pod(rows, label="dbl")
context_job.run()
for path in sorted(context_store.list_sources()):
print(path[-1], stored_data_contexts(context_store, path))
# rdb_v2 {'std:v1:default': 2}
# pdb_v2 {'std:v1:default': 2}
import pyarrow as pa
from orcapod.contexts import get_default_context
hasher = get_default_context().arrow_hasher
digest = hasher.hash_table(pa.table({"x": pa.array([1, 2, 3], pa.int64())}))
print(digest.method, digest.digest[:3].hex())
# arrow_v1 000002

10. pandas Value Hash (pandas_dataframe_v1, pandas_series_v1)

Section titled “10. pandas Value Hash (pandas_dataframe_v1, pandas_series_v1)”

This hash is provisional in v0.3 (see section 8).

A pandas.DataFrame or pandas.Series value (a pod input or output, a value in a source column, or any object given to the semantic hasher) is hashed by pandas_frame_content_hash / pandas_series_content_hash in orcapod/hashing/pandas_value_hash.py.

The pandas dependency is capped below 3 (pandas>=2.2.3,<3) for v0.3.0: the hash and its round trip are verified on pandas 2 only. ITL-1054 lifts the cap.

Storage and hashing are separate (developer, November 12, 2026). Storage is the library's own serialization: the Arrow IPC stream of pa.Table.from_pandas(value, preserve_index=True), read back with to_pandas(). The hash is defined by Orcapod and is about the content of the value; it is never a hash of the stored bytes.

ContentHash("pandas_dataframe_v1" | "pandas_series_v1",
sha256(canonical_json(descriptor) + b"\x00" + b"arrow_ordered_v1:" + D))
  • D is the 35-byte digest of StarfixArrowHasher.hash_table_ordered (§9) over the values table: pa.Table.from_pandas(value, preserve_index=True) with its fields renamed by position, c0 .. cN-1 for the columns in frame order, then i0 .. iM-1 for the index levels in order. A Series is digested as its one-column frame. D covers the values, the Arrow types, row order, the row count, column order, index level order and struct field order.
  • descriptor is canonical JSON (sorted keys, , and : separators, UTF-8) in a grammar Orcapod owns, built from the pandas objects themselves. It carries what the Arrow values do not show: attrs, typed labels, columns.names, the dtype of the column axis, index level names and the pandas dtypes.
  • D enters with its label, arrow_ordered_v1:: it is computed by the data context's Arrow hasher, and a hash fed into another hash carries its label (ITL-1154).
frame: {"kind": "orcapod.pandas.dataframe", "height": <rows>, "attrs": <attrs>,
"columns": [[<label>, <dtype>], ...], in column order
"columns_names": [<label>, ...], columns.names, one per column level
"columns_dtypes": [<dtype>, ...], column axis, one per column level; a
pd.ArrowDtype level is
{"arrow": <Arrow type text>}
"index": [[<name>, <dtype>], ...]} index levels in order: name, dtype
series: {"kind": "orcapod.pandas.series", "height": <rows>, "attrs": <attrs>,
"name": <label>, "dtype": <dtype>,
"index": [[<name>, <dtype>], ...]}
<attrs> := attrs as the JSON object storage keeps ({1: "x"} is {"1": "x"}; a tuple is a
list); attrs with an entry JSON cannot write are refused
<name> := <str> | null an index level name as storage restores
it: its text
<label> := <str> | null | {"bool": b} | {"int": n} | {"float": <float.hex()>}
| {"tuple": [<label>, ...]}
| {"timestamp": [<count since the epoch, in unit>, <unit>, <tz or null>]}
| {"date": <iso date>}
| {"timedelta": [<count, in unit>, <unit>]}
| {"period": [<ordinal>, <period alias>]}
| {"nat": "datetime" | "timedelta"}
<dtype> := {"numpy": <numpy dtype name>}
| {"masked": "Int8" | ... | "UInt64" | "Float32" | "Float64" | "boolean"}
| "string" pd.StringDtype, any storage
| {"datetimetz": [<unit>, <tz>]} <tz>: the text storage writes:
"UTC", "Europe/Berlin", "+01:00"
| {"categorical": {"categories": [<dtype>, [<label>, ...]], "ordered": <bool>}}
| {"period": <period alias>} | {"interval": [<subtype dtype>, <closed>]}
| {"arrow": <Arrow type text>} pd.ArrowDtype of a non-nested Arrow type:
str(type), "large_binary", "null",
"timestamp[ns, tz=UTC]"

A nested, dictionary or run-end pd.ArrowDtype has no term: storage cannot read such a column back, so the value is refused (see "What is refused" below).

kind is the name of the value's logical type (its Arrow extension name).

The hash is strict. Everything about a value that can be changed is part of its identity (developer, November 11 and 12, 2026), with one exception: a property storage does not restore never counts, so that a value read back from a database hashes exactly like the value that was written. PANDAS_IDENTITY_PROPERTIES in orcapod/hashing/pandas_value_hash.py is the one list in code of every property below.

Property Part of the hash Carried by
Column order yes D (ordered)
MultiIndex level order (frame and Series) yes D (ordered)
Values, row order, row count, Arrow type (int width, datetime unit and timezone) yes D
Index values, index level names (as the text storage writes), index dtype (as storage restores it) yes D, descriptor
columns.name / columns.names yes descriptor
Dtype of the column axis, per level (an int64 versus an object Index of the same labels) yes descriptor
Column label type where the labels have one type (0 versus "0"), MultiIndex columns versus flat labels yes descriptor
Series name, typed (0 versus "0"); a frame versus its one-column Series yes descriptor
pandas dtype where the Arrow values coincide: Int64 versus int64, boolean versus bool, string versus object yes descriptor
Categorical versus string; the category list, including unused categories; category order; ordered yes descriptor
attrs (as the JSON object storage keeps: {1: "x"} is {"1": "x"}) yes, although pandas' equality ignores it descriptor
-0.0 versus 0.0 yes, although pandas calls them equal: the values are hashed as bytes D
flags never: storage does not restore it
Type of an index level name (1 versus "1") never: storage writes the name as text
Types of column labels of mixed type (0 beside "b" versus "0" beside "b") never: storage writes such labels as text
A masked, string or Arrow dtype of an index level (Int64 versus int64) never: storage restores an index level as NumPy
RangeIndex versus an int64 index with the same values never: storage restores an Index
freq of a DatetimeIndex or TimedeltaIndex never: storage does not restore it
Unused levels of a MultiIndex never: storage does not restore them
Storage of a string dtype (string[python] versus string[pyarrow]) never: storage restores string[python]
object dtype of a column that reads back as another dtype never: see below
Key order of dicts in an object column never: a dict is unordered

object columns are hashed in the form storage restores. An object column or index level whose values are not strings does not read back as object: [1, 2] reads back int64, [1, None] reads back float64, datetime objects read back datetime64[us]. Such a column is described and digested in the form it reads back in, so it hashes like the typed column it becomes. An object column of lists or dicts stays object, but its elements can change (a list [1, None] reads back [1.0, nan]), so the Arrow form of what is read back is digested. Any other object column that reads back as object (strings, bytes, decimals, dates) is hashed as it is.

The axes are described as storage restores them. Storage writes the name of an index level as text. The index names 1 and "1" are therefore one name once read back, and they hash equal: a hash that told them apart would change when the value is read from a store. columns.names and a Series' name are restored with their types, so they stay typed (0 versus "0" differ). The dtype of each level of the column axis is restored, so it is part of the hash. A timestamp or duration label is the integer count in its own unit, with the unit.

More consequences of the same rule:

  • Column labels of mixed type hash as their text. Storage restores labels of one type with that type (0, 1 stay integers), but writes labels of mixed type as text: the columns 0, "b" read back "0", "b". So a frame with the columns 0, "b" and one with "0", "b" hash equal.
  • An index level hashes as the NumPy level it reads back as. Storage restores a column with its masked, string or Arrow dtype, but an index level as NumPy: an Int64 index reads back int64 (float64 when it holds a missing value), a string index reads back object. So an Int64 index and the int64 index with the same values hash equal. Categorical, timezone-aware, period and interval index levels are restored and keep their dtype.
  • The categories of a categorical are described with the dtype storage restores. The categories are read back from the Arrow dictionary: object categories [1, 2] read back int64, and string categories read back object. So a categorical with object categories [1, 2] and one with int64 categories [1, 2] hash equal. The category list, its order, unused categories and ordered are restored and count.
  • A column is described with the dtype its Arrow schema restores. A pd.ArrowDtype(pa.string()) column reads back as string, so the two hash equal. An Arrow decimal column reads back as object holding Decimal values and is hashed in that form. Other pd.ArrowDtype columns are restored as they are, with their exact Arrow type.
  • A column that reads back as Arrow null is digested as an all-null large_string column. Starfix refuses a null column, and storage cannot restore the type of a column with no value: an all-null or empty pd.ArrowDtype decimal column reads back as object holding None, exactly like a plain object column of None. Every column whose restored Arrow type is null is therefore digested as an all-null large_string column of the same length (a cast of a null array holds no value, so nothing is inferred), for a value and for its read-back alike. The dtype descriptor of both the all-null or empty decimal column and the all-null or empty object column is {"numpy": "object"}, so they hash equal; the height still counts, and a string-dtype column of None (descriptor "string") stays different. Other all-null or empty columns (Arrow string, timestamp, integer, Int64, string, categorical) keep their own types and are unchanged. A pd.ArrowDtype of a non-nested Arrow type is described by its Arrow type text, so pd.ArrowDtype(pa.null()) ({"arrow": "null"}) never hashes like a pd.ArrowDtype(pa.large_string()) column of None, and binary and large_binary columns (which storage restores as themselves, although D unifies large and non-large variants) hash differently. A decimal column is the exception that shows the rule: storage restores it as object (each value comes back exactly, digits and scale included; the dtype and its declared precision do not), so two precisions hash equal. A nested pd.ArrowDtype (list, struct, map, union), a dictionary or a run-end type is refused: storage cannot read it back (ITL-1054).
  • A timezone is the text storage writes for it ("UTC", "Europe/Berlin", "+01:00"), whatever class holds it: datetime.timezone(timedelta(hours=1)) is read back as another class of the same offset, and both hash equal.

The descriptor holds the column labels, columns.names, a Series' name and the index names as read from the frame storage builds for the value's Arrow schema. The value's own labels and names must still be in the grammar: a label of another type raises ValueError.

attrs must be JSON. Storage writes attrs as JSON. A key that is not text is written as text ({1: "x"} reads back {"1": "x"}, and hashes like it), and a tuple reads back as a list. attrs with an entry JSON cannot write (a set, an arbitrary object, a tuple as a key) are refused: the hash and python_to_storage raise ValueError naming the key, on every supported pyarrow version (pyarrow itself would drop the whole attrs with a warning, or raise TypeError, depending on its version). With attrs in the ignore list the hash does not read them (it never copies them either, so attrs that cannot be copied do not matter then).

A label or dtype outside the grammar raises ValueError naming it (a dtype error names its column).

What is refused. Storage refuses, when the value is written, a value it would not restore, and the hash refuses the same values with the same ValueError (one check, require_restorable in orcapod/hashing/pandas_value_hash.py; ITL-1054). The message names the column, index level or column-axis level and the reason. Refused, on every supported pyarrow version:

  • MultiIndex levels that would be written in another order. pyarrow fetches index level i with index.get_level_values(i), which pandas resolves as a level name first. So level names that are also level positions (names=[1, 0]; 1.0 and True count as 1) would be written, and read back, with the levels swapped, and nothing would raise.
  • Values storage cannot read back. A pd.ArrowDtype column (or Series) of a list, large list, fixed-size list, list view, struct, map, dictionary, run-end or union type, of fixed-size binary, or of an Arrow extension type (uuid, json, bool8, a fixed-shape tensor); an index level of a pd.ArrowDtype list, large list, fixed-size list, struct, map, dictionary, fixed-size binary or float16 type; a column axis (any level of it) that is categorical, period, timedelta, interval, datetime.date objects, or a pd.ArrowDtype timestamp, duration, time, bool, date, decimal or unsigned integer, which includes a Series named with a pd.Timedelta, pd.Period or datetime.date; a categorical column, Series or index whose categories are tuples (they read back as arrays, which cannot be categories; ITL-1115) or intervals.
  • Index levels whose values cannot be read back. A pd.ArrowDtype time64[ns] index level holding a value that is not a whole number of microseconds (ITL-1118), and a MultiIndex level holding tuples or lists, which read back as arrays that cannot be the values of a MultiIndex level (ITL-1120). These failures depend on the values, so each index level is converted once, in full, and a MultiIndex is assembled once, when a value is written or hashed; columns are checked without converting any row.
  • Index level names that would read back as another name. pyarrow drops a name of the form __index_level_<n>__ (its own name for an unnamed level), so such a level reads back unnamed (ITL-1119). Names are compared as the text storage writes for them.
  • Column labels that would read back as another label. pyarrow restores a bool label from its text, so the label False (a one-column bool axis, a Series named False, a False in a level of a MultiIndex column axis) reads back True (ITL-1116). Labels are compared as text, so mixed-type labels, which storage writes as text, are not a change.
  • Labels and names that collide once written as text (the index name 1 with a column "1"; integer names of MultiIndex levels): two fields of one name, and the value read back would have lost its index. Distinct labels that read back equal (the labels True and False both read back True) would come back as duplicates.
  • Values Arrow cannot convert (mixed or non-Arrow values, complex128, sparse columns, a categorical of periods, a float16 column axis, an index of a type Arrow does not support).

Before v0.3 such values were written: a value of the first kind reads back changed, and one of the second kind cannot be read at all.

import pandas as pd
from orcapod.contexts import get_default_context
from orcapod.logical_types.pandas_type import LogicalPandasDataFrame
semantic_hasher = get_default_context().semantic_hasher
swapped = pd.DataFrame(
{"a": [10, 20]},
index=pd.MultiIndex.from_tuples([("p", 7), ("q", 8)], names=[1, 0]),
)
try:
LogicalPandasDataFrame().python_to_storage(swapped)
except ValueError as error:
print(str(error).split(":")[0])
# index level 0 (name 1)
try:
semantic_hasher.hash_object(swapped)
except ValueError as error:
print(str(error).split(":")[0])
# index level 0 (name 1)

What a data context can make the hash ignore. Nothing is ignored by default. The registration file (orcapod/contexts/logical_types.json, hash_ignore) can name, per type, properties the hash ignores. The list is saved in the logical type's extension metadata, so two lists are two logical types, and a column stored under another list is refused on read. The preimage keeps its shape: an ignored property is hashed with a fixed value.

Option Type When ignored
column_order DataFrame the columns are hashed sorted by the canonical JSON of their typed labels
series_name Series name is hashed as null
attrs both attrs is hashed as {}

Every other property always counts, and the properties storage does not restore never do; neither kind is an option.

Names. The logical types are orcapod.pandas.dataframe and orcapod.pandas.series with metadata {"ignore":[],"version":1} (the version and the ignore list). Before v0.3 they were stored as pandas.dataframe / pandas.series; those names are not registered any more, and a column stored under them is refused on read. Before v0.3 a pandas value was also hashed as an Arrow table (§9, order ignored), so a frame and the same frame with its columns reordered shared a hash, and a cached pod could be served the result computed for the other one (ITL-897, ITL-887).

import pandas as pd
from orcapod.contexts import get_default_context
semantic_hasher = get_default_context().semantic_hasher
frame = pd.DataFrame({"a": [1, 2], "b": ["x", "y"]})
print(semantic_hasher.hash_object(frame).method)
# pandas_dataframe_v1
print(semantic_hasher.hash_object(frame) == semantic_hasher.hash_object(frame.copy()))
# True
print(semantic_hasher.hash_object(frame) == semantic_hasher.hash_object(frame[["b", "a"]]))
# False
print(semantic_hasher.hash_object(frame["a"]).method)
# pandas_series_v1