Arrays and Tensors
A numpy.ndarray can be a data value like any other: a source column, a function pod
parameter or a return value. Storage and hashing are two separate things:
- Storage is NumPy's own
.npyformat. An array reads back as it was given: dtype, byte order, shape, C or Fortran memory order and structured field offsets are kept. - The hash is defined by Orcapod on the array's value, never on the stored bytes. It is strict: everything about the array that storage keeps is part of its identity.
import numpy as np
from orcapod import function_podfrom orcapod.sources import DictSource
@function_pod(output_keys="scaled")def scale(signal: np.ndarray) -> np.ndarray: return signal * 2
source = DictSource( data=[{"trial": "t1", "signal": np.arange(6, dtype=np.float32).reshape(2, 3)}], tag_columns=["trial"], source_id="trials",)for tag, data in scale.pod(source): print(tag["trial"], data["scaled"].dtype, data["scaled"].shape)# t1 float32 (2, 3)Provisional value hashes
Section titled “Provisional value hashes”The value hashes and stored identities of the library value types are provisional in v0.3:
numpy.ndarray, torch.Tensor, and the pandas and polars DataFrame and Series. They are
provisional until the library value type audit (after 0.4.0rc1) and may change then. Every other type, and the hashing of everything
else in a pipeline, is unaffected.
Using one of these types issues orcapod.errors.ProvisionalHashWarning once per type per
process, on whichever use comes first: a value hashed, a source column of the type, a function
pod with a parameter or output of the type, or stored results of the type read back. It names
the type, for example orcapod.numpy.ndarray.
In practice nothing needs to change. Use these types as described on this page. If the audit changes how one of them is hashed, a result Orcapod computed earlier from such a value gets a different identity, so the release that carries the change computes it again instead of reusing it. Nothing stored is lost or becomes wrong: the earlier rows stay where they are. Plan for that recomputation if a pipeline is expensive. To silence the warning once you have taken note of it, filter its class after importing Orcapod:
import warnings
from orcapod.errors import ProvisionalHashWarning
warnings.filterwarnings("ignore", category=ProvisionalHashWarning)Python's -W option and PYTHONWARNINGS cannot name this class: they are read before
installed packages can be imported.
What counts as the same array
Section titled “What counts as the same array”Two arrays are the same value when everything below that counts is equal.
Always part of the hash:
- Dtype, including width:
int32andint64arrays with equal values differ. So dodatetime64[s]anddatetime64[ms],U5andU6. For a structured dtype the field names, the declared field order and the field types count, at every nesting depth:[("a", "i4"), ("b", "f8")]and[("b", "f8"), ("a", "i4")]are different arrays, as they are to NumPy's own dtype equality. - Shape, including empty shapes:
(0, 3)and(3, 0)differ, and so do a 0-d array and a one-element 1-d array. - Every element bit:
0.0and-0.0differ, and so do NaNs with different payloads.
Part of the hash by default; each has a name and can be put on the type's ignore list:
| Name | What it is |
|---|---|
memory_order |
C or Fortran. An array counts as Fortran when it is Fortran-contiguous and not C-contiguous; every other array counts as C. |
byte_order |
Little- or big-endian, for each scalar field. A native array hashes as the byte order it actually has, never as "native": the same array bytes hash the same on every machine, and an array created natively on a big-endian machine is a big-endian array. |
struct_layout |
The field offsets and the item size of a structured dtype, at every nesting depth. An aligned (padded) dtype and the packed dtype with the same fields differ. |
Never part of the hash, because .npy does not keep it. A value read back from a database must
hash like the value that was written, or every cached run would compute again:
- Strides other than C or Fortran order. A step slice (
a[:, ::2]), a reversed view or a broadcast view is stored C-contiguous, and hashes like its C-contiguous copy. - The subclass.
np.matrix,np.recarray,np.memmapand other subclasses are stored as their values and read back as plainnp.ndarray. The exception is a subclass with its own registered logical type, which is used for that subclass and its own subclasses. - The
writeableflag. An array always reads back writeable. - Dtype
metadata, and thealign=Trueflag of a structured dtype (the offsets that flag produced are kept and count, asstruct_layout). - Padding bytes between structured fields. NumPy does not define their content (an
np.emptyarray holds whatever the memory held). They are not part of the value: they are written to the database as zero, so residual memory never reaches a database that may be shared, and they read back as zero. Every field reads back as written.
from orcapod.contexts import get_default_context
hasher = get_default_context().semantic_hashera = np.arange(6, dtype=np.float32).reshape(2, 3)
print(hasher.hash_object(a) == hasher.hash_object(a.copy()))# Trueprint(hasher.hash_object(a) == hasher.hash_object(np.asfortranarray(a)))# Falseprint(hasher.hash_object(a) == hasher.hash_object(a.astype(">f4")))# Falseprint(hasher.hash_object(a) == hasher.hash_object(a.astype(np.float64)))# Falsewide = np.arange(12, dtype=np.float32).reshape(2, 6)view = wide[:, ::2]print(hasher.hash_object(view) == hasher.hash_object(np.ascontiguousarray(view)))# Truepacked = np.zeros(1, dtype=[("a", "i1"), ("b", "f8")])aligned = np.zeros(1, dtype=np.dtype([("a", "i1"), ("b", "f8")], align=True))print(hasher.hash_object(packed) == hasher.hash_object(aligned))# FalseThe ignore list
Section titled “The ignore list”What the hash ignores is a setting of the array type, not of a single call. The data context
holds one list of ignored property names for the type; the default list is empty. The list
is written into the type's Arrow extension metadata, so it is part of the
logical type identifier: the array type with an empty list
and the array type that ignores memory_order are two different logical types, with
different schema hashes.
- A bare array (a function default, a value inside a Python structure) is hashed with the data context's list.
- An array in a table is hashed with the list of its own column.
- A data context reads only array columns stored under its own list. A column stored under
another list raises
LogicalTypeIdentifierError, naming the column and both lists.
When a property is ignored, arrays that differ only in it hash equal, and they hash like
the array in canonical form (C order, little-endian, packed fields) does under the empty
list. The list is configured in the hash_ignore object of Orcapod's logical type
registration file (contexts/logical_types.json); v0.3 has no other way to change it. A
name that is not one of the three above is an error when the data context is built.
The same mechanism serves the other library value types, each with its own names.
What cannot be stored
Section titled “What cannot be stored”Some arrays are refused when they are written, with a ValueError (an
orcapod.errors.UnsupportedValueError) that names the column and the dtype:
| Array | Why | What to do |
|---|---|---|
Masked arrays (np.ma.MaskedArray) |
extended data: the mask would be lost | store the data and the mask as two arrays |
dtype=object, or a structured dtype with an object field |
extended data: would need pickling | convert to a fixed dtype |
NumPy 2 StringDType |
extended data: variable width | use a fixed-width string dtype such as U10 |
longdouble, clongdouble |
width and format differ by platform | .astype(np.float64) or .astype(np.complex128) |
datetime64 or timedelta64 without a unit |
the integer has no defined meaning | give a unit, e.g. datetime64[ns] |
| Structured dtypes with field titles | titles are aliases of fields | drop the titles |
| Structured dtypes whose fields overlap or are not in offset order | .npy has no form for them |
declare the fields in offset order |
Structured dtypes with several hundred fields (a .npy header over 10,000 bytes) |
NumPy does not read such a header without pickle support | split the array |
Extended data may be supported later, stored as several parts rather than as one array.
Whether longdouble and clongdouble, unit-less datetimes and timedeltas, and structured
dtypes with titles should be accepted in a later release is tracked in ITL-911; v0.3 refuses
them.
try: DictSource( data=[{"trial": "t1", "signal": np.ma.array([1.0, 2.0], mask=[False, True])}], tag_columns=["trial"], )except ValueError as exc: print(type(exc).__name__)# UnsupportedValueErrorA function pod that declares -> np.ndarray must return an array. A NumPy scalar, for
example x * 2 on a 0-d array, raises TypeError that names the output. Return
np.asarray(...) instead.
How an array is stored
Section titled “How an array is stored”An array column has the Arrow extension type orcapod.numpy.ndarray over large_binary.
Each cell is one .npy file, written by NumPy itself
(numpy.lib.format.write_array, format version 3.0, allow_pickle=False) and read with
numpy.lib.format.read_array(..., allow_pickle=False). Format 3.0 is the one that stores
field names outside Latin-1. Pickle is never written and never read: a cell that holds an
object array, is not a complete .npy file of format 3.0, or does not hold exactly the
element bytes its header declares, raises a ValueError that names the column.
The extension metadata is {"ignore":[],"version":1}: the ignore list and the version of
the type. A null array is a null cell, allowed when the column is annotated
np.ndarray | None.
import io
table = source.as_table()print(table.schema.field("signal").type.extension_name)# orcapod.numpy.ndarraycell = table.column("signal").chunk(0).storage[0].as_py()print(np.load(io.BytesIO(cell), allow_pickle=False).dtype)# float32What reads back:
| Property | Reads back |
|---|---|
| dtype, including byte order, units, string widths, field names, offsets and item size | as given |
| shape and element bits | as given |
| C or Fortran memory order | as given |
| any other strides (step slices, reversed or broadcast views) | C-contiguous |
| subclass | plain np.ndarray |
writeable flag |
writeable |
dtype metadata, the align flag |
not kept |
| padding bytes between structured fields | zero |
big = np.asfortranarray(np.arange(6, dtype=">f4").reshape(2, 3))kept = DictSource( data=[{"trial": "t1", "signal": big}], tag_columns=["trial"], source_id="kept")for tag, data in kept: print(data["signal"].dtype.str, data["signal"].flags.f_contiguous)# >f4 TrueA stored array costs at least 128 bytes of .npy header. The stored bytes are NumPy's and
may differ between NumPy versions; nothing in Orcapod depends on them. Where an operator has
to order arrays (MergeJoin sorts the values it merges), it orders them by their value hash.
How an array is hashed
Section titled “How an array is hashed”The hash of an array is labelled ndarray_v1 and is computed in two steps. The first is the
element digest of the array's dtype, shape and elements, taken on the array in canonical
form (C order, little-endian, fields packed). The second adds the type's name and the three
properties of the table above. The byte-level definition, the dtype tags and a reference
implementation that does not use Orcapod are in the
Hashing Reference. A PyTorch tensor type shares
the element digest and has its own name and properties, so an array and a tensor with equal
values have different hashes.
PyTorch tensors
Section titled “PyTorch tensors”A torch.Tensor can be a data value too. PyTorch is an optional dependency. Tensor support
works whenever PyTorch can be imported, however it was installed: uv add "orcapod[torch]"
(or pip install "orcapod[torch]") installs it from PyPI, or install the build that suits
your hardware yourself (see pytorch.org; on Linux the PyPI build includes CUDA, and pytorch.org also offers smaller
CPU-only builds). Orcapod needs PyTorch 2.6 or later, the first version whose restricted
loader (below) reads uint16, uint32 and uint64 tensors, and is tested with PyTorch 2.15. Orcapod
hashes a tensor's elements through NumPy (Tensor.numpy()), so your PyTorch build must
support the NumPy version you have installed: some older builds cannot use NumPy 2 and fail
with "Numpy is not available". If you see that error, upgrade PyTorch or install a NumPy
version your PyTorch build supports.
Importing Orcapod never imports PyTorch. The tensor type is set up the first time Orcapod
meets a tensor or a torch.Tensor annotation, or reads a stored tensor column. Reading a
stored tensor column without PyTorch installed raises
orcapod.errors.MissingOptionalDependencyError, which names the missing package.
import torch
from orcapod import function_podfrom orcapod.sources import DictSource
@function_pod(output_keys="scaled")def scale_tensor(signal: torch.Tensor) -> torch.Tensor: return signal * 2
tensors = DictSource( data=[{"trial": "t1", "signal": torch.arange(6, dtype=torch.float32).reshape(2, 3)}], tag_columns=["trial"], source_id="tensor_trials",)for tag, data in scale_tensor.pod(tensors): print(tag["trial"], data["scaled"].dtype, tuple(data["scaled"].shape))# t1 torch.float32 (2, 3)A tensor is a different type from an array. The column's extension name is
orcapod.torch.tensor and the hash label is torch_tensor_v1. A tensor and an ndarray with
equal dtype, shape and values never hash the same: not as values, not inside a dict or
list, not as columns, schemas or types. A pod that declares x: torch.Tensor cannot take an
np.ndarray column (or the reverse). Convert at the pod boundary when you need to:
torch.from_numpy(a) or t.numpy().
How a tensor is stored
Section titled “How a tensor is stored”Storage and hash are decided separately. A tensor is stored with PyTorch's own serialization:
one cell holds the bytes torch.save writes for that tensor, in a large_binary column. It
is read with torch.load(..., map_location="cpu", weights_only=True). That is PyTorch's
restricted loader: it rebuilds tensors and runs no other pickled code. Orcapod always passes
both arguments, never falls back to an unrestricted load, and never adds a class to the
loader's allow list; the environment variable TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD does not
change that. A cell that is not a tensor archive raises an error naming the column.
What a stored tensor keeps:
| Property | Stored and read back? |
|---|---|
| dtype, shape, element bits | yes |
requires_grad |
yes |
Memory layout: strides (transposed views, channels_last) |
yes, for a tensor that covers its buffer (below) |
Lazy conjugate and negative bits (t.conj()) |
yes, for a tensor that covers its buffer |
Class: torch.Tensor or torch.nn.Parameter |
yes |
| Device (CUDA, MPS) | no: a stored tensor is read on the CPU |
Autograd history: grad_fn, .grad, hooks |
no: a stored tensor is read as a leaf |
Only the elements you pass are stored. torch.save writes the whole buffer under a
tensor, so a 3-element slice of a 1,000,000-element tensor would store all million elements.
When a tensor does not cover its underlying buffer exactly (a slice, a step slice such as
t[::2], an expanded view), Orcapod stores value.clone(memory_format=torch.preserve_format)
instead. The clone holds just the elements of the view. It has PyTorch's strides for that
clone, and a lazy conjugate or negative bit is resolved into the elements: x.conj().imag,
which is a negative view of every second number of x, is stored as plain negated values.
A tensor that covers its buffer (x.conj(), x.transpose(0, 1)) is stored as it is.
Orcapod never modifies your tensor. One stored tensor costs about 1.5 KB more than its elements, so a column of many very small tensors is larger than the same data in one tensor.
What counts as the same tensor
Section titled “What counts as the same tensor”The hash is defined by Orcapod and never reads the stored bytes. It is strict: by default two tensors hash the same only when all of these are equal.
| Property | In the hash | Can be ignored (option name) |
|---|---|---|
dtype, shape, element bits (-0.0 and NaN payloads count) |
always | no |
requires_grad |
by default | requires_grad |
| Memory layout (strides) | by default | memory_layout |
| Lazy conjugate bit | by default | conj_bit |
| Lazy negative bit | by default | neg_bit |
Class (torch.Tensor or torch.nn.Parameter) |
by default | subclass |
Device, autograd history, .grad, position inside a larger buffer |
never |
A property that storage does not keep is never in the hash: otherwise a value read back from a database would hash differently from the value that was written, and every cached result would be computed again. For the same reason the optional properties are those of the stored tensor: a step slice hashes like its contiguous clone.
The properties the hash ignores are the type's ignore list. It is empty by default. It is
configured per data context (hash_ignore in orcapod/contexts/logical_types.json) and is
written in the column's extension metadata, {"ignore":[],"version":1}, so it is part of the
type's identifier: columns with different lists are different
types, and a data context refuses to read a tensor column stored under another list. An
ignored property is still stored and read back; it only stops counting in the hash. Orcapod
v0.3 has no user API for the list.
from orcapod.contexts import get_default_context
hasher = get_default_context().semantic_hashert = torch.arange(6, dtype=torch.float32).reshape(2, 3)print(hasher.hash_object(t) == hasher.hash_object(t.clone()))# Trueprint(hasher.hash_object(t.T) == hasher.hash_object(t.T.contiguous()))# Falseprint(hasher.hash_object(t) == hasher.hash_object(t.clone().requires_grad_(True)))# Falseprint(hasher.hash_object(t) == hasher.hash_object(torch.nn.Parameter(t, requires_grad=False)))# Falseprint(hasher.hash_object(t).digest == hasher.hash_object(t.numpy()).digest)# Falseprint(hasher.hash_object(t).to_string())# torch_tensor_v1:...The byte-level definition is in the Hashing Reference.
Device and autograd history
Section titled “Device and autograd history”A pod may return a GPU tensor, or one computed from tensors that track gradients (it has a
grad_fn), and a source may hold one. Storage keeps neither the device nor the history, so
Orcapod stores a CPU leaf tensor and issues orcapod.errors.ValueConversionWarning, which
names the function and output (or the source and column), says what was converted, and how to
avoid it: return or pass value.detach().cpu(). A function warns once per output and
conversion, however many rows it processes; a source warns once per column, with the number
of values converted. A CPU tensor without a grad_fn never warns, also when it has
requires_grad=True or is an nn.Parameter: those are stored. To treat the conversion as an
error, call warnings.simplefilter("error", ValueConversionWarning) in your program (import
the class from orcapod.errors); every conversion then raises, and in a pipeline that row
fails. Python's -W command-line option and PYTHONWARNINGS cannot name this class: they are
read before installed packages can be imported, and Python ignores the filter.
import warnings
from orcapod.errors import ValueConversionWarning
@function_pod(output_keys="weights")def fit(signal: torch.Tensor) -> torch.Tensor: return signal.clone().requires_grad_(True) * 0.5
with warnings.catch_warnings(record=True) as caught: warnings.simplefilter("always") list(fit.pod(tensors).iter_data())print(caught[0].category is ValueConversionWarning)# Trueprint(caught[0].message)# Function 'fit' output 'weights' returned a torch tensor with autograd history (a grad_fn); Orcapod stored a CPU tensor without autograd history, and the returned tensor is unchanged. Return value.detach().cpu() to avoid this warning.What comes back. A tensor read from a database or a source is a CPU leaf tensor with the
stored dtype, shape, elements, strides, requires_grad, lazy bits and class (a bfloat16
tensor reads back as bfloat16, an nn.Parameter as an nn.Parameter). Within a single
in-memory run, though, the next pod receives the very object the previous pod returned, on its
device and with its autograd history. The same holds for NumPy arrays (a Fortran-ordered array
passes through in Fortran order). Write pods that do not depend on the device or the autograd
history of their inputs.
Dtypes and refusals
Section titled “Dtypes and refusals”bool, int8 to int64, uint8 to uint64, float16, float32, float64, complex64
and complex128 are hashed with the NumPy tags and bytes from the table above. The dtypes
NumPy does not have are hashed as their raw bits, each element little-endian, under the
PyTorch dtype name as the tag:
| Dtype | Element bytes |
|---|---|
bfloat16 |
2: the bfloat16 bit pattern |
complex32 |
4: the real part, then the imaginary part, each a float16 |
float8_e4m3fn, float8_e5m2, float8_e4m3fnuz, float8_e5m2fnuz, float8_e8m0fnu |
1: the float8 bit pattern |
A stored dtype that your PyTorch does not have (float8_e8m0fnu needs PyTorch 2.7) raises an
error naming the column when it is read; everything else still works. These are refused when
they are written, with an UnsupportedValueError that names the column and the reason:
| Tensor | What to do |
|---|---|
sparse or mkldnn layout |
.to_dense() |
| quantized | .dequantize() |
| nested (jagged) | store the parts as separate tensors |
on the meta device |
it holds no data |
| named dimensions (PyTorch versions that have them) | .rename(None) |
sub-byte and packed dtypes (int1 to int7, uint1 to uint7, float4_e2m1fn_x2, bits*), bcomplex32 |
convert to a supported dtype |
a subclass other than torch.nn.Parameter |
.as_subclass(torch.Tensor), or register a logical type and a value handler for the class (PyTorch's restricted loader cannot read it back, and the tensor hash, which includes the class, is defined for torch.Tensor and nn.Parameter only) |
A function pod that declares -> torch.Tensor must return a tensor. An np.ndarray or a
Python number raises OutputTypeError that names the function and the output. Return
torch.as_tensor(...) instead. An nn.Parameter is a tensor, so it is accepted; it is
stored, read back and hashed as an nn.Parameter. A pod that declares -> torch.nn.Parameter
must return one.