Types
Core type definitions used throughout Orcapod.
Schema
Section titled “Schema”class · orcapod.types.Schema
class Schema( fields: Mapping[str, DataType] | None = None, optional_fields: Collection[str] | None = None, **kwargs: type,)Bases: Mapping[str, DataType]
Immutable schema representing a mapping of field names to Python types.
Serves as the canonical internal schema representation in OrcaPod, with interop to/from Arrow schemas. Hashable and suitable for use in content-addressable contexts.
Parameters:
- fields (
Mapping[str, DataType] | None) defaultNone: An optional mapping of field names to their data types. - **kwargs (
type) default{}: Additional field name / type pairs. These are merged withfields; keyword arguments take precedence on conflict.
Example::
schema = Schema({"x": int, "y": float})schema = Schema(x=int, y=float)optional_fields
Section titled “optional_fields”property · orcapod.types.Schema.optional_fields
Type: frozenset[str]
Field names that are optional (have a default value in the source function).
required_fields
Section titled “required_fields”property · orcapod.types.Schema.required_fields
Type: frozenset[str]
Field names that must be present in an incoming data.
is_required
Section titled “is_required”method · orcapod.types.Schema.is_required
def is_required(field: str) -> boolReturn True if field must be present (has no default).
method · orcapod.types.Schema.merge
def merge(other: Mapping[str, type]) -> SchemaReturn a new Schema that is the union of self and other.
Parameters:
- other (
Mapping[str, type]): A mapping of field names to types to merge in.
Returns:
Schema: A newSchemacontaining all fields from both schemas.
Raises:
ValueError: If any shared field has a different type inother.
with_values
Section titled “with_values”method · orcapod.types.Schema.with_values
def with_values(other: dict[str, type] | None, **kwargs: type) -> SchemaReturn a new Schema with the specified fields added or overridden.
Unlike merge, this method silently overrides existing fields when
a key already exists.
Parameters:
- other (
dict[str, type] | None): An optional mapping of field names to types. - **kwargs (
type) default{}: Additional field name / type pairs.
Returns:
Schema: A newSchemawith the updated fields.
select
Section titled “select”method · orcapod.types.Schema.select
def select(*fields: str) -> SchemaReturn a new Schema containing only the specified fields.
Parameters:
- *fields (
str) default(): Names of the fields to keep.
Returns:
Schema: A newSchemawith only the requested fields.
Raises:
KeyError: If any of the requested fields are not present.
method · orcapod.types.Schema.drop
def drop(*fields: str) -> SchemaReturn a new Schema with the specified fields removed.
Parameters:
- *fields (
str) default(): Names of the fields to drop. Fields not present in the schema are silently ignored.
Returns:
Schema: A newSchemawithout the dropped fields.
is_compatible_with
Section titled “is_compatible_with”method · orcapod.types.Schema.is_compatible_with
def is_compatible_with(other: Schema) -> boolCheck whether other is a superset of this schema.
Parameters:
- other (
Schema): The schema to compare against.
Returns:
bool:Trueifothercontains every field inselfwith abool: matching type.
as_required
Section titled “as_required”method · orcapod.types.Schema.as_required
def as_required() -> SchemaReturn a copy of this schema with all fields marked as required.
Strips optional-field metadata so the result reflects the structural (Arrow-level) schema, where every field is unconditionally present.
Returns:
Schema: A newSchemacontaining the same fields and types but withSchema:optional_fieldsset to the empty frozenset.
classmethod · orcapod.types.Schema.empty
def empty() -> SchemaCreate an empty schema with no fields.
Returns:
Schema: A newSchemacontaining zero fields.
ColumnConfig
Section titled “ColumnConfig”class · orcapod.types.ColumnConfig
class ColumnConfig( meta: bool | Collection[str] = False, context: bool = False, source: bool = False, system_tags: bool = False, content_hash: bool | str = False, sort_by_tags: bool = False, all_info: bool = False,)Configuration for column inclusion in DatagramProtocol/DataProtocol/TagProtocol operations.
Controls which column types to include when converting to tables, dicts, or querying keys/types.
Attributes:
- meta (
bool | Collection[str]): Include meta columns (with '__' prefix).- False: exclude all meta columns (default)
- True: include all meta columns
- Collection[str]: include specific meta columns by name (prefix '__' is added automatically if not present)
- context (
bool): Include context column - source (
bool): Include source info columns (DataProtocol only, ignored for others) - system_tags (
bool): Include system tag columns (TagProtocol only, ignored for others) - all_info (
bool): Include all available columns (overrides other settings)
Examples:
>>> # Data columns only (default)>>> ColumnConfig()>>> # Everything>>> ColumnConfig(all_info=True)>>> # Or use convenience method:>>> ColumnConfig.all()>>> # Specific combinations>>> ColumnConfig(meta=True, context=True)>>> ColumnConfig(meta=["pipeline", "processed"], source=True)>>> # As dict (alternative syntax)>>> {"meta": True, "source": True}classmethod · orcapod.types.ColumnConfig.all
def all() -> SelfConvenience: include all available columns
data_only
Section titled “data_only”classmethod · orcapod.types.ColumnConfig.data_only
def data_only() -> SelfConvenience: include only data columns (default)
handle_config
Section titled “handle_config”classmethod · orcapod.types.ColumnConfig.handle_config
def handle_config(config: Self | dict[str, Any] | None, all_info: bool = False) -> SelfNormalize column configuration input.
Parameters:
- config (
Self | dict[str, Any] | None): ColumnConfig instance or dict to normalize. - all_info (
bool) defaultFalse: If True, override config to include all columns.
Returns:
Self: Normalized ColumnConfig instance. Whenall_infois set, by theSelf: keyword or insideconfig({"all_info": True}orSelf:ColumnConfig(all_info=True)), every group is on, whatever elseSelf:configsays; an embeddedall_info=Trueis not cancelled by aSelf:Falsekeyword.
ContentHash
Section titled “ContentHash”class · orcapod.types.ContentHash
class ContentHash(method: str, digest: bytes)Content-addressable hash pairing a hashing method with a raw digest.
ContentHash is the standard way to represent hashes throughout OrcaPod.
It is immutable (frozen dataclass) and provides convenience methods to
convert the digest into hex strings, integers, UUIDs, base64, and
human-friendly display names.
Attributes:
- method (
str): Identifier for the hashing algorithm / strategy used (e.g."arrow_v2.1"). - digest (
bytes): The raw hash bytes.
to_hex
Section titled “to_hex”method · orcapod.types.ContentHash.to_hex
def to_hex(char_count: int | None = None) -> strConvert the digest to a hexadecimal string.
Parameters:
- char_count (
int | None) defaultNone: If given, truncate the hex string to this many characters.
Returns:
str: The full (or truncated) hex representation of the digest.
to_int
Section titled “to_int”method · orcapod.types.ContentHash.to_int
def to_int(hexdigits: int | None = None) -> intConvert the digest to an integer.
Parameters:
- hexdigits (
int | None) defaultNone: Number of hex digits to use. If provided, the hex string is truncated before conversion.
Returns:
int: Integer representation of the (optionally truncated) digest.
to_uuid
Section titled “to_uuid”method · orcapod.types.ContentHash.to_uuid
def to_uuid(namespace: uuid.UUID = uuid.NAMESPACE_OID) -> uuid.UUIDDerive a deterministic UUID from the digest.
Uses uuid5 with the full hex string to ensure deterministic output.
Parameters:
- namespace (
uuid.UUID) defaultuuid.NAMESPACE_OID: UUID namespace foruuid5generation. Defaults touuid.NAMESPACE_OID.
Returns:
uuid.UUID: A UUID derived from this hash.
to_base64
Section titled “to_base64”method · orcapod.types.ContentHash.to_base64
def to_base64() -> strConvert the digest to a base64-encoded ASCII string.
Returns:
str: Base64 representation of the digest.
to_string
Section titled “to_string”method · orcapod.types.ContentHash.to_string
def to_string(prefix_method: bool = True, hexdigits: int | None = None) -> strConvert the digest to a human-readable string.
Parameters:
- prefix_method (
bool) defaultTrue: IfTrue(the default), prepend the method name followed by a colon (e.g."sha256:abcd1234"). - hexdigits (
int | None) defaultNone: Optional number of hex digits to include.
Returns:
str: String representation of the hash.
to_prefixed_digest
Section titled “to_prefixed_digest”method · orcapod.types.ContentHash.to_prefixed_digest
def to_prefixed_digest() -> bytesReturn the hash as method-prefixed raw bytes: b"{method}:{digest}".
Encodes the method name as ASCII, appends a literal : byte, then
appends the raw digest bytes. The result is suitable for storage in a
pa.large_binary() column and preserves the hash method for
introspection without the hex-encoding overhead of to_string().
Example::
h = ContentHash("arrow_v2.1", b"\x00" * 32)h.to_prefixed_digest()# b"arrow_v2.1:\x00\x00...\x00"Returns:
bytes: Raw bytes in the formb"{method}:{raw_digest}".
from_prefixed_digest
Section titled “from_prefixed_digest”classmethod · orcapod.types.ContentHash.from_prefixed_digest
def from_prefixed_digest(data: bytes) -> 'ContentHash'Parse method-prefixed raw bytes back into a ContentHash.
Inverse of to_prefixed_digest().
Parameters:
- data (
bytes): Bytes in the formb"{method}:{raw_digest}", as produced byto_prefixed_digest().
Returns:
'ContentHash': A newContentHashinstance.
Raises:
ValueError: Ifdatadoes not contain a colon separator.
from_string
Section titled “from_string”classmethod · orcapod.types.ContentHash.from_string
def from_string(hash_string: str) -> 'ContentHash'Parse a "method:hex_digest" string into a ContentHash.
Parameters:
- hash_string (
str): A string in the format"method:hex_digest".
Returns:
'ContentHash': A newContentHashinstance.
display_name
Section titled “display_name”method · orcapod.types.ContentHash.display_name
def display_name(length: int = 8) -> strReturn a short, human-friendly label for this hash.
Parameters:
- length (
int) default8: Number of hex characters to include after the method prefix. Defaults to 8.
Returns:
str: A string like"arrow_v2.1:1a2b3c4d".