ExArrow.Dataset.Fragment (ex_arrow v0.9.0)

One readable unit in a Dataset: a file path plus optional Hive partition values discovered from the path.

Fragments are produced by ExArrow.Dataset.open/2. They describe what can be scanned; they do not hold open file handles. Scanning opens each path on demand via ExArrow.Scanner.

Example

{:ok, dataset} =
  ExArrow.Dataset.open("/data/events",
    partitioning: {:hive, schema: [{"year", :int32}]}
  )

[frag | _] = ExArrow.Dataset.fragments(dataset)
frag.path
frag.partition_values
# => %{"year" => 2026}

{:ok, meta} = ExArrow.Dataset.Fragment.metadata(frag)
meta.num_rows

Summary

Types

format()

On-disk format of this fragment (matches the parent Dataset).

t()

A discovered fragment.

Functions

metadata(fragment)

Read Parquet footer metadata for this fragment without decoding data pages.

Types

format()

@type format() :: :parquet | :ipc

On-disk format of this fragment (matches the parent Dataset).

t()

@type t() :: %ExArrow.Dataset.Fragment{
  format: format(),
  partition_values: %{optional(String.t()) => term()},
  path: String.t(),
  size: non_neg_integer()
}

A discovered fragment.

Fields

:path — absolute or dataset-relative file path (Local discovery typically expands to an absolute path)
:format — :parquet or :ipc
:partition_values — map of Hive column name => coerced term (empty map when partitioning is :none)
:size — file size in bytes as reported by the filesystem. Current backends (Local, Memory) always resolve a real size; a future object-store backend may report 0 when size is unavailable without an extra round-trip

Functions

metadata(fragment)

@spec metadata(t()) :: {:ok, ExArrow.Parquet.Metadata.t()} | {:error, String.t()}

Read Parquet footer metadata for this fragment without decoding data pages.

Parameters

fragment — must have format: :parquet and an OS-readable :path

Returns

{:ok, %ExArrow.Parquet.Metadata{}} with row-group and column stats
{:error, message} for IPC fragments, missing files, or read failures

Examples

{:ok, meta} = ExArrow.Dataset.Fragment.metadata(frag)
meta.num_row_groups
meta.num_rows