ExArrow.Dataset (ex_arrow v0.9.0)

View Source

Dataset discovery over Parquet (and IPC) files.

A Dataset is the result of finding files and describing them as fragments. It does not decode row groups. Use ExArrow.Scanner to project, filter, and stream batches.

Typical workflow

alias ExArrow.Compute.Expression, as: E

{:ok, dataset} =
  ExArrow.Dataset.open("/data/events",
    format: :parquet,
    partitioning: {:hive, schema: [{"year", :int32}, {"month", :int32}]}
  )

fragments = ExArrow.Dataset.fragments(dataset)
schema = ExArrow.Dataset.schema(dataset)

filter =
  E.and_(
    E.gte(E.field("year"), E.scalar(2026)),
    E.gt(E.field("amount"), E.scalar(0.0))
  )

{:ok, scanner} =
  ExArrow.Dataset.scanner(dataset, columns: ["id", "amount"], filter: filter)

{:ok, stream} = ExArrow.Scanner.to_stream(scanner)
batches = Enum.to_list(stream)
:ok = ExArrow.Stream.close(stream)

Sources for open/2

  • a directory path (recursive discovery of matching files)
  • a single file path
  • a glob pattern (* within a segment, ** across segments)
  • an explicit list of file paths

Options for open/2

  • :format — :parquet (default) or :ipc
  • :partitioning — :none (default) or {:hive, schema: [{name, type}, ...]} (see partition_schema/0)
  • :filesystem — ExArrow.FileSystem handle (default ExArrow.FileSystem.Local.new/0)
  • :ignore_hidden — skip path components whose basename starts with . or _ (default true)
  • :schema — optional ExArrow.Schema.t() to skip footer / IPC schema resolution (required for Memory-only discovery when paths are not OS-readable)
  • :root — dataset root used when parsing Hive relative paths (inferred from the source when omitted)

See also: guides/11_datasets.md, livebook/06_datasets.livemd.

Summary

Types

format()

On-disk format of every fragment in this dataset.

partition_schema()

Ordered list of {column_name, type} pairs for Hive partitioning.

partition_type()

Arrow-ish type atom used when coercing Hive key=value path segments.

partitioning()

How fragment paths contribute partition columns.

t()

A discovered Dataset.

Functions

fragments(dataset)

Return discovered fragments in lexicographic path order.

open(source, opts \\ [])

Discover fragments for source and resolve the dataset schema.

scanner(dataset, opts \\ [])

Build a lazy ExArrow.Scanner over this dataset.

schema(dataset)

Return the Arrow schema resolved at open time.

Types

format()

@type format() :: :parquet | :ipc

On-disk format of every fragment in this dataset.

partition_schema()

@type partition_schema() :: [{String.t(), partition_type()}]

Ordered list of {column_name, type} pairs for Hive partitioning.

Example: [{"year", :int32}, {"month", :int32}] matches paths like .../year=2026/month=01/part-0.parquet.

partition_type()

@type partition_type() ::
  :int8
  | :int16
  | :int32
  | :int64
  | :uint8
  | :uint16
  | :uint32
  | :uint64
  | :float32
  | :float64
  | :utf8
  | :boolean
  | :date32

Arrow-ish type atom used when coercing Hive key=value path segments.

Integers are range-checked for the named width. :date32 accepts ISO-8601 date strings. :utf8 URL-decodes the value. :boolean accepts true/false/1/0 (case-insensitive).

partitioning()

@type partitioning() :: :none | {:hive, partition_schema()}

How fragment paths contribute partition columns.

  • :none — no path parsing; every fragment has partition_values: %{}
  • {:hive, schema} — parse key=value segments under the dataset root using schema (see partition_schema/0)

t()

@type t() :: %ExArrow.Dataset{
  filesystem: ExArrow.FileSystem.t(),
  format: format(),
  fragments: [ExArrow.Dataset.Fragment.t()],
  ignore_hidden: boolean(),
  partitioning: partitioning(),
  root: String.t(),
  schema: ExArrow.Schema.t()
}

A discovered Dataset.

Fields

  • :format — :parquet or :ipc (from open/2)
  • :partitioning — :none or {:hive, schema} used at open time
  • :filesystem — discovery backend (Local or Memory)
  • :ignore_hidden — whether hidden path components were skipped
  • :root — root path for Hive relative parsing (often absolute on Local)
  • :fragments — path-sorted ExArrow.Dataset.Fragment list
  • :schema — Arrow schema from the first fragment footer / IPC metadata, or the caller-supplied :schema option

Functions

fragments(dataset)

@spec fragments(t()) :: [ExArrow.Dataset.Fragment.t()]

Return discovered fragments in lexicographic path order.

Parameters

  • dataset — an ExArrow.Dataset.t() from open/2

Examples

frags = ExArrow.Dataset.fragments(dataset)
Enum.map(frags, & &1.partition_values)
# => [%{"year" => 2025, "month" => 12}, %{"year" => 2026, "month" => 1}]

open(source, opts \\ [])

@spec open(
  String.t() | [String.t()],
  keyword()
) :: {:ok, t()} | {:error, String.t()}

Discover fragments for source and resolve the dataset schema.

Performs discovery IO (list/glob/exists) and, unless :schema is passed, opens the first fragment's footer (Parquet) or IPC file metadata. Does not decode data pages.

Parameters

  • source — directory, file path, glob string, or list of file paths
  • opts — see the module documentation (format, partitioning, filesystem, ignore_hidden, schema, root)

Returns

  • {:ok, dataset} on success
  • {:error, message} for validation failures, missing paths, empty discovery, malformed Hive segments, or schema resolution errors

Examples

Open a Hive-partitioned directory:

{:ok, dataset} =
  ExArrow.Dataset.open("/data/events",
    partitioning: {:hive, schema: [{"year", :int32}, {"month", :int32}]}
  )

Open an explicit file list with a known schema (no footer read):

{:ok, dataset} =
  ExArrow.Dataset.open(
    ["/data/a.parquet", "/data/b.parquet"],
    schema: schema,
    root: "/data"
  )

Discover via Memory filesystem (tests):

{:ok, fs} =
  ExArrow.FileSystem.Memory.new(%{
    "/data/year=2026/part-0.parquet" => 128
  })

{:ok, dataset} =
  ExArrow.Dataset.open("/data",
    filesystem: fs,
    schema: schema,
    partitioning: {:hive, schema: [{"year", :int32}]},
    root: "/data"
  )

scanner(dataset, opts \\ [])

@spec scanner(
  t(),
  keyword()
) :: {:ok, ExArrow.Scanner.t()} | {:error, String.t()}

Build a lazy ExArrow.Scanner over this dataset.

Performs no IO. Validation of :columns / :filter / :batch_size happens here; file opens start in ExArrow.Scanner.to_stream/1.

Parameters

  • dataset — discovered dataset

  • opts — scanner options:

    • :columns — non-empty list of column name strings to project, or omit for all columns
    • :filter — ExArrow.Compute.Expression.t(), legacy Parquet filter tuple ({:gt, "col", value}, {:and, [...]}, ...), or omit/nil
    • :batch_size — positive integer accepted for API stability; reserved in 0.9 (batches follow Parquet row-group sizing)

Returns

  • {:ok, scanner} when options validate
  • {:error, message} for unknown options, bad columns, or filter validation failures (including unknown Expression fields)

Examples

alias ExArrow.Compute.Expression, as: E

{:ok, scanner} =
  ExArrow.Dataset.scanner(dataset,
    columns: ["id"],
    filter: E.gte(E.field("year"), E.scalar(2026))
  )

{:ok, stream} = ExArrow.Scanner.to_stream(scanner)

schema(dataset)

@spec schema(t()) :: ExArrow.Schema.t()

Return the Arrow schema resolved at open time.

Comes from the first fragment's Parquet footer / IPC file metadata, or from the :schema option passed to open/2. No data pages are read.

Parameters

  • dataset — an ExArrow.Dataset.t() from open/2

Examples

schema = ExArrow.Dataset.schema(dataset)
ExArrow.Schema.field_names(schema)
# => ["id", "amount", "account_id"]