ExArrow.Compute.Expression (ex_arrow v0.9.0)

View Source

Analyzable compute expression AST for filters and Dataset scanners.

Builders are the canonical API for 0.9. Macro sugar (expr do ... end) is out of scope. Expressions are data: they can be validated against a schema, printed for diagnostics, partially compiled to the Parquet filter tuple AST (v0.8.0), and evaluated as residuals via ExArrow.Compute.filter/2. They are not Elixir closures.

Typical usage

alias ExArrow.Compute.Expression, as: E

filter =
  E.and_(
    E.gte(E.field("year"), E.scalar(2026)),
    E.ne(E.field("amount"), E.scalar(0))
  )

{:ok, ^filter} = E.validate(filter, schema)
{pushed, residual} = E.to_parquet_filters(filter)

Use Expressions with ExArrow.Dataset.scanner/2 so the Scanner can prune partitions, push Parquet filters, and apply residuals.

Examples

iex> alias ExArrow.Compute.Expression, as: E
iex> to_string(E.eq(E.field("id"), E.scalar(1)))
"eq(field(\"id\"), scalar(1))"

iex> alias ExArrow.Compute.Expression, as: E
iex> E.expression?(E.field("x"))
true

Summary

Types

expr_node()

Internal AST node.

op()

Comparison and boolean operators in the AST.

scalar()

Literal values accepted by scalar/1.

t()

An expression tree.

Functions

and_(l, r)

Boolean AND of two expressions.

eq(l, r)

Equality comparison (left == right).

expression?(arg1)

Returns true if term is an ExArrow.Compute.Expression.

field(name)

Reference a column by name.

gt(l, r)

Greater-than comparison (left > right).

gte(l, r)

Greater-than-or-equal comparison (left >= right).

lt(l, r)

Less-than comparison (left < right).

lte(l, r)

Less-than-or-equal comparison (left <= right).

ne(l, r)

Inequality comparison (left != right).

not_(e)

Boolean NOT.

or_(l, r)

Boolean OR of two expressions.

scalar(d)

A scalar literal.

to_parquet_filters(expression)

Split expr into a Parquet-pushable filter AST and an optional residual expression.

to_string(expression)

Render expr as a diagnostic string.

validate(expr, fields)

Type-check expr against a schema or a field-name map.

Types

expr_node()

@type expr_node() ::
  {:field, String.t()} | {:scalar, scalar()} | {:call, op(), [expr_node()]}

Internal AST node.

  • {:field, name} — column reference
  • {:scalar, value} — literal (scalar/0)
  • {:call, op, args} — comparison or boolean operator

op()

@type op() :: :eq | :ne | :gt | :gte | :lt | :lte | :and | :or | :not

Comparison and boolean operators in the AST.

scalar()

@type scalar() ::
  integer()
  | float()
  | boolean()
  | String.t()
  | Date.t()
  | NaiveDateTime.t()
  | DateTime.t()

Literal values accepted by scalar/1.

Temporal values (Date, NaiveDateTime, DateTime) validate against date/timestamp columns but are residual for Parquet pushdown in 0.9 (they are not bound into the v0.8 filter tuple AST yet).

t()

@type t() :: %ExArrow.Compute.Expression{node: expr_node()}

An expression tree.

Fields

Functions

and_(l, r)

@spec and_(t(), t()) :: t()

Boolean AND of two expressions.

Named and_/2 because and/2 is a Kernel special form.

Examples

iex> alias ExArrow.Compute.Expression, as: E
iex> expr = E.and_(E.gt(E.field("a"), E.scalar(0)), E.lt(E.field("a"), E.scalar(10)))
iex> to_string(expr) =~ "and_("
true

eq(l, r)

@spec eq(t(), t()) :: t()

Equality comparison (left == right).

Parameters

  • left, right — field or scalar expressions

Examples

iex> alias ExArrow.Compute.Expression, as: E
iex> to_string(E.eq(E.field("ok"), E.scalar(true)))
"eq(field(\"ok\"), scalar(true))"

expression?(arg1)

@spec expression?(term()) :: boolean()

Returns true if term is an ExArrow.Compute.Expression.

Examples

iex> ExArrow.Compute.Expression.expression?(ExArrow.Compute.Expression.field("x"))
true

iex> ExArrow.Compute.Expression.expression?(:nope)
false

field(name)

@spec field(String.t() | atom()) :: t()

Reference a column by name.

Parameters

Examples

iex> alias ExArrow.Compute.Expression, as: E
iex> to_string(E.field("amount"))
"field(\"amount\")"

iex> alias ExArrow.Compute.Expression, as: E
iex> E.field(:amount) == E.field("amount")
true

gt(l, r)

@spec gt(t(), t()) :: t()

Greater-than comparison (left > right).

Examples

iex> alias ExArrow.Compute.Expression, as: E
iex> to_string(E.gt(E.field("score"), E.scalar(0.9)))
"gt(field(\"score\"), scalar(0.9))"

gte(l, r)

@spec gte(t(), t()) :: t()

Greater-than-or-equal comparison (left >= right).

lt(l, r)

@spec lt(t(), t()) :: t()

Less-than comparison (left < right).

lte(l, r)

@spec lte(t(), t()) :: t()

Less-than-or-equal comparison (left <= right).

ne(l, r)

@spec ne(t(), t()) :: t()

Inequality comparison (left != right).

Examples

iex> alias ExArrow.Compute.Expression, as: E
iex> to_string(E.ne(E.field("amount"), E.scalar(0)))
"ne(field(\"amount\"), scalar(0))"

not_(e)

@spec not_(t()) :: t()

Boolean NOT.

Named not_/1 because not/1 is a Kernel special form.

Always residual for Parquet pushdown (the v0.8 filter AST has no NOT).

Examples

iex> alias ExArrow.Compute.Expression, as: E
iex> {nil, residual} = E.to_parquet_filters(E.not_(E.eq(E.field("ok"), E.scalar(true))))
iex> to_string(residual) =~ "not_("
true

or_(l, r)

@spec or_(t(), t()) :: t()

Boolean OR of two expressions.

Named or_/2 because or/2 is a Kernel special form.

scalar(d)

@spec scalar(scalar()) :: t()

A scalar literal.

Parameters

Raises ArgumentError for invalid UTF-8 or unsupported terms.

Examples

iex> alias ExArrow.Compute.Expression, as: E
iex> to_string(E.scalar(42))
"scalar(42)"

iex> alias ExArrow.Compute.Expression, as: E
iex> to_string(E.scalar(true))
"scalar(true)"

to_parquet_filters(expression)

@spec to_parquet_filters(t()) :: {term() | nil, t() | nil}

Split expr into a Parquet-pushable filter AST and an optional residual expression.

Parameters

  • expr — expression to split

Returns

{pushed, residual} where:

  • pushed is nil or a v0.8.0 filter tuple ({:eq|:ne|:gt|:gte|:lt|:lte, col, value} / {:and|:or, [...]})
  • residual is nil or an Expression that still needs post-decode evaluation (e.g. not_/1, temporal scalars the Parquet reader cannot bind yet, field-vs-field comparisons)

AND may push one side and residual the other. OR is pushed only when both sides are fully pushable; otherwise the whole OR is residual.

Note: ExArrow.Parquet.Reader :filters accepts an Expression only when it is fully pushable (residual is nil). For mixed pushable/residual filters, use ExArrow.Dataset.scanner/2.

Examples

iex> alias ExArrow.Compute.Expression, as: E
iex> E.to_parquet_filters(E.gt(E.field("score"), E.scalar(0.9)))
{{:gt, "score", 0.9}, nil}

iex> alias ExArrow.Compute.Expression, as: E
iex> {pushed, residual} =
...>   E.to_parquet_filters(
...>     E.and_(E.gt(E.field("amount"), E.scalar(0)), E.gte(E.field("day"), E.scalar(~D[2026-01-01])))
...>   )
iex> pushed
{:gt, "amount", 0}
iex> match?(%E{}, residual)
true

to_string(expression)

@spec to_string(t()) :: String.t()

Render expr as a diagnostic string.

Examples

iex> alias ExArrow.Compute.Expression, as: E
iex> ExArrow.Compute.Expression.to_string(E.lt(E.field("x"), E.scalar(3)))
"lt(field(\"x\"), scalar(3))"

validate(expr, fields)

@spec validate(t(), ExArrow.Schema.t() | %{optional(String.t()) => term()}) ::
  {:ok, t()} | {:error, String.t()}

Type-check expr against a schema or a field-name map.

Checks that field names exist and that comparisons are type-compatible with the referenced column (and the other side, when both are fields).

Parameters

  • expr — expression to validate

  • schema_or_fields — either:

    • an ExArrow.Schema.t(), or
    • a %{String.t() => type_atom} map (useful when merging Hive partition types into the file schema for Scanner validation)

Returns

  • {:ok, expr} when valid
  • {:error, message} for unknown fields or type mismatches

Examples

{:ok, batch} = ExArrow.RecordBatch.from_lists([{"amount", :s64, [1]}])
schema = ExArrow.RecordBatch.schema(batch)
alias ExArrow.Compute.Expression, as: E
{:ok, _} = E.validate(E.gt(E.field("amount"), E.scalar(0)), schema)

# Partition keys for Dataset.scanner/2:
fields = Map.merge(%{"amount" => :int64}, %{"year" => :int32})
{:ok, _} = E.validate(E.gte(E.field("year"), E.scalar(2026)), fields)