ExArrow.FileSystem behaviour (ex_arrow v0.9.0)

View Source

Capability-oriented filesystem abstraction for Dataset discovery.

Reads still go through path-based NIFs (Parquet, IPC). This module only answers discovery questions: what paths exist, which match a glob, and whether a path is present.

Implementations

S3 / object-store adapters are out of scope for 0.9.0; the behaviour leaves room for them later.

Hidden entries

When ignore_hidden: true (the default), any path component whose basename starts with . or _ is skipped. That matches Dataset's :ignore_hidden option (dotfiles and _-prefixed Hive / staging dirs).

Typical usage

fs = ExArrow.FileSystem.Local.new()
{:ok, entries} = ExArrow.FileSystem.list(fs, "/data/events", recursive: true)
{:ok, paths} = ExArrow.FileSystem.glob(fs, "/data/events/**/*.parquet")
true = ExArrow.FileSystem.exists?(fs, "/data/events")

{:ok, dataset} = ExArrow.Dataset.open("/data/events", filesystem: fs)

Summary

Types

entry()

One discovered path.

glob_opt()

Option for glob/3.

list_opt()

Option for list/3.

t()

Filesystem handle (struct whose module implements this behaviour).

Callbacks

exists?(t, t)
glob(t, t, keyword)
list(t, t, keyword)

Functions

exists?(fs, path)

Return whether path exists as a file or directory.

glob(fs, pattern, opts \\ [])

Return file paths matching pattern (sorted).

list(fs, path, opts \\ [])

List entries under path.

Types

entry()

@type entry() :: %{
  path: String.t(),
  type: :file | :directory,
  size: non_neg_integer()
}

One discovered path.

Keys

  • :path — absolute or normalized path string
  • :type — :file or :directory
  • :size — byte size for files; 0 for directories (and when unknown)

glob_opt()

@type glob_opt() :: {:ignore_hidden, boolean()}

Option for glob/3.

list_opt()

@type list_opt() :: {:recursive, boolean()} | {:ignore_hidden, boolean()}

Option for list/3.

t()

@type t() :: struct()

Filesystem handle (struct whose module implements this behaviour).

Callbacks

exists?(t, t)

@callback exists?(t(), String.t()) :: boolean()

glob(t, t, keyword)

@callback glob(t(), String.t(), keyword()) :: {:ok, [String.t()]} | {:error, String.t()}

list(t, t, keyword)

@callback list(t(), String.t(), keyword()) :: {:ok, [entry()]} | {:error, String.t()}

Functions

exists?(fs, path)

@spec exists?(t(), String.t()) :: boolean()

Return whether path exists as a file or directory.

Parameters

  • fs — filesystem handle
  • path — path string (non-binaries return false)

Examples

fs = ExArrow.FileSystem.Local.new()
ExArrow.FileSystem.exists?(fs, "/data/events")

glob(fs, pattern, opts \\ [])

@spec glob(t(), String.t(), [glob_opt()]) ::
  {:ok, [String.t()]} | {:error, String.t()}

Return file paths matching pattern (sorted).

Patterns use / separators. * matches within one path segment; ** matches across segments (including zero segments).

Parameters

  • fs — filesystem handle

  • pattern — glob string (for example "/data/**/*.parquet")

  • opts:

    • :ignore_hidden — when true (default), skip matches with a . / _-prefixed path component

Returns

  • {:ok, paths} — sorted list of matching file paths
  • {:error, message} — invalid pattern or opts

Examples

fs = ExArrow.FileSystem.Local.new()
{:ok, paths} = ExArrow.FileSystem.glob(fs, "/data/events/year=*/**/*.parquet")

list(fs, path, opts \\ [])

@spec list(t(), String.t(), [list_opt()]) :: {:ok, [entry()]} | {:error, String.t()}

List entries under path.

Parameters

  • fs — filesystem handle (Local or Memory)

  • path — directory or file to list

  • opts:

    • :recursive — when true (default), walk the whole tree; when false, only immediate children
    • :ignore_hidden — when true (default), skip . / _-prefixed names

Returns

  • {:ok, entries} — list of entry/0 maps, typically path-sorted
  • {:error, message} — missing path, invalid opts, or backend failure

Examples

fs = ExArrow.FileSystem.Local.new()
{:ok, entries} = ExArrow.FileSystem.list(fs, "/data/events", recursive: false)
Enum.map(entries, &{&1.type, &1.path})