ExArrow.FileSystem.Memory (ex_arrow v0.9.0)

View Source

In-memory filesystem for Dataset discovery tests.

Holds a flat map of normalized absolute paths to entries. Adding a file also registers parent directories so list/3 can walk a Hive-style tree without touching the OS.

Fields

  • :entries — %{path => ExArrow.FileSystem.entry()}

Examples

iex> {:ok, fs} = ExArrow.FileSystem.Memory.new(%{"/data/a.parquet" => 128})
iex> ExArrow.FileSystem.exists?(fs, "/data/a.parquet")
true

iex> fs = ExArrow.FileSystem.Memory.new()
iex> {:ok, fs} = ExArrow.FileSystem.Memory.put_file(fs, "/data/year=2026/part.parquet", size: 64)
iex> {:ok, paths} = ExArrow.FileSystem.glob(fs, "/data/**/*.parquet")
iex> paths
["/data/year=2026/part.parquet"]

Summary

Types

entry_map()

Internal path => entry map.

t()

Memory filesystem handle.

Functions

new()

Build an empty memory filesystem.

new(seed)

Build a memory filesystem from a path → size map or {path, size} list.

put_file(fs, path, opts \\ [])

Register a file at path with size (default 0).

Types

entry_map()

@type entry_map() :: %{optional(String.t()) => ExArrow.FileSystem.entry()}

Internal path => entry map.

t()

@type t() :: %ExArrow.FileSystem.Memory{entries: entry_map()}

Memory filesystem handle.

Fields

Functions

new()

@spec new() :: t()

Build an empty memory filesystem.

Examples

iex> %ExArrow.FileSystem.Memory{entries: %{}} = ExArrow.FileSystem.Memory.new()

new(seed)

@spec new(map() | [{String.t(), non_neg_integer()}]) ::
  {:ok, t()} | {:error, String.t()}

Build a memory filesystem from a path → size map or {path, size} list.

Parameters

  • seed — %{path => size} or [{path, size}, ...] where size is a non-negative integer (file byte size)

Returns

  • {:ok, fs} on success
  • {:error, message} for invalid paths or sizes

Examples

iex> {:ok, fs} = ExArrow.FileSystem.Memory.new(%{"/data/a.parquet" => 10})
iex> ExArrow.FileSystem.exists?(fs, "/data")
true

put_file(fs, path, opts \\ [])

@spec put_file(t(), String.t(), keyword()) :: {:ok, t()} | {:error, String.t()}

Register a file at path with size (default 0).

Creates missing parent directories. Returns {:ok, fs} or {:error, msg}.

Parameters

  • fs — memory filesystem

  • path — absolute-style path string (normalized with a leading /)

  • opts:

    • :size — non-negative integer byte size (default 0)

Examples

iex> fs = ExArrow.FileSystem.Memory.new()
iex> {:ok, fs} = ExArrow.FileSystem.Memory.put_file(fs, "/data/x.parquet", size: 32)
iex> ExArrow.FileSystem.exists?(fs, "/data/x.parquet")
true