# `Slap.SlateDB`
[🔗](https://github.com/mjrusso/slap/blob/slap_slatedb-v0.2.0/slap_slatedb/lib/slap/slatedb.ex#L1)

Elixir bindings for [SlateDB](https://slatedb.io), an embedded key-value
store that keeps its data in object storage.

    {:ok, db} = Slap.SlateDB.open("my-db", store: {:local, "/tmp/slatedb"})
    {:ok, _seq} = Slap.SlateDB.put(db, "hello", "world")
    {:ok, "world"} = Slap.SlateDB.get(db, "hello")
    [{"hello", "world"}] = db |> Slap.SlateDB.scan(prefix: "hel") |> Enum.to_list()
    :ok = Slap.SlateDB.close(db)

Keys and values are binaries. Keys cannot be empty.
Elixir-side argument and option checks raise. The NIF reports invalid
operations or store settings as `{:error, %Slap.SlateDB.Error{kind: :invalid}}`;
other database and object-store failures also return an error tuple.

## How calls run

SlateDB does its I/O on a Tokio runtime inside the NIF library. A call does
not block a BEAM scheduler: the NIF starts the work and returns, and the
calling process waits in `receive` for the result. So many processes can
use one database at the same time, and a slow object store only makes the
waiting processes slow.

Set `config :slap_slatedb, runtime_threads: n` in `config/runtime.exs` or
earlier, or set `SLAP_SLATEDB_RUNTIME_THREADS=n` before starting the VM.
Application config takes precedence when it is not `nil`. `System.put_env/2` in
`config/runtime.exs` works too; your application's `start/2` runs too late.
Invalid values fail application startup. The default is one per CPU.

## Durability

By default a write returns when it is in memory and visible to reads. SlateDB
writes it to object storage within the `flush_interval` setting (100 ms by
default). Every write returns `{:ok, seq}`, its sequence number. A write is
durable once `durable_seq/1` is at least `seq`. To find out:

  * pass `await_durable: true` to the write, which waits before returning,
  * call `flush/1`, or
  * `subscribe/3` to get a message each time the durable sequence number
    goes up. This lets one process write without waiting and confirm many
    writes at once.

## Timeouts

Calls wait for their reply with no timeout by default. Pass
`timeout: ms` to a call to get
`{:error, %Slap.SlateDB.Error{kind: :timeout}}` after `ms` milliseconds.
A timeout does not cancel the operation: a write that timed out may still be
applied. A reply that comes later is dropped. A call with a timeout runs
through a short-lived process, so it costs a little more than one without.

## Logs and metrics

SlateDB's log records go to `Logger` with a `:slatedb_target` metadata key.
Only records at `config :slap_slatedb, log_level: level` or above are sent;
the default is `:warning`. Change it at runtime with `set_log_level/1`.
`stats/1` and `metrics/1` report on a database, and `Slap.SlateDB.Telemetry`
emits its durability progress, stats and close as `:telemetry` events.

## Write ordering

Each call waits for its reply, and SlateDB assigns the sequence number
before the reply is sent. So the writes of one process are applied in the
order the process makes them, and each gets a higher `seq` than the one
before. Writes from different processes at the same time have no defined
order between them. Use `write/3` or a transaction when several writes must
be applied together.

This holds only for calls that got a reply. A write that timed out (see
"Timeouts") is still running and may be applied after the process's next
write. Code that depends on write order should not pass `:timeout` to
writes, and should treat any write error as the end of the handle: stop,
reopen and reload state from storage.

# `key_value`

```elixir
@type key_value() :: %{
  key: binary(),
  value: binary(),
  seq: non_neg_integer() | nil,
  create_ts: non_neg_integer(),
  expire_ts: non_neg_integer() | nil
}
```

A key, value and metadata returned by `get_key_value/3`.

# `store`

```elixir
@type store() ::
  :memory
  | :memory_ignoring_preconditions
  | {:local, Path.t()}
  | {:url, String.t()}
  | {:url, String.t(), [{atom() | String.t(), String.t()}]}
```

# `t`

```elixir
@opaque t()
```

# `write_op`

```elixir
@type write_op() ::
  {:put, binary(), binary()}
  | {:put, binary(), binary(), non_neg_integer()}
  | {:merge, binary(), binary()}
  | {:merge, binary(), binary(), non_neg_integer()}
  | {:delete, binary()}
```

# `begin`

```elixir
@spec begin(t(), keyword()) ::
  {:ok, Slap.SlateDB.Transaction.t()} | {:error, Slap.SlateDB.Error.t()}
```

Starts a transaction. See `Slap.SlateDB.Transaction`.

`:isolation` is `:snapshot` (the default), which detects write-write
conflicts, or `:serializable`, which also detects read-write conflicts.

# `close`

```elixir
@spec close(t(), keyword()) :: :ok | {:error, Slap.SlateDB.Error.t()}
```

Flushes buffered writes and closes the database.

Close first waits for calls already in flight on this database to finish.
Calls made after close starts fail with
`{:error, %Slap.SlateDB.Error{kind: :closed, reason: :clean}}`, including calls
on snapshots, transactions and iterators of this database. Closing again
returns `:ok`.

If you never call `close/1`, the database is closed in the background once
it and everything opened from it (snapshots, transactions, iterators) have
been garbage collected, and a warning is logged. Do not rely on that for
shutdown.

After a fence (see `open/2`), close returns `:ok` if the handle already
knew it was fenced, for example from a failed write or a
`{:slap_slatedb_closed, ref, tag, :fenced}` message. If close is what discovers the
fence, it returns `{:error, %Slap.SlateDB.Error{kind: :closed, reason: :fenced}}`:
writes that were not yet durable are lost. Either way the handle is closed.

# `create_checkpoint`

```elixir
@spec create_checkpoint(t(), keyword()) ::
  {:ok, %{id: String.t(), manifest_id: non_neg_integer()}}
  | {:error, Slap.SlateDB.Error.t()}
```

Creates a checkpoint: a named, durable view of the database that garbage
collection will not remove. Returns `{:ok, %{id: id, manifest_id: id}}`.

Open a `Slap.SlateDB.Reader` at the checkpoint, or clone the database from it
with `Slap.SlateDB.Admin.clone/3`. Manage checkpoints with `Slap.SlateDB.Admin`.

## Options

  * `:scope` - `:durable` (default) covers what is durable now. `:all`
    first flushes writes still in memory, so it covers every write so far.
  * `:lifetime` - milliseconds until the checkpoint expires. By default it
    never expires.
  * `:name` - a name, for `Slap.SlateDB.Admin.list_checkpoints/2`.
  * `:source` - the id of an existing checkpoint to copy.
  * `:timeout` - see "Timeouts" above.

# `delete`

```elixir
@spec delete(t(), binary(), keyword()) ::
  {:ok, non_neg_integer()} | {:error, Slap.SlateDB.Error.t()}
```

Deletes `key`. Returns `{:ok, seq}`. Takes the `:await_durable` option of
`put/4`.

# `durability_lag`

```elixir
@spec durability_lag(t()) :: non_neg_integer()
```

Returns `last_write_seq/1 - durable_seq/1`, or 0: how many sequence numbers
written through this handle are not durable yet. Unlike `stats/1`, it does
not collect metrics, so it is cheap enough to sample often.

# `durable_seq`

```elixir
@spec durable_seq(t()) :: non_neg_integer()
```

Returns the highest sequence number that is durable in object storage.

A write is durable when its `seq` is at most this value.

# `flush`

```elixir
@spec flush(t(), keyword()) :: :ok | {:error, Slap.SlateDB.Error.t()}
```

Writes buffered writes to object storage.

## Options

  * `:type` - `:wal` (default) writes the WAL, which makes all writes so
    far durable. `:memtable` also writes the memtable out as an L0 SST.
  * `:timeout` - see "Timeouts" above.

# `get`

```elixir
@spec get(t(), binary(), keyword()) ::
  {:ok, binary() | nil} | {:error, Slap.SlateDB.Error.t()}
```

Reads the value for `key`. Returns `{:ok, nil}` when the key is not found.

## Options

  * `:durability` - `:memory` (default) reads the latest write, even if it
    is not durable yet. `:remote` reads only data that is durable in object
    storage.
  * `:dirty` - when `true`, also reads writes that are not yet committed
    to the WAL. Defaults to `false`.
  * `:cache_blocks` - whether blocks read from object storage go into the
    block cache. Defaults to `true`.
  * `:timeout` - see "Timeouts" above.

# `get_key_value`

```elixir
@spec get_key_value(t(), binary(), keyword()) ::
  {:ok, key_value() | nil} | {:error, Slap.SlateDB.Error.t()}
```

Reads the current row for `key` with its metadata.

Returns `{:ok, nil}` when the key is not found, or `{:ok, row}` where `row`
is a map with these keys:

  * `:key` and `:value`
  * `:seq` - the sequence number of the write
  * `:create_ts` - when the row was written, in Unix milliseconds
  * `:expire_ts` - when the row expires, in Unix milliseconds, or `nil`

Takes the options of `get/3`.

# `increment`

```elixir
@spec increment(t(), binary()) ::
  {:ok, non_neg_integer()} | {:error, Slap.SlateDB.Error.t()}
```

Adds `by` to the counter at `key`, without reading it first.

The database must be opened with `merge_operator: :u64_add` or `:i64_add`.
Read the counter with `get/3` and `Slap.SlateDB.MergeOperator.decode_u64/1` or
`decode_i64/1`. Takes the options of `merge/4`. With no `by`, the increment
is 1; options can be passed as the third argument.

# `increment`

```elixir
@spec increment(t(), binary(), integer() | keyword()) ::
  {:ok, non_neg_integer()} | {:error, Slap.SlateDB.Error.t()}
```

# `increment`

```elixir
@spec increment(t(), binary(), integer(), keyword()) ::
  {:ok, non_neg_integer()} | {:error, Slap.SlateDB.Error.t()}
```

# `iterator`

```elixir
@spec iterator(t(), keyword()) ::
  {:ok, Slap.SlateDB.Iterator.t()} | {:error, Slap.SlateDB.Error.t()}
```

Opens a `Slap.SlateDB.Iterator`. Takes `scan/2`'s bounds, prefix and read
options. `:with_versions` fixes the row shape when the iterator opens;
`Slap.SlateDB.Iterator.next_batch/3` sets the fetch size.

# `last_write_seq`

```elixir
@spec last_write_seq(t()) :: non_neg_integer()
```

Returns the highest sequence number written through this handle (0 before
any write). Every write so far is durable once `durable_seq/1` reaches it.

# `merge`

```elixir
@spec merge(t(), binary(), binary(), keyword()) ::
  {:ok, non_neg_integer()} | {:error, Slap.SlateDB.Error.t()}
```

Writes a merge operand for `key`. Returns `{:ok, seq}`.

The database must be opened with a `:merge_operator`, which combines the
operand with the key's current value when the key is read or compacted.
See `Slap.SlateDB.MergeOperator` for the operators and their operand formats.
An operand in the wrong format is rejected with an `:invalid` error.

Takes the `:ttl`, `:await_durable` and `:timeout` options of `put/4`.

# `metrics`

```elixir
@spec metrics(t()) :: [%{name: String.t(), labels: map(), value: term()}]
```

Returns every metric SlateDB records for the database, as a list of
`%{name: name, labels: %{label => value}, value: value}`.

`value` is an integer for counters and gauges. For a histogram it is a map
with `:count`, `:sum`, `:min`, `:max`, `:boundaries` and `:bucket_counts`.
Names start with `slatedb.`, for example `"slatedb.db.write_ops"`.

# `open`

```elixir
@spec open(String.t(), keyword()) :: {:ok, t()} | {:error, Slap.SlateDB.Error.t()}
```

Opens the database at `path` in the given object store, or creates it.

Only one writer can have a database open at a time. When a second writer
opens it, the first one is fenced: its next write fails with a
`:closed` error with reason `:fenced`.

## Options

  * `:store` - where the data lives. Required. One of:
    * `:memory` - an in-memory store, lost when the VM exits. Good for tests.
    * `{:local, dir}` - a directory on the local file system.
    * `{:url, url}` or `{:url, url, options}` - an object storage URL:
      `"s3://bucket/prefix"` (Amazon S3 and compatible stores),
      `"az://container/prefix"` (Azure Blob Storage; also `abfs://` and
      `azure://`) or `"gs://bucket/prefix"` (Google Cloud Storage).
      Configuration comes from environment variables (`AWS_*`, `AZURE_*`
      or `GOOGLE_*`, such as `AWS_REGION`, `AZURE_STORAGE_ACCOUNT_NAME`
      or `GOOGLE_SERVICE_ACCOUNT`), then from `options`, which win. For
      example `[aws_region: "us-east-1", aws_endpoint: "http://localhost:9000"]`.
      An unknown option key is an error.

    SlateDB uses conditional puts to fence other writers. S3 stores always
    use ETag conditional puts, and a `conditional_put` setting other than
    `"etag"`, in the environment or in `options`, is an error. Azure and
    GCS support conditional puts without configuration.
  * `:settings` - a map of SlateDB settings, merged over the defaults. The
    keys match SlateDB's JSON settings format, for example
    `%{flush_interval: "50ms", default_ttl_millis: 60_000}`. To keep a
    local disk cache of object storage files, set
    `%{object_store_cache_options: %{root_folder: "/var/cache/slatedb"}}`.
  * `:cache` - the in-memory block and metadata cache. By default each
    database has its own. Pass a `Slap.SlateDB.Cache` to share one between
    databases, or `:disabled` to turn it off.
  * `:merge_operator` - a built-in merge operator for `merge/4`. See
    `Slap.SlateDB.MergeOperator`. Every process that opens the database,
    including readers, must use the same one.
  * `:compaction_filter` - a `Slap.SlateDB.CompactionFilter` that deletes keys
    under a set of prefixes as they are compacted. Needs the compactor,
    which is on by default.
  * `:timeout` - see "Timeouts" above.

# `probe_store`

```elixir
@spec probe_store(store(), String.t(), keyword()) ::
  {:ok, [{atom(), term()}]}
  | {:error, {:probe_failed, [{atom(), term()}]} | Slap.SlateDB.Error.t()}
```

Checks that a store honours conditional writes, which SlateDB's writer
fencing depends on. Run it before opening databases on a store you have
not checked, such as a new S3-compatible server.

It works on a new object at `path` (under the store's URL prefix, if any)
in five steps:

  * `:create` - a create-if-absent PUT succeeds.
  * `:create_again` - a second create-if-absent PUT is refused.
  * `:stale_if_match` - a PUT with `If-Match` and a stale ETag is refused.
  * `:current_if_match` - a PUT with `If-Match` and the current ETag
    succeeds.
  * `:delete` - the object is deleted.

Each step is `:ok`, `{:failed, message}`, `:skipped` (after an earlier
failure) or, for the two `If-Match` steps, `:unsupported`: SlateDB only
needs create-if-absent, and some stores (the local file system) do not
implement `If-Match`. A store that accepts a stale ETag fails.

Returns `{:ok, steps}` when nothing failed, `{:error, {:probe_failed, steps}}` when a step
failed, or `{:error, %Slap.SlateDB.Error{}}` if the store cannot be reached.
`{:probe_failed, steps}` means the probe ran and found a failed condition;
`%Slap.SlateDB.Error{}` means a store operation failed.

The `:memory_ignoring_preconditions` store, an in-memory store that treats
every conditional PUT as an overwrite, exists only to test code that runs
this probe. Do not open databases on it.

Takes the `:timeout` option.

# `put`

```elixir
@spec put(t(), binary(), binary(), keyword()) ::
  {:ok, non_neg_integer()} | {:error, Slap.SlateDB.Error.t()}
```

Writes `value` for `key`. Returns `{:ok, seq}`, the write's sequence number.

## Options

  * `:ttl` - time to live in milliseconds. By default, the `default_ttl_millis`
    setting applies (no expiry unless you set it). SlateDB removes expired
    rows during compaction, so a read can still return a row for a while
    after it expires. Check `:expire_ts` from `get_key_value/2` if you need
    an exact cutoff.
  * `:await_durable` - when `true`, returns only after the write is durable
    in object storage. Defaults to `false`.
  * `:timeout` - see "Timeouts" above.

Values of 64 KiB or more are passed to SlateDB without copying.

# `scan`

```elixir
@spec scan(t(), keyword()) :: Enumerable.t()
```

Returns a lazy stream of `{key, value}` tuples in key order. With
`with_versions: true`, each row is `{key, value, version}`.

The scan starts when the stream is run. Each run sees the data as it is at
that time. Enumeration raises `Slap.SlateDB.Error` if a read fails.

## Options

  * `:gte` / `:gt` - lower bound, inclusive or exclusive.
  * `:lte` / `:lt` - upper bound, inclusive or exclusive.
  * `:prefix` - only keys that start with this binary. Can be combined with
    the bounds above.
  * `:batch_size` - rows fetched per native call. Defaults to 256.
  * `:with_versions` - include each row's sequence number as its version.
  * `:order` - `:asc` (default) or `:desc`.
  * `:durability`, `:dirty` - as for `get/3`.
  * `:cache_blocks` - whether blocks read by the scan go into the block
    cache. Defaults to `false`, so a large scan does not push out hot blocks.
  * `:read_ahead_bytes` - how many bytes to fetch ahead from object storage.
  * `:max_fetch_tasks` - how many fetches to run at the same time.
  * `:timeout` - applies to opening the scan and to each batch fetch.

## Examples

    Slap.SlateDB.scan(db, gte: "user:", lt: "user;") |> Enum.take(10)
    Slap.SlateDB.scan(db, prefix: "user:") |> Stream.map(&elem(&1, 0)) |> Enum.to_list()

# `set_log_level`

```elixir
@spec set_log_level(:debug | :info | :warning | :error | :none) :: :ok
```

Sets the lowest level of SlateDB log records sent to `Logger`: `:debug`,
`:info`, `:warning`, `:error`, or `:none` to send none.

# `snapshot`

```elixir
@spec snapshot(t(), keyword()) ::
  {:ok, Slap.SlateDB.Snapshot.t()} | {:error, Slap.SlateDB.Error.t()}
```

Takes a consistent, read-only snapshot. See `Slap.SlateDB.Snapshot`.

# `stats`

```elixir
@spec stats(t()) :: %{required(atom()) => non_neg_integer()}
```

Returns a summary of the database's state:

  * `:durable_seq` - see `durable_seq/1`.
  * `:last_write_seq` - the highest `seq` written through this handle.
  * `:durability_lag` - see `durability_lag/1`.
  * `:l0_sst_count` and `:sorted_run_count` - the shape of the LSM tree. A
    growing L0 count means compaction is falling behind.
  * `:cache_hits` and `:cache_misses` - block cache lookups, over all entry
    kinds, since the database opened.

It works after `close/1` too.

# `subscribe`

```elixir
@spec subscribe(t(), term(), pid()) :: {:ok, Slap.SlateDB.Subscription.t()}
```

Subscribes `pid` to durability and close events.

`pid` gets `{:slap_slatedb_durable, ref, tag, durable_seq}` straight away
and then each time the durable sequence number goes up. Updates are
coalesced: a busy process gets the latest value, not every step.

When the database closes, `pid` gets `{:slap_slatedb_closed, ref, tag,
reason}` once, where `reason` is `:clean`, `:fenced`, `:panic` or
`:unknown`, and the subscription ends. A fenced writer finds out that it is
fenced when it next writes or flushes, or when it next reads the manifest
(every `manifest_poll_interval`).

`ref` is the returned subscription's `:ref`, unique to this subscription;
`tag` is the caller's, and several subscriptions may share it.

The subscription also ends when `pid` exits (it is monitored) or on
`unsubscribe/2`. It does not end when the returned value is garbage
collected.

# `transaction`

```elixir
@spec transaction(
  t(),
  (Slap.SlateDB.Transaction.t() -> {:ok, any()} | {:error, any()}),
  keyword()
) ::
  {:ok, any()} | {:error, any()}
```

Runs `fun` in a transaction and commits it.

`fun` returns `{:ok, value}` to commit or `{:error, reason}` to roll back.
Returns `{:ok, value}` only after the commit is durable. A rollback returns
`{:error, reason}`. If `fun` raises, throws or exits, the transaction is
rolled back and the failure is propagated.

## Options

  * `:isolation` - `:snapshot` (default) or `:serializable`.
  * `:retries` - how many times to run `fun` again after a commit conflict.
    Defaults to `0`. `fun` must be safe to run more than once.
  * `:timeout` - applies to starting and to committing the transaction.

Use `begin/2` and `Slap.SlateDB.Transaction.commit/2` to commit without
waiting for durability and receive the commit's sequence number.

## Example

    Slap.SlateDB.transaction(db, fn tx ->
      {:ok, balance} = Slap.SlateDB.Transaction.get(tx, "balance")
      new_balance = String.to_integer(balance || "0") + 10
      :ok = Slap.SlateDB.Transaction.put(tx, "balance", Integer.to_string(new_balance))
      {:ok, new_balance}
    end, retries: 3)

# `unsubscribe`

```elixir
@spec unsubscribe(Slap.SlateDB.Subscription.t(), keyword()) :: :ok
```

Ends a subscription from `subscribe/3`. Ending it twice is allowed.

No message is sent once this returns, but messages sent before it may be
waiting in the subscriber's mailbox. With `flush: true`, called from the
subscriber, they are removed.

# `validate_settings`

```elixir
@spec validate_settings(map()) :: :ok | {:error, Slap.SlateDB.Error.t()}
```

Checks a `:settings` map with the same parser used by `open/2`, without
opening a database. Returns `:ok` or an `:invalid` error.

# `validate_store!`

```elixir
@spec validate_store!(store()) :: :ok
```

Checks a store specification with the same Elixir parser used by `open/2`,
without opening the store. Returns `:ok` or raises on a malformed
specification, as `open/2` does. Store connectivity is checked when opened.

# `write`

```elixir
@spec write(t(), [write_op()], keyword()) ::
  {:ok, non_neg_integer()} | {:error, Slap.SlateDB.Error.t()}
```

Applies a list of writes atomically: either all of them happen or none do.
Returns `{:ok, seq}`. All writes in the batch share that sequence number.

    Slap.SlateDB.write(db, [
      {:put, "a", "1"},
      {:put, "b", "2", 60_000},
      {:delete, "c"}
    ])

A `{:merge, key, operand}` needs a `:merge_operator` (see `merge/4`). A
four-element `:put` or `:merge` sets a TTL in milliseconds. Takes the
`:await_durable` and `:timeout` options of `put/4`.

A batch of more than 1000 operations is converted on a dirty
CPU scheduler, so it does not hold up a normal scheduler.

---

*Consult [api-reference.md](api-reference.md) for complete listing*