@spec begin(t(), keyword()) ::
  {:ok, Slap.SlateDB.Transaction.t()} | {:error, Slap.SlateDB.Error.t()}

Starts a transaction. See Slap.SlateDB.Transaction.

:isolation is :snapshot (the default), which detects write-write conflicts, or :serializable, which also detects read-write conflicts.

@spec close(t(), keyword()) :: :ok | {:error, Slap.SlateDB.Error.t()}

Flushes buffered writes and closes the database.

Close first waits for calls already in flight on this database to finish. Calls made after close starts fail with {:error, %Slap.SlateDB.Error{kind: :closed, reason: :clean}}, including calls on snapshots, transactions and iterators of this database. Closing again returns :ok.

If you never call close/1, the database is closed in the background once it and everything opened from it (snapshots, transactions, iterators) have been garbage collected, and a warning is logged. Do not rely on that for shutdown.

After a fence (see open/2), close returns :ok if the handle already knew it was fenced, for example from a failed write or a {:slap_slatedb_closed, ref, tag, :fenced} message. If close is what discovers the fence, it returns {:error, %Slap.SlateDB.Error{kind: :closed, reason: :fenced}}: writes that were not yet durable are lost. Either way the handle is closed.

@spec create_checkpoint(t(), keyword()) ::
  {:ok, %{id: String.t(), manifest_id: non_neg_integer()}}
  | {:error, Slap.SlateDB.Error.t()}

Creates a checkpoint: a named, durable view of the database that garbage collection will not remove. Returns {:ok, %{id: id, manifest_id: id}}.

Open a Slap.SlateDB.Reader at the checkpoint, or clone the database from it with Slap.SlateDB.Admin.clone/3. Manage checkpoints with Slap.SlateDB.Admin.

Options

• :scope - :durable (default) covers what is durable now. :all first flushes writes still in memory, so it covers every write so far.
• :lifetime - milliseconds until the checkpoint expires. By default it never expires.
• :name - a name, for Slap.SlateDB.Admin.list_checkpoints/2.
• :source - the id of an existing checkpoint to copy.
• :timeout - see "Timeouts" above.

@spec delete(t(), binary(), keyword()) ::
  {:ok, non_neg_integer()} | {:error, Slap.SlateDB.Error.t()}

Deletes key. Returns {:ok, seq}. Takes the :await_durable option of put/4.

@spec durability_lag(t()) :: non_neg_integer()

Returns last_write_seq/1 - durable_seq/1, or 0: how many sequence numbers written through this handle are not durable yet. Unlike stats/1, it does not collect metrics, so it is cheap enough to sample often.

@spec durable_seq(t()) :: non_neg_integer()

Returns the highest sequence number that is durable in object storage.

A write is durable when its seq is at most this value.

@spec flush(t(), keyword()) :: :ok | {:error, Slap.SlateDB.Error.t()}

Writes buffered writes to object storage.

Options

• :type - :wal (default) writes the WAL, which makes all writes so far durable. :memtable also writes the memtable out as an L0 SST.
• :timeout - see "Timeouts" above.

@spec get(t(), binary(), keyword()) ::
  {:ok, binary() | nil} | {:error, Slap.SlateDB.Error.t()}

Reads the value for key. Returns {:ok, nil} when the key is not found.

Options

• :durability - :memory (default) reads the latest write, even if it is not durable yet. :remote reads only data that is durable in object storage.
• :dirty - when true, also reads writes that are not yet committed to the WAL. Defaults to false.
• :cache_blocks - whether blocks read from object storage go into the block cache. Defaults to true.
• :timeout - see "Timeouts" above.

@spec get_key_value(t(), binary(), keyword()) ::
  {:ok, key_value() | nil} | {:error, Slap.SlateDB.Error.t()}

Reads the current row for key with its metadata.

Returns {:ok, nil} when the key is not found, or {:ok, row} where row is a map with these keys:

• :key and :value
• :seq - the sequence number of the write
• :create_ts - when the row was written, in Unix milliseconds
• :expire_ts - when the row expires, in Unix milliseconds, or nil

Takes the options of get/3.

@spec increment(t(), binary()) ::
  {:ok, non_neg_integer()} | {:error, Slap.SlateDB.Error.t()}

Adds by to the counter at key, without reading it first.

The database must be opened with merge_operator: :u64_add or :i64_add. Read the counter with get/3 and Slap.SlateDB.MergeOperator.decode_u64/1 or decode_i64/1. Takes the options of merge/4. With no by, the increment is 1; options can be passed as the third argument.

@spec iterator(t(), keyword()) ::
  {:ok, Slap.SlateDB.Iterator.t()} | {:error, Slap.SlateDB.Error.t()}

Opens a Slap.SlateDB.Iterator. Takes scan/2's bounds, prefix and read options. :with_versions fixes the row shape when the iterator opens; Slap.SlateDB.Iterator.next_batch/3 sets the fetch size.

@spec last_write_seq(t()) :: non_neg_integer()

Returns the highest sequence number written through this handle (0 before any write). Every write so far is durable once durable_seq/1 reaches it.

@spec merge(t(), binary(), binary(), keyword()) ::
  {:ok, non_neg_integer()} | {:error, Slap.SlateDB.Error.t()}

Writes a merge operand for key. Returns {:ok, seq}.

The database must be opened with a :merge_operator, which combines the operand with the key's current value when the key is read or compacted. See Slap.SlateDB.MergeOperator for the operators and their operand formats. An operand in the wrong format is rejected with an :invalid error.

Takes the :ttl, :await_durable and :timeout options of put/4.

@spec metrics(t()) :: [%{name: String.t(), labels: map(), value: term()}]

Returns every metric SlateDB records for the database, as a list of %{name: name, labels: %{label => value}, value: value}.

value is an integer for counters and gauges. For a histogram it is a map with :count, :sum, :min, :max, :boundaries and :bucket_counts. Names start with slatedb., for example "slatedb.db.write_ops".

@spec open(String.t(), keyword()) :: {:ok, t()} | {:error, Slap.SlateDB.Error.t()}

Opens the database at path in the given object store, or creates it.

Only one writer can have a database open at a time. When a second writer opens it, the first one is fenced: its next write fails with a :closed error with reason :fenced.

Options

• :store - where the data lives. Required. One of:

  • :memory - an in-memory store, lost when the VM exits. Good for tests.
  • {:local, dir} - a directory on the local file system.
  • {:url, url} or {:url, url, options} - an object storage URL: "s3://bucket/prefix" (Amazon S3 and compatible stores), "az://container/prefix" (Azure Blob Storage; also abfs:// and azure://) or "gs://bucket/prefix" (Google Cloud Storage). Configuration comes from environment variables (AWS_*, AZURE_* or GOOGLE_*, such as AWS_REGION, AZURE_STORAGE_ACCOUNT_NAME or GOOGLE_SERVICE_ACCOUNT), then from options, which win. For example [aws_region: "us-east-1", aws_endpoint: "http://localhost:9000"]. An unknown option key is an error.

  SlateDB uses conditional puts to fence other writers. S3 stores always use ETag conditional puts, and a conditional_put setting other than "etag", in the environment or in options, is an error. Azure and GCS support conditional puts without configuration.

• :settings - a map of SlateDB settings, merged over the defaults. The keys match SlateDB's JSON settings format, for example %{flush_interval: "50ms", default_ttl_millis: 60_000}. To keep a local disk cache of object storage files, set %{object_store_cache_options: %{root_folder: "/var/cache/slatedb"}}.

• :cache - the in-memory block and metadata cache. By default each database has its own. Pass a Slap.SlateDB.Cache to share one between databases, or :disabled to turn it off.

• :merge_operator - a built-in merge operator for merge/4. See Slap.SlateDB.MergeOperator. Every process that opens the database, including readers, must use the same one.

• :compaction_filter - a Slap.SlateDB.CompactionFilter that deletes keys under a set of prefixes as they are compacted. Needs the compactor, which is on by default.

• :timeout - see "Timeouts" above.

@spec probe_store(store(), String.t(), keyword()) ::
  {:ok, [{atom(), term()}]}
  | {:error, {:probe_failed, [{atom(), term()}]} | Slap.SlateDB.Error.t()}

Checks that a store honours conditional writes, which SlateDB's writer fencing depends on. Run it before opening databases on a store you have not checked, such as a new S3-compatible server.

It works on a new object at path (under the store's URL prefix, if any) in five steps:

• :create - a create-if-absent PUT succeeds.
• :create_again - a second create-if-absent PUT is refused.
• :stale_if_match - a PUT with If-Match and a stale ETag is refused.
• :current_if_match - a PUT with If-Match and the current ETag succeeds.
• :delete - the object is deleted.

Each step is :ok, {:failed, message}, :skipped (after an earlier failure) or, for the two If-Match steps, :unsupported: SlateDB only needs create-if-absent, and some stores (the local file system) do not implement If-Match. A store that accepts a stale ETag fails.

Returns {:ok, steps} when nothing failed, {:error, {:probe_failed, steps}} when a step failed, or {:error, %Slap.SlateDB.Error{}} if the store cannot be reached. {:probe_failed, steps} means the probe ran and found a failed condition; %Slap.SlateDB.Error{} means a store operation failed.

The :memory_ignoring_preconditions store, an in-memory store that treats every conditional PUT as an overwrite, exists only to test code that runs this probe. Do not open databases on it.

Takes the :timeout option.

@spec put(t(), binary(), binary(), keyword()) ::
  {:ok, non_neg_integer()} | {:error, Slap.SlateDB.Error.t()}

Writes value for key. Returns {:ok, seq}, the write's sequence number.

Options

• :ttl - time to live in milliseconds. By default, the default_ttl_millis setting applies (no expiry unless you set it). SlateDB removes expired rows during compaction, so a read can still return a row for a while after it expires. Check :expire_ts from get_key_value/2 if you need an exact cutoff.
• :await_durable - when true, returns only after the write is durable in object storage. Defaults to false.
• :timeout - see "Timeouts" above.

Values of 64 KiB or more are passed to SlateDB without copying.

@spec scan(t(), keyword()) :: Enumerable.t()

Returns a lazy stream of {key, value} tuples in key order. With with_versions: true, each row is {key, value, version}.

The scan starts when the stream is run. Each run sees the data as it is at that time. Enumeration raises Slap.SlateDB.Error if a read fails.

Options

• :gte / :gt - lower bound, inclusive or exclusive.
• :lte / :lt - upper bound, inclusive or exclusive.
• :prefix - only keys that start with this binary. Can be combined with the bounds above.
• :batch_size - rows fetched per native call. Defaults to 256.
• :with_versions - include each row's sequence number as its version.
• :order - :asc (default) or :desc.
• :durability, :dirty - as for get/3.
• :cache_blocks - whether blocks read by the scan go into the block cache. Defaults to false, so a large scan does not push out hot blocks.
• :read_ahead_bytes - how many bytes to fetch ahead from object storage.
• :max_fetch_tasks - how many fetches to run at the same time.
• :timeout - applies to opening the scan and to each batch fetch.

Examples

Slap.SlateDB.scan(db, gte: "user:", lt: "user;") |> Enum.take(10)
Slap.SlateDB.scan(db, prefix: "user:") |> Stream.map(&elem(&1, 0)) |> Enum.to_list()

@spec set_log_level(:debug | :info | :warning | :error | :none) :: :ok

Sets the lowest level of SlateDB log records sent to Logger: :debug, :info, :warning, :error, or :none to send none.

@spec snapshot(t(), keyword()) ::
  {:ok, Slap.SlateDB.Snapshot.t()} | {:error, Slap.SlateDB.Error.t()}

Takes a consistent, read-only snapshot. See Slap.SlateDB.Snapshot.

@spec stats(t()) :: %{required(atom()) => non_neg_integer()}

Returns a summary of the database's state:

• :durable_seq - see durable_seq/1.
• :last_write_seq - the highest seq written through this handle.
• :durability_lag - see durability_lag/1.
• :l0_sst_count and :sorted_run_count - the shape of the LSM tree. A growing L0 count means compaction is falling behind.
• :cache_hits and :cache_misses - block cache lookups, over all entry kinds, since the database opened.

It works after close/1 too.

@spec subscribe(t(), term(), pid()) :: {:ok, Slap.SlateDB.Subscription.t()}

Subscribes pid to durability and close events.

pid gets {:slap_slatedb_durable, ref, tag, durable_seq} straight away and then each time the durable sequence number goes up. Updates are coalesced: a busy process gets the latest value, not every step.

When the database closes, pid gets {:slap_slatedb_closed, ref, tag, reason} once, where reason is :clean, :fenced, :panic or :unknown, and the subscription ends. A fenced writer finds out that it is fenced when it next writes or flushes, or when it next reads the manifest (every manifest_poll_interval).

ref is the returned subscription's :ref, unique to this subscription; tag is the caller's, and several subscriptions may share it.

The subscription also ends when pid exits (it is monitored) or on unsubscribe/2. It does not end when the returned value is garbage collected.

@spec transaction(
  t(),
  (Slap.SlateDB.Transaction.t() -> {:ok, any()} | {:error, any()}),
  keyword()
) ::
  {:ok, any()} | {:error, any()}

Runs fun in a transaction and commits it.

fun returns {:ok, value} to commit or {:error, reason} to roll back. Returns {:ok, value} only after the commit is durable. A rollback returns {:error, reason}. If fun raises, throws or exits, the transaction is rolled back and the failure is propagated.

Options

• :isolation - :snapshot (default) or :serializable.
• :retries - how many times to run fun again after a commit conflict. Defaults to 0. fun must be safe to run more than once.
• :timeout - applies to starting and to committing the transaction.

Use begin/2 and Slap.SlateDB.Transaction.commit/2 to commit without waiting for durability and receive the commit's sequence number.

Example

Slap.SlateDB.transaction(db, fn tx ->
  {:ok, balance} = Slap.SlateDB.Transaction.get(tx, "balance")
  new_balance = String.to_integer(balance || "0") + 10
  :ok = Slap.SlateDB.Transaction.put(tx, "balance", Integer.to_string(new_balance))
  {:ok, new_balance}
end, retries: 3)

@spec unsubscribe(Slap.SlateDB.Subscription.t(), keyword()) :: :ok

Ends a subscription from subscribe/3. Ending it twice is allowed.

No message is sent once this returns, but messages sent before it may be waiting in the subscriber's mailbox. With flush: true, called from the subscriber, they are removed.

@spec validate_settings(map()) :: :ok | {:error, Slap.SlateDB.Error.t()}

Checks a :settings map with the same parser used by open/2, without opening a database. Returns :ok or an :invalid error.

@spec validate_store!(store()) :: :ok

Checks a store specification with the same Elixir parser used by open/2, without opening the store. Returns :ok or raises on a malformed specification, as open/2 does. Store connectivity is checked when opened.

@spec write(t(), [write_op()], keyword()) ::
  {:ok, non_neg_integer()} | {:error, Slap.SlateDB.Error.t()}

Applies a list of writes atomically: either all of them happen or none do. Returns {:ok, seq}. All writes in the batch share that sequence number.

Slap.SlateDB.write(db, [
  {:put, "a", "1"},
  {:put, "b", "2", 60_000},
  {:delete, "c"}
])

A {:merge, key, operand} needs a :merge_operator (see merge/4). A four-element :put or :merge sets a TTL in milliseconds. Takes the :await_durable and :timeout options of put/4.

A batch of more than 1000 operations is converted on a dirty CPU scheduler, so it does not hold up a normal scheduler.