ScrivaDB

A lightweight, append-only, file-based document database — single binary, zero dependencies, human-readable storage.

# Homebrew (via the tap; published when a release has the tap token)
brew install --cask srjn45/scriva/scriva

# Go
go install github.com/srjn45/scriva/cmd/scriva@latest

# Docker (published GHCR image; explicitly expose its loopback listeners)
docker run -p 5433:5433 -p 8080:8080 -e SCRIVA_API_KEY=dev-key ghcr.io/srjn45/scriva \
  --grpc-addr 0.0.0.0:5433 --rest-addr 0.0.0.0:8080

The published GHCR image uses scriva serve --data /data, whose listeners default to loopback. Pass the address flags above when publishing Docker ports; binding 0.0.0.0 makes the server require an API key or TLS, and the example supplies an API key. The local image built by the root Dockerfile for Compose already bakes in the address flags and likewise requires an API key or TLS. See Getting Started for release archives, .deb/.rpm packages, Scoop and AUR.

# 1. Start the server
scriva serve --data ./data --api-key dev-key

# 2. Insert a record
scriva-cli insert users '{"name":"alice","age":30}' --api-key dev-key

# 3. Query it
scriva-cli find users '{"field":"name","op":"eq","value":"alice"}' --api-key dev-key

Or use the interactive REPL:

scriva-cli repl --api-key dev-key

# Start the server (builds the image from the local Dockerfile on first run)
SCRIVA_API_KEY=dev-key docker compose up -d

# Insert a record against the REST gateway on :8080
curl -H "x-api-key: dev-key" \
  -d '{"data":{"name":"alice","age":30}}' \
  http://localhost:8080/v1/users/records

gRPC is exposed on :5433 and REST on :8080; data persists in the named scriva-data volume. See docker-compose.yml and the fully commented scriva.example.yaml for configuration.

ScrivaDB stores each collection as a set of NDJSON segment files — one JSON object per line, appended in order. There is no binary format, no external runtime, and no hidden state. You can inspect, backup, or migrate your data with any text editor or Unix tool.

Key properties:

• Exclusive directory lock — the engine takes an OS-level advisory lock (LOCK file) on the data directory to prevent corruption from concurrent writers; a second process trying to open the directory fails fast
• Fail-closed, verifiable recovery — open validates the persisted indexes against the segments (v2 coverage + fingerprints; replay only the unpersisted tail), refuses to start on damaged segments or ambiguous history (--integrity-policy fail), and scriva verify / scriva repair inspect and rebuild a stopped data directory with a verified backup first. See the recovery runbook
• Append-only writes — inserts, updates, and deletes are always new lines; no in-place modification
• Configurable durability — choose none (OS flush), always (fsync per write), or interval (fsync on a timer) to trade throughput against crash-loss window
• End-to-end integrity — every segment entry carries a CRC32C checksum, so silent on-disk bit-rot is caught on read instead of returning wrong data
• Encryption at rest (embedded) — transparent field- or record-level AEAD encryption (XChaCha20-Poly1305) configured through the embedded façade: scriva.WithPassphrase/WithEncryptionKey/WithKeyProvider + WithCollectionEncryption(EncryptFields/EncryptRecord). Values are opaque on disk under a reserved marker while reads return plaintext; a wrong key/passphrase fails fast on open; key rotation and enable/disable migrate existing data lazily or in bulk via a re-encrypting compaction pass. Segments, backups, and replication carry ciphertext for free. See docs/encryption-at-rest.md
• Background compaction — a goroutine per collection merges and deduplicates sealed segments; operators can also force a synchronous pass on demand (scriva-cli compact)
• Online backup — scriva-cli backup streams a consistent gzip snapshot of the live database; restore is a plain tar xzf into a data directory
• Leader→follower replication — start a hot standby with --replicate-from <leader>: the follower bootstraps from a snapshot then tails the leader's committed writes (each tagged with a monotonic global LSN), applying them through the normal write path so its indexes match exactly. Async (bounded lag), resumes after a disconnect from its persisted applied-LSN with no gaps or duplicates, and exposes ReplicationStatus (leader LSN, per-follower lag).
• Read replicas — a follower serves reads (Find/FindById/FindByKey/Aggregate) from its applied state and refuses writes with a typed FAILED_PRECONDITION, so read traffic scales horizontally across followers. Bound staleness by diffing a follower's applied_lsn against the leader's leader_lsn
• Manual failover — after a leader loss, promote a caught-up follower to leader with the admin Promote RPC (scriva-cli promote): it stops replicating, lifts the read-only guard, and accepts writes. A lag guard refuses promoting a stale replica (override with --force); promotion is one-way and automatic election is out of scope. See the operator runbook
• In-memory index — O(1) lookup by id, persisted with a checksum for fast restarts
• Secondary indexes — per-field indexes for O(1) equality lookups and O(matches) range queries (gt/lt/…); automatically maintained and persisted
• Streaming queries — Find pushes limit/offset, a multi-field order_by_fields sort, and a keyset page_token cursor into the engine and streams results as it reads; a limited query is bounded by the page size, not the collection size, and honours client cancellation. Cursor pagination seeks past already-returned rows in O(page) — concatenated pages cover every row exactly once, no dupes or gaps. Optional field projection (--fields) returns only the requested fields (id/key/rev always included)
• Aggregations — Aggregate computes a count and numeric sum/avg/min/max over the same filter as Find, optionally grouped by a field; it streams one result per group and folds records in the engine (memory bounded by distinct groups, not rows), so clients count and total server-side instead of pulling the whole collection. CLI: aggregate --group-by --field --aggs count,sum,avg,min,max
• Transactions — optimistic multi-operation transactions via BeginTx / CommitTx / RollbackTx
• Cross-collection transactions — stage point writes against several collections of one database and commit them atomically (BeginXTx / StageXTx / GetXTx / CommitXTx / RollbackXTx, scriva-cli begin-xtx …, db.Transact when embedded). Isolation is optimistic point-read/write validation at commit — a changed document fails the commit with a typed conflict and nothing is applied; there is no phantom protection, so scans are not available inside a transaction. Commits are idempotent under a client key, and an unknown outcome is resolved with XTxStatus instead of a blind retry
• Keyed CRUD, upsert & CAS over the wire — caller-supplied string keys, insert-or-replace upsert, natural-key find/update/delete, and revision-checked compare-and-swap (update-if-rev) are now exposed over gRPC/REST (not just the embedded engine); every record carries a key and a monotonic rev, with duplicate/missing keys mapped to AlreadyExists/NotFound
• TTL / expiring records — per-record deadlines (--ttl / ttl_seconds on Insert & Update), a per-collection default (create-collection --default-ttl, persisted), and a server-wide default (--default-ttl); expired records are hidden from reads immediately and reclaimed by compaction
• gRPC + REST — dual API served from one binary; CLI uses the Unix socket when local
• OpenAPI spec — docs/openapi/scriva.swagger.json generated from the proto; generate clients for any language with openapi-generator
• Official client SDKs — idiomatic, hand-written libraries for 10 languages (Python, JavaScript/TypeScript, PHP, Java, Kotlin, Scala, Clojure, Ruby, Rust, C#/.NET) under clients/
• Scoped API keys — multiple named keys with read or read-write scope; a read-only key is refused on writes, and keys hot-reload on SIGHUP for rotation without a restart
• Secure by default — loopback binds, non-loopback exposure requires auth/TLS (or explicit --insecure-allow-remote), 0600 socket, HTTP timeouts/body limits, CORS off, TLS/mTLS replication; see upgrade guide
• Optional TLS — TCP gRPC listener can be secured with a cert/key pair; CLI verifies via --tls-ca
• YAML config file — --config scriva.yaml with CLI flag overrides always winning
• Prometheus metrics — per-collection gauges, compaction histograms, gRPC request duration, and per-query rows-scanned at --metrics-addr
• Structured logging — leveled log/slog output (--log-level, --log-format json|text); one record per RPC with method, principal, duration, and status code
• Slow-query log — opt-in --slow-query-ms logs any Find over the threshold at WARN with filter shape, rows scanned vs returned, and whether an index was used, so unindexed hot queries surface from logs and metrics (off by default)
• Health & readiness — standard grpc.health.v1.Health service (SERVING → NOT_SERVING on graceful shutdown) plus HTTP /healthz (liveness) and /readyz (DB open + data dir writable) probes
• Backpressure & limits — opt-in --max-inflight in-flight ceiling and per-key --rate-limit token bucket shed load with RESOURCE_EXHAUSTED instead of unbounded resource growth; --max-concurrent-streams caps per-connection HTTP/2 streams (all off by default)
• Distributed tracing — opt-in OpenTelemetry spans to an OTLP collector (--otlp-endpoint, --otlp-sample-ratio); one span per RPC plus child engine.scan/engine.compaction spans, so a slow Find is traceable gateway → gRPC → engine (off by default; the embeddable engine gains no OTel dependency)
• Single binary — no JVM, no Python, no config files required to get started
• Web admin UI — browser-based collection and record manager at clients/web/ (React + Vite, talks to the REST gateway)

ScrivaDB is the right tool when:

• You need persistence without standing up PostgreSQL or MongoDB
• Your data fits on one machine and you want human-readable files
• You're building CLI tools, local services, IoT daemons, or small web apps
• You want a simple HTTP/gRPC API you can call from any language

It is not the right tool for multi-node replication, complex joins, or datasets too large to compact on a single machine.

ScrivaDB's storage engine is a plain Go library, so you can skip the server entirely and run the database in-process — no gRPC, no network, no separate daemon. This is the right choice when your program is the only writer and you want ScrivaDB's durability and query model directly inside your binary.

go get github.com/srjn45/scriva/engine   # the storage engine
go get github.com/srjn45/scriva          # the ergonomic façade (recommended)

import "github.com/srjn45/scriva"

db, _ := scriva.Open("./data")            // embedded durability defaults (fsync ~1s)
defer db.Close()

sessions := db.MustCollection("sessions")
id, _, _ := sessions.InsertWithKey("sess-1", map[string]any{"status": "open"})
rec, _   := sessions.GetByKey("sess-1")   // caller-supplied string keys
_, _ = sessions.UpdateIfRev("sess-1", rec.Rev, map[string]any{"status": "closed"}) // CAS

Opt into transparent encryption at rest with a key option plus a per-collection policy — secrets are sealed on write and returned as plaintext on read, with nothing else in your code changing:

db, _ := scriva.Open("./data",
    scriva.WithPassphrase(os.Getenv("SCRIVA_PASSPHRASE")),
    scriva.WithCollectionEncryption("users", scriva.EncryptFields("password", "ssn")), // field-level
    scriva.WithCollectionEncryption("audit", scriva.EncryptRecord("id", "tenant")),     // record-level
)
users := db.MustCollection("users")
id, _, _ := users.Insert(map[string]any{"email": "a@b.com", "password": "hunter2"})
rec, _   := users.Get(id)          // rec.Data["password"] == "hunter2"; ciphertext on disk

A wrong passphrase fails fast on Open. Key rotation and enable/disable are runtime operations on the returned collection (RotateKey, SetEncryptionPolicy, MigrateNow, EncryptionStatus). See docs/encryption-at-rest.md for the full design.

The embedded surface includes caller-supplied string keys, per-record revisions with compare-and-swap, upsert, count/exists, secondary indexes, in-process Watch subscriptions, transparent encryption at rest, and a LoadJSONL bulk-import path. The engine pulls in no gRPC/protobuf/Prometheus/cobra/OpenTelemetry dependencies — a CI gate enforces that.

This is a distinct distribution channel: go get for embedding, while the standalone server ships via GHCR, tagged binary releases (with .deb/.rpm packages), and the Homebrew tap. Both build from the same repo. See docs/embedding.md for the full API reference, durability modes, the Watch overflow contract, the versioning/stability policy, and a migration guide.

Idiomatic, hand-written client libraries are available for ten languages. Each wraps the same gRPC API, takes the same connection config (host, port, api_key, optional TLS CA cert), and wraps the core collection, record, index, single-collection transaction, Aggregate, Compact and Snapshot calls, including the streaming Find and Watch calls, plus the cross-collection transaction methods (BeginXTx, StageXTx, GetXTx, CommitXTx, RollbackXTx, XTxStatus). The replication/admin RPCs (Replicate, ReplicationStatus, Promote) have no hand-written SDK methods; call them through the generated gRPC stubs (each SDK bundles them) or, where a REST route exists, over REST.

Unknown commit outcomes. If CommitXTx fails with XTX_OUTCOME_UNKNOWN (gRPC UNKNOWN) or XTX_IN_PROGRESS (UNAVAILABLE), never retry the commit blindly. Call XTxStatus with the error's status_ref (or the idempotency key you chose at begin): COMMITTED means success, ABORTED/EXPIRED permits a new transaction, and PENDING means poll with backoff. Choose and keep an explicit idempotency key if a commit must be recoverable across a server restart. Inspect the scriva.xtx ErrorInfo reason, never the message text. Contract: docs/sdk-cross-collection-transaction-contract.md.

Verification status. The table below is what was actually run locally; "unverified" SDKs have the code and (where noted) tests but could not be executed by the release verifier, so treat their cross-collection transaction support as unverified until CI or a maintainer runs them.

Versions. The SDKs are versioned independently of the server (currently 1.3.0, which adds the additive XTx methods; existing BeginTx/CommitTx/RollbackTx are unchanged). The XTx methods need a server that supports BeginXTx (v1.6.0+).

Prefer to generate your own? The checked-in OpenAPI spec describes the 29 REST-annotated operations of the gateway. The five gRPC-only RPCs (BeginTx, CommitTx, RollbackTx, Snapshot, Replicate) are not in it; generate from proto/scriva.proto for those — see Getting Started.

git clone https://github.com/srjn45/scriva
cd scriva
make build          # produces bin/scriva and bin/scriva-cli
make test           # run tests with race detector
make proto          # regenerate gRPC code (requires buf)

Contributions are welcome — see CONTRIBUTING.md for how to build, test, and submit changes.

MIT — see LICENSE.

Rebuilt from FileDB PHP — original college project, now in Go.