Lutaml::Store

Purpose

Lutaml::Store provides a store-centric database-style API for Lutaml::Model objects with model registry, polymorphic support, composite relationships, and multiple storage backends.

It offers a unified interface for storing and retrieving complex model hierarchies, batch file I/O with multiple serialization formats, HTTP-aware caching, and package-based persistence with ZIP and directory transports.

It also provides read-only Sources for working with data repositories that live outside your process — GitHub Pages, any static host, a REST API, a local package directory, or a .zip distribution — through one facade that never guesses which layer answered.

Quick start: read a LutaML data repository

The same read API whether the data lives in the cloud, a local package, or a downloaded .zip — backends are configured, never guessed:

require "lutaml/store"

# From a cloud API (api.relaton.org is the reference implementation)
repo = Lutaml::Store::Repository.new(
  source: Lutaml::Store::Source.for(:rest,
    base_url: "https://api.relaton.org", collection: "ietf"),
  cache: Lutaml::Store::Source.for(:directory, path: "~/.cache/relaton/ietf"),
)

# From a local GCR-style package — identical read API
repo = Lutaml::Store::Repository.new(
  source: Lutaml::Store::Source.for(:directory, path: "~/gcr/relaton/ietf"),
)

# From a downloaded .zip distribution — same API
repo = Lutaml::Store::Repository.new(
  source: Lutaml::Store::Source.for(:zip, path: "ietf.zip"),
)

# Read
repo.read("RFC 7231")        # => bytes
repo.exist?("RFC 7231")      # => true
repo.keys                     # => ["RFC 7231", ...]
repo.manifest                 # => Lutaml::Store::Manifest

# Parse into your model — the model class is declared, never inferred
repo.get("RFC 7231", MyModel)

# Search by metadata fields
repo.search(docid: "RFC 7231")

# Pull the whole collection into a local GCR-style package
repo.pull!(into: "~/gcr/relaton", collection: "ietf")

# Offline mode: never touch the source, serve from the local package only
offline = Lutaml::Store::Repository.new(source: repo.source, cache: repo.cache, mode: :offline)
offline.read("RFC 7231")     # served from the cache, no network



== Features

* **Repository** — the uniform facade: one read API over any source (cloud, local, zip), explicit cache, online/offline modes
* **Source** (read-only) — Directory, Zip, Https (ETag/304), Rest (the lutaml cloud store API); 404 = NotFoundError, transport = BackendError
* **Manifest** — the enumeration SSOT: keys, sha256 digests, opaque shard metadata
* **Mirror** — pull any source into a GCR-style package (incremental, digest-verified); pack to a distributable .zip
* **DatabaseStore** — high-level CRUD with model registry, polymorphism, composites
* **PackageStore** — structured multi-model packages with ZIP and directory transport
* **BasicStore** — low-level key-value store with optional cache, events, monitoring
* **CacheStore** — TTL-aware cache with LRU eviction extending BasicStore
* **HttpCache** — HTTP-aware caching with ETags, conditional requests, Cache-Control
* **Format-aware file I/O** — YAML, YAMLS, JSON, JSONL, Marshal with layout strategies
* **Multiple backends** — Memory, FileSystem, SQLite (all thread-safe)
* **Model registry** — configurable key fields, polymorphic inheritance, composite relationships
* **Dot-notation updates** — nested attribute paths with block-based updates
* **Ruby autoload** — lazy constant loading, only loads what you use

== Installation

Add this line to your application's Gemfile:

[source,ruby]

gem 'lutaml-store'

And then execute:

[source,sh]

$ bundle install

Or install it yourself as:

[source,sh]

$ gem install lutaml-store

For SQLite backend support, also add:

[source,ruby]

gem 'sqlite3'

== Quick start

[source,ruby]

require 'lutaml/model' require 'lutaml/store'

Define your models

class Studio < Lutaml::Model::Serializable attribute :studio_key, :string attribute :name, :string attribute :location, :string end

class PotteryClass < Lutaml::Model::Serializable attribute :studio, Studio attribute :class_id, :string attribute :description, :string end

Create a store with model registry

store = Lutaml::Store.new( adapter: :memory, models: [ { model: PotteryClass, key: :class_id }, { model: Studio, key: :studio_key } ] )

Save with composite relationships (both stored independently)

pottery = PotteryClass.new( class_id: "pottery_101", studio: Studio.new(studio_key: "main_studio", name: "Main Studio"), description: "Beginner pottery class" ) store.save(pottery)

Fetch by model and key

retrieved = store.fetch(model: PotteryClass, class_id: "pottery_101") puts retrieved.studio.name # ⇒ "Main Studio"

Nested update with dot notation

store.update( model: PotteryClass, class_id: "pottery_101", attributes: [ { key: :description, value: "Advanced pottery class" }, { key: "studio.location", value: "Building A" } ] )

== Architecture

Lutaml::Store has two layers:

**DatabaseStore** (via `Lutaml::Store.new`)::
High-level CRUD with model registry.
Handles polymorphic dispatch, composite model decomposition,
dot-notation updates, file I/O (`save_all`, `load_all`, `import_all`, `export`).

**BasicStore**::
Low-level key-value store wrapping an adapter.
Provides `get`, `set`, `delete`, `exists?`, `all`, `keys`, bulk operations,
with optional caching, monitoring, and event emission.

=== Key classes

[cols="1,3"]
|===
| Class | Role

| `DatabaseStore`
| High-level CRUD with model registry, composites, polymorphism

| `PackageStore`
| Multi-model packages with directory/ZIP transport

| `PackageDefinition`
| Declarative schema for package structure (models, assets, metadata)

| `BasicStore`
| Low-level key-value store with optional cache/events/monitoring

| `CacheStore`
| TTL-aware cache store extending BasicStore

| `HttpCache`
| HTTP-aware caching with ETags, conditional requests, Cache-Control

| `ModelRegistry` / `ModelRegistration`
| Register models with key fields and polymorphic config

| `CompositeModelHandler`
| Stores nested registered models independently, restores references

| `AttributeUpdater`
| Processes dot-notation paths and block-based updates

| `ModelSerializer`
| Serialization/deserialization with custom serializer support

| `Format`
| Multi-format file I/O (YAML, JSON, JSONL, Marshal) with layout strategies

| `Config`
| Parses and validates store configuration
|===

=== Storage adapters

All adapters inherit from `Adapter::Base` and provide `get`, `set`, `delete`,
`exists?`, `keys`, `all`, `clear`, `size`, `each_key`, `update`, and bulk
operations. `each_key` defaults to `keys.each`, so a custom adapter only needs
`keys`.

`update(key) { |old| new }` is a read-modify-write: the block gets the current
value (`nil` when the key is missing) and its result is stored. It is atomic
across threads for Memory, and across threads and processes for FileSystem and
SQLite. `BasicStore#update` calls it and refreshes the read cache.

[source,ruby]

store = Lutaml::Store::BasicStore.new(adapter_type: :filesystem, adapter_options: { path: "./data" }) store.update("counter") { |n| (n.to_i + 1).to_s }

[cols="1,1,3"]
|===
| Adapter | Type symbol | Use case

| `Adapter::Memory`
| `:memory`
| Fast in-memory storage for testing, caching, temporary data

| `Adapter::FileSystem`
| `:filesystem`
| Persistent file-based storage with integrity checks

| `Adapter::Sqlite`
| `:sqlite`
| ACID-compliant database storage for production use
|===

== Model registry

=== Registration

Register models with their unique key fields:

[source,ruby]

store = Lutaml::Store.new( adapter: :memory, models: [ { model: User, key: :user_id }, { model: Post, key: :post_id } ] )

=== Polymorphic models

For inheritance hierarchies, register the base class with a polymorphic class key:

[source,ruby]

class Studio < Lutaml::Model::Serializable attribute :studio_key, :string attribute :name, :string attribute :_class, :string, default: → { "Studio" }, polymorphic_class: true end

class CeramicStudio < Studio attribute :clay_type, :string attribute :_class, :string, default: → { "CeramicStudio" } end

store = Lutaml::Store.new( adapter: :memory, models: [ { model: Studio, key: :studio_key, polymorphic_class_key: :_class } ] )

store.save(CeramicStudio.new(studio_key: "cs1", name: "Clay Haus", clay_type: "Porcelain")) retrieved = store.fetch(model: Studio, studio_key: "cs1") puts retrieved.class.name # ⇒ "CeramicStudio"

=== Composite models

When registered models are nested within other registered models, they are stored
independently while maintaining references:

[source,ruby]

store = Lutaml::Store.new( adapter: :memory, models: [ { model: PotteryClass, key: :class_id }, { model: Studio, key: :studio_key } ] )

pottery = PotteryClass.new( class_id: "p101", studio: Studio.new(studio_key: "s1", name: "Main Studio") ) store.save(pottery)

Both accessible independently

store.fetch(model: Studio, studio_key: "s1").name # ⇒ "Main Studio" store.fetch(model: PotteryClass, class_id: "p101").studio.name # ⇒ "Main Studio"

== CRUD operations

=== Save

[source,ruby]

store.save(User.new(user_id: "u1", name: "Ada")) store.save([user1, user2, user3]) # bulk save

=== Fetch

[source,ruby]

user = store.fetch(model: User, user_id: "u1")

=== Update

[source,ruby]

Hash-based attributes

store.update( model: User, user_id: "u1", attributes: [ { key: :name, value: "Grace" }, { key: "studio.location", value: "Building A" } # dot notation for nesting ] )

Block-based

store.update(model: User, user_id: "u1") do |user| user.name = "Grace" end

=== Destroy

[source,ruby]

store.destroy(model: User, user_id: "u1")

== PackageStore

`PackageStore` provides structured multi-model persistence with directory and ZIP
transports. Use it when you need to bundle multiple model types, binary assets,
and metadata into a single loadable package.

The workflow is: define the package schema with `PackageDefinition`, then load,
query, modify, and save using `PackageStore`.

=== Define a package

`PackageDefinition` declares the package structure — which models, assets, and
metadata the package contains:

[source,ruby]

require "lutaml/model" require "lutaml/store"

Define your models

class Concept < Lutaml::Model::Serializable attribute :term, :string attribute :definition, :string end

class Author < Lutaml::Model::Serializable attribute :name, :string attribute :email, :string end

class GlossaryInfo < Lutaml::Model::Serializable attribute :title, :string attribute :version, :string end

Declare the package schema

glossary = Lutaml::Store::PackageDefinition.new( name: "glossary", metadata_model: GlossaryInfo, metadata_file: "glossary.yaml", metadata_key: :title ) do |pkg| # Models stored in subdirectories, one file per instance pkg.model(model: Concept, key: :term, dir: "concepts", default_format: :yaml) pkg.model(model: Author, key: :name, dir: "authors", default_format: :json)

  # Binary assets bundled in the package
  pkg.asset("logo.png", type: :file)
  pkg.asset("attachments", type: :directory)
end

Each `pkg.model` call registers a model type with:

* `model:` — the `Lutaml::Model::Serializable` class
* `key:` — attribute used as the unique identifier (becomes the filename)
* `dir:` — subdirectory for this model type (use `file:` instead for a single-file model)
* `layout:` — `:separate` (one file per instance, default) or `:grouped` (multiple instances per file)
* `default_format:` — `:yaml`, `:json`, `:jsonl`, `:yamls`, or `:marshal`

=== Directory structure

When saved to disk, the glossary package produces:

my_glossary/ ├── glossary.yaml # metadata (GlossaryInfo) ├── concepts/ # Concept model instances │ ├── API.yaml │ ├── REST.yaml │ └── YAML.yaml ├── authors/ # Author model instances │ ├── john_doe.json │ └── jane_smith.json ├── logo.png # asset file └── attachments/ # asset directory └── spec.pdf

=== Load and query

[source,ruby]

Load from directory

store = Lutaml::Store::PackageStore.load(glossary, "./my_glossary")

Load from ZIP file

store = Lutaml::Store::PackageStore.load(glossary, "./glossary.zip", transport: :zip)

Access metadata

store.metadata # ⇒ GlossaryInfo instance

Query models

concepts = store.models_for(Concept) # ⇒ array of Concept instances concept = store.fetch_model(Concept, "API") store.model_count(Concept) # ⇒ 3 store.model_exists?(Concept, "REST") # ⇒ true

Access assets

store.asset("logo.png") # ⇒ binary content store.asset_paths # ⇒ ["logo.png", "attachments/spec.pdf"]

Package statistics

store.stats # ⇒ { package: "glossary", models: { "Concept" ⇒ 3, "Author" ⇒ 2 }, # assets: 2, metadata: true }

=== Modify and save

[source,ruby]

Add models

store.add_model(Concept.new(term: "JSON", definition: "…")) store.add_models([concept1, concept2])

Remove models

store.remove_model(Concept, "deprecated_term")

Add/remove assets

store.add_asset("diagram.svg", svg_content) store.remove_asset("old_diagram.svg")

Save to directory (default format per model)

store.save("./output")

Save to ZIP with per-model format overrides

store.save("./glossary.zip", transport: :zip, formats: { Concept ⇒ :json })

Save with global format override

store.save("./output", format: :yaml)

Bulk operations

store.clear_models(Concept) # remove all Concept instances store.clear_all # remove everything

=== Per-model format override

The `save` method accepts format overrides:

[source,ruby]

Global format for all models

store.save("./out", format: :json)

Per-model format

store.save("./out", formats: { Concept ⇒ :yaml, Author ⇒ :jsonl })

Default: each model uses its default_format from the definition

store.save("./out")

=== Package transports

[cols="1,1,3"]
|===
| Transport | Symbol | Description

| `DirectoryTransport`
| `:directory`
| Filesystem directory with subdirectories per model type

| `ZipTransport`
| `:zip`
| ZIP archive containing the same directory structure
|===

Transports are resolved via registry (`PackageTransport.resolve(:zip)`),
extensible without modifying existing code.

== File I/O

DatabaseStore provides batch file I/O through `Format` handlers.

=== Format handlers

[cols="1,1,1,1"]
|===
| Format | Symbol | Extension | Description

| YAML | `:yaml` | `.yaml` | Single-document YAML files
| YAMLS | `:yamls` | `.yaml` | Multi-document YAML streams
| JSON | `:json` | `.json` | Single JSON objects
| JSONL | `:jsonl` | `.jsonl` | Line-delimited JSON
| Marshal | `:marshal` | `.bin` | Ruby Marshal binary format
|===

=== save_all / load_all / import_all / export

[source,ruby]

Write models to directory

store.save_all(concepts, path: "./data", format: :yaml, layout: :separate)

Read models (returns array, does NOT store in backend)

models = store.load_all(Concept, path: "./data", format: :yaml, layout: :separate)

Read AND store in backend (makes them queryable)

store.import_all(Concept, path: "./data", format: :yaml, layout: :separate) store.fetch(model: Concept, term: "API") # now available

Export to a single file

store.export(all_concepts, path: "output.yaml", format: :yaml)

Layout strategies: `:separate` (one file per model), `:grouped` (models grouped
by key), `:flat` (one file per model, no subdirectory).

== HTTP caching

`HttpCache` provides HTTP-aware caching with ETags, conditional requests (304),
Cache-Control directives, and Vary header support. It uses a storage adapter
internally and serializes cache entries as `Lutaml::Model` objects via JSON.

[source,ruby]

cache = Lutaml::Store::HttpCache.new( adapter_type: "memory", default_ttl: 3600, respect_http_headers: true, enable_conditional_requests: true )

Fetch with automatic caching

response = cache.fetch("GET", "https://api.example.com/resource", {}) do |headers| http_client.get("https://api.example.com/resource", headers) end

Second call returns cached response (no HTTP request made)

cached = cache.fetch("GET", "https://api.example.com/resource", {}) { raise "shouldn’t be called" }

Supports `no-store`, `no-cache`, `must-revalidate`, `max-age`, ETag-based
conditional requests, query parameter normalization, and filesystem/sqlite
adapters for persistent cache storage.

== CacheStore

`CacheStore` extends `BasicStore` with TTL-aware caching and LRU eviction:

[source,ruby]

cache = Lutaml::Store::CacheStore.new( adapter: :memory, max_size: 1000, default_ttl: 3600 )

cache.set("key1", "value1", ttl: 600) cache.get("key1") # ⇒ "value1" cache.fetch("key2", "default_value") # ⇒ "default_value" (stored) cache.fetch("key3") { expensive_compute } # computes and caches

== Storage backends

=== Memory

[source,ruby]

store = Lutaml::Store.new(adapter: :memory, models: […])

Fast in-memory storage. Data lost on process exit. Thread-safe via mutex.

=== FileSystem

[source,ruby]

store = Lutaml::Store.new( adapter: { type: :filesystem, path: "./data", extension: ".json" }, models: […] )

Persistent file-based storage with SHA-256 integrity checks. Files organized
by key in subdirectories.

* *File names.* Every byte outside `[A-Za-z0-9._-]` is percent-encoded, so
  `ISO/19115` and `ISO:19115` get different files and `keys` returns the
  original keys. A leading `.` is encoded too, so no file is hidden or outside
  the root. A name longer than 200 bytes becomes `~` plus the SHA-256 of the
  key; the `.meta` file beside it keeps the key. Files written by 0.3.0 (with
  `_` in place of unsafe characters) are still read, and move to the new name
  on the next write of that key.
* *Values.* A `String` is written as is. Any other value (for example the
  Hash that `DatabaseStore` saves) is written as JSON and read back as JSON.
* *Locks.* Writes hold an exclusive `flock` on `<path>/.lock` and reads hold a
  shared one. Data and `.meta` files are written to a unique temp file and
  renamed into place, so readers never see a partial write.
* *Case-insensitive file systems* (the macOS default) can still merge keys that
  differ only in letter case.

=== SQLite

[source,ruby]

store = Lutaml::Store.new( adapter: { type: :sqlite, path: "./store.db" }, models: […] )

ACID-compliant database storage. Requires `sqlite3` gem. Thread-safe with
connection pooling.

== Event system

[source,ruby]

store.on(:model_save) { |data| logger.info("Saved #{data[:model].class}") } store.on(:model_fetch) { |data| logger.debug("Fetched #{data[:key]}") } store.on(:model_destroy) { |data| audit_log << data }

Events: `:model_save`, `:model_fetch`, `:model_update`, `:model_destroy`,
`:model_save_all`, `:model_import`, `:model_export`, `:model_load_error`,
`:composite_model_stored`, `:polymorphic_model_resolved`.

== Error handling

Error hierarchy:

* `Lutaml::Store::Error`
** `ConfigurationError` — invalid store or adapter config
** `BackendError` — adapter-level failures
** `ModelNotRegisteredError` — operations on unregistered models
** `InvalidKeyError` — missing or invalid key fields
** `PolymorphicUpdateError` — polymorphic type conflicts
** `CompositeModelError` — composite model handling failures

== Thread safety

* `Adapter::Memory`: writes are synchronized with a Monitor; reads use a
  frozen snapshot.
* `Adapter::FileSystem`: safe across threads and processes that share one
  directory (see the FileSystem section above). `transaction { ... }` holds the exclusive
  lock for the whole block.
* `Adapter::Sqlite`: SQLite locking with a busy timeout; `update` uses
  `BEGIN IMMEDIATE`.
* `CacheStore`: all operations run under one lock per instance. The TTL and
  LRU bookkeeping is per process.
* `BasicStore`: the optional read cache is per process. Disable it
  (`cache: { enabled: false }`) when other processes write the same store.

A `get` followed by a `set` is never atomic. Use `update` for a
read-modify-write.

== Development

[source,sh]

bin/setup # Install dependencies bundle exec rake # Run specs + rubocop bundle exec rspec # Run all specs bundle exec rspec spec/lutaml/store/database_store_spec.rb # Single spec file bundle exec rubocop # Lint

Ruby >= 3.1 required.

== Contributing

Bug reports and pull requests are welcome on GitHub at
https://github.com/lutaml/lutaml-store.

== License

BSD-2-Clause. See the link:LICENSE[] file for details.

Copyright Ribose.

lutaml-store

Runtime

Lutaml::Store

Purpose

Quick start: read a LutaML data repository

Define your models

Create a store with model registry

Save with composite relationships (both stored independently)

Fetch by model and key

Nested update with dot notation

Both accessible independently

Hash-based attributes

Block-based

Define your models

Declare the package schema

Load from directory

Load from ZIP file

Access metadata

Query models

Access assets

Package statistics

Add models

Remove models

Add/remove assets

Save to directory (default format per model)

Save to ZIP with per-model format overrides

Save with global format override

Bulk operations

Global format for all models

Per-model format

Default: each model uses its default_format from the definition

Write models to directory

Read models (returns array, does NOT store in backend)

Read AND store in backend (makes them queryable)

Export to a single file

Fetch with automatic caching

Second call returns cached response (no HTTP request made)