0.0
The project is in a healthy, maintained state
Provides a unified store interface for Lutaml::Model objects with model registry, polymorphic support, composite relationships, and multiple storage backends (memory, filesystem, SQLite).
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
2026
 Dependencies

Runtime

~> 0.8.15
~> 2.3
 Project Readme

Lutaml::Store

GitHub Stars GitHub Forks License Build Status RubyGems Version

Purpose

Lutaml::Store provides a store-centric database-style API for Lutaml::Model objects with model registry, polymorphic support, composite relationships, and multiple storage backends.

It offers a unified interface for storing and retrieving complex model hierarchies, batch file I/O with multiple serialization formats, HTTP-aware caching, and package-based persistence with ZIP and directory transports.

Features

  • DatabaseStore — high-level CRUD with model registry, polymorphism, composites

  • PackageStore — structured multi-model packages with ZIP and directory transport

  • BasicStore — low-level key-value store with optional cache, events, monitoring

  • CacheStore — TTL-aware cache with LRU eviction extending BasicStore

  • HttpCache — HTTP-aware caching with ETags, conditional requests, Cache-Control

  • Format-aware file I/O — YAML, YAMLS, JSON, JSONL, Marshal with layout strategies

  • Multiple backends — Memory, FileSystem, SQLite (all thread-safe)

  • Model registry — configurable key fields, polymorphic inheritance, composite relationships

  • Dot-notation updates — nested attribute paths with block-based updates

  • Ruby autoload — lazy constant loading, only loads what you use

Installation

Add this line to your application’s Gemfile:

gem 'lutaml-store'

And then execute:

$ bundle install

Or install it yourself as:

$ gem install lutaml-store

For SQLite backend support, also add:

gem 'sqlite3'

Quick start

require 'lutaml/model'
require 'lutaml/store'

# Define your models
class Studio < Lutaml::Model::Serializable
  attribute :studio_key, :string
  attribute :name, :string
  attribute :location, :string
end

class PotteryClass < Lutaml::Model::Serializable
  attribute :studio, Studio
  attribute :class_id, :string
  attribute :description, :string
end

# Create a store with model registry
store = Lutaml::Store.new(
  adapter: :memory,
  models: [
    { model: PotteryClass, key: :class_id },
    { model: Studio, key: :studio_key }
  ]
)

# Save with composite relationships (both stored independently)
pottery = PotteryClass.new(
  class_id: "pottery_101",
  studio: Studio.new(studio_key: "main_studio", name: "Main Studio"),
  description: "Beginner pottery class"
)
store.save(pottery)

# Fetch by model and key
retrieved = store.fetch(model: PotteryClass, class_id: "pottery_101")
puts retrieved.studio.name # => "Main Studio"

# Nested update with dot notation
store.update(
  model: PotteryClass,
  class_id: "pottery_101",
  attributes: [
    { key: :description, value: "Advanced pottery class" },
    { key: "studio.location", value: "Building A" }
  ]
)

Architecture

Lutaml::Store has two layers:

DatabaseStore (via Lutaml::Store.new)

High-level CRUD with model registry. Handles polymorphic dispatch, composite model decomposition, dot-notation updates, file I/O (save_all, load_all, import_all, export).

BasicStore

Low-level key-value store wrapping an adapter. Provides get, set, delete, exists?, all, keys, bulk operations, with optional caching, monitoring, and event emission.

Key classes

Class Role

DatabaseStore

High-level CRUD with model registry, composites, polymorphism

PackageStore

Multi-model packages with directory/ZIP transport

PackageDefinition

Declarative schema for package structure (models, assets, metadata)

BasicStore

Low-level key-value store with optional cache/events/monitoring

CacheStore

TTL-aware cache store extending BasicStore

HttpCache

HTTP-aware caching with ETags, conditional requests, Cache-Control

ModelRegistry / ModelRegistration

Register models with key fields and polymorphic config

CompositeModelHandler

Stores nested registered models independently, restores references

AttributeUpdater

Processes dot-notation paths and block-based updates

ModelSerializer

Serialization/deserialization with custom serializer support

Format

Multi-format file I/O (YAML, JSON, JSONL, Marshal) with layout strategies

Config

Parses and validates store configuration

Storage adapters

All adapters inherit from Adapter::Base and provide get, set, delete, exists?, keys, all, clear, size, each_key, and bulk operations.

Adapter Type symbol Use case

Adapter::Memory

:memory

Fast in-memory storage for testing, caching, temporary data

Adapter::FileSystem

:filesystem

Persistent file-based storage with integrity checks

Adapter::Sqlite

:sqlite

ACID-compliant database storage for production use

Model registry

Registration

Register models with their unique key fields:

store = Lutaml::Store.new(
  adapter: :memory,
  models: [
    { model: User, key: :user_id },
    { model: Post, key: :post_id }
  ]
)

Polymorphic models

For inheritance hierarchies, register the base class with a polymorphic class key:

class Studio < Lutaml::Model::Serializable
  attribute :studio_key, :string
  attribute :name, :string
  attribute :_class, :string, default: -> { "Studio" }, polymorphic_class: true
end

class CeramicStudio < Studio
  attribute :clay_type, :string
  attribute :_class, :string, default: -> { "CeramicStudio" }
end

store = Lutaml::Store.new(
  adapter: :memory,
  models: [
    { model: Studio, key: :studio_key, polymorphic_class_key: :_class }
  ]
)

store.save(CeramicStudio.new(studio_key: "cs1", name: "Clay Haus", clay_type: "Porcelain"))
retrieved = store.fetch(model: Studio, studio_key: "cs1")
puts retrieved.class.name # => "CeramicStudio"

Composite models

When registered models are nested within other registered models, they are stored independently while maintaining references:

store = Lutaml::Store.new(
  adapter: :memory,
  models: [
    { model: PotteryClass, key: :class_id },
    { model: Studio, key: :studio_key }
  ]
)

pottery = PotteryClass.new(
  class_id: "p101",
  studio: Studio.new(studio_key: "s1", name: "Main Studio")
)
store.save(pottery)

# Both accessible independently
store.fetch(model: Studio, studio_key: "s1").name  # => "Main Studio"
store.fetch(model: PotteryClass, class_id: "p101").studio.name  # => "Main Studio"

CRUD operations

Save

store.save(User.new(user_id: "u1", name: "Ada"))
store.save([user1, user2, user3])  # bulk save

Fetch

user = store.fetch(model: User, user_id: "u1")

Update

# Hash-based attributes
store.update(
  model: User, user_id: "u1",
  attributes: [
    { key: :name, value: "Grace" },
    { key: "studio.location", value: "Building A" }  # dot notation for nesting
  ]
)

# Block-based
store.update(model: User, user_id: "u1") do |user|
  user.name = "Grace"
end

Destroy

store.destroy(model: User, user_id: "u1")

PackageStore

PackageStore provides structured multi-model persistence with directory and ZIP transports. Use it when you need to bundle multiple model types, binary assets, and metadata into a single loadable package.

The workflow is: define the package schema with PackageDefinition, then load, query, modify, and save using PackageStore.

Define a package

PackageDefinition declares the package structure — which models, assets, and metadata the package contains:

require "lutaml/model"
require "lutaml/store"

# Define your models
class Concept < Lutaml::Model::Serializable
  attribute :term, :string
  attribute :definition, :string
end

class Author < Lutaml::Model::Serializable
  attribute :name, :string
  attribute :email, :string
end

class GlossaryInfo < Lutaml::Model::Serializable
  attribute :title, :string
  attribute :version, :string
end

# Declare the package schema
glossary = Lutaml::Store::PackageDefinition.new(
  name: "glossary",
  metadata_model: GlossaryInfo,
  metadata_file: "glossary.yaml",
  metadata_key: :title
) do |pkg|
  # Models stored in subdirectories, one file per instance
  pkg.model(model: Concept, key: :term, dir: "concepts", default_format: :yaml)
  pkg.model(model: Author, key: :name, dir: "authors", default_format: :json)

  # Binary assets bundled in the package
  pkg.asset("logo.png", type: :file)
  pkg.asset("attachments", type: :directory)
end

Each pkg.model call registers a model type with:

  • model: — the Lutaml::Model::Serializable class

  • key: — attribute used as the unique identifier (becomes the filename)

  • dir: — subdirectory for this model type (use file: instead for a single-file model)

  • layout::separate (one file per instance, default) or :grouped (multiple instances per file)

  • default_format::yaml, :json, :jsonl, :yamls, or :marshal

Directory structure

When saved to disk, the glossary package produces:

my_glossary/
├── glossary.yaml          # metadata (GlossaryInfo)
├── concepts/              # Concept model instances
│   ├── API.yaml
│   ├── REST.yaml
│   └── YAML.yaml
├── authors/               # Author model instances
│   ├── john_doe.json
│   └── jane_smith.json
├── logo.png               # asset file
└── attachments/           # asset directory
    └── spec.pdf

Load and query

# Load from directory
store = Lutaml::Store::PackageStore.load(glossary, "./my_glossary")

# Load from ZIP file
store = Lutaml::Store::PackageStore.load(glossary, "./glossary.zip", transport: :zip)

# Access metadata
store.metadata  # => GlossaryInfo instance

# Query models
concepts = store.models_for(Concept)   # => array of Concept instances
concept = store.fetch_model(Concept, "API")
store.model_count(Concept)             # => 3
store.model_exists?(Concept, "REST")   # => true

# Access assets
store.asset("logo.png")  # => binary content
store.asset_paths        # => ["logo.png", "attachments/spec.pdf"]

# Package statistics
store.stats
# => { package: "glossary", models: { "Concept" => 3, "Author" => 2 },
#      assets: 2, metadata: true }

Modify and save

# Add models
store.add_model(Concept.new(term: "JSON", definition: "..."))
store.add_models([concept1, concept2])

# Remove models
store.remove_model(Concept, "deprecated_term")

# Add/remove assets
store.add_asset("diagram.svg", svg_content)
store.remove_asset("old_diagram.svg")

# Save to directory (default format per model)
store.save("./output")

# Save to ZIP with per-model format overrides
store.save("./glossary.zip", transport: :zip, formats: { Concept => :json })

# Save with global format override
store.save("./output", format: :yaml)

# Bulk operations
store.clear_models(Concept)  # remove all Concept instances
store.clear_all              # remove everything

Per-model format override

The save method accepts format overrides:

# Global format for all models
store.save("./out", format: :json)

# Per-model format
store.save("./out", formats: { Concept => :yaml, Author => :jsonl })

# Default: each model uses its default_format from the definition
store.save("./out")

Package transports

Transport Symbol Description

DirectoryTransport

:directory

Filesystem directory with subdirectories per model type

ZipTransport

:zip

ZIP archive containing the same directory structure

Transports are resolved via registry (PackageTransport.resolve(:zip)), extensible without modifying existing code.

File I/O

DatabaseStore provides batch file I/O through Format handlers.

Format handlers

Format Symbol Extension Description

YAML

:yaml

.yaml

Single-document YAML files

YAMLS

:yamls

.yaml

Multi-document YAML streams

JSON

:json

.json

Single JSON objects

JSONL

:jsonl

.jsonl

Line-delimited JSON

Marshal

:marshal

.bin

Ruby Marshal binary format

save_all / load_all / import_all / export

# Write models to directory
store.save_all(concepts, path: "./data", format: :yaml, layout: :separate)

# Read models (returns array, does NOT store in backend)
models = store.load_all(Concept, path: "./data", format: :yaml, layout: :separate)

# Read AND store in backend (makes them queryable)
store.import_all(Concept, path: "./data", format: :yaml, layout: :separate)
store.fetch(model: Concept, term: "API")  # now available

# Export to a single file
store.export(all_concepts, path: "output.yaml", format: :yaml)

Layout strategies: :separate (one file per model), :grouped (models grouped by key), :flat (one file per model, no subdirectory).

HTTP caching

HttpCache provides HTTP-aware caching with ETags, conditional requests (304), Cache-Control directives, and Vary header support. It uses a storage adapter internally and serializes cache entries as Lutaml::Model objects via JSON.

cache = Lutaml::Store::HttpCache.new(
  adapter_type: "memory",
  default_ttl: 3600,
  respect_http_headers: true,
  enable_conditional_requests: true
)

# Fetch with automatic caching
response = cache.fetch("GET", "https://api.example.com/resource", {}) do |headers|
  http_client.get("https://api.example.com/resource", headers)
end

# Second call returns cached response (no HTTP request made)
cached = cache.fetch("GET", "https://api.example.com/resource", {}) { raise "shouldn't be called" }

Supports no-store, no-cache, must-revalidate, max-age, ETag-based conditional requests, query parameter normalization, and filesystem/sqlite adapters for persistent cache storage.

CacheStore

CacheStore extends BasicStore with TTL-aware caching and LRU eviction:

cache = Lutaml::Store::CacheStore.new(
  adapter: :memory,
  max_size: 1000,
  default_ttl: 3600
)

cache.set("key1", "value1", ttl: 600)
cache.get("key1")        # => "value1"
cache.fetch("key2", "default_value")  # => "default_value" (stored)
cache.fetch("key3") { expensive_compute }  # computes and caches

Storage backends

Memory

store = Lutaml::Store.new(adapter: :memory, models: [...])

Fast in-memory storage. Data lost on process exit. Thread-safe via mutex.

FileSystem

store = Lutaml::Store.new(
  adapter: { type: :filesystem, path: "./data", extension: ".json" },
  models: [...]
)

Persistent file-based storage with SHA-256 integrity checks. Files organized by key in subdirectories.

SQLite

store = Lutaml::Store.new(
  adapter: { type: :sqlite, path: "./store.db" },
  models: [...]
)

ACID-compliant database storage. Requires sqlite3 gem. Thread-safe with connection pooling.

Event system

store.on(:model_save) { |data| logger.info("Saved #{data[:model].class}") }
store.on(:model_fetch) { |data| logger.debug("Fetched #{data[:key]}") }
store.on(:model_destroy) { |data| audit_log << data }

Events: :model_save, :model_fetch, :model_update, :model_destroy, :model_save_all, :model_import, :model_export, :model_load_error, :composite_model_stored, :polymorphic_model_resolved.

Error handling

Error hierarchy:

  • Lutaml::Store::Error

    • ConfigurationError — invalid store or adapter config

    • BackendError — adapter-level failures

    • ModelNotRegisteredError — operations on unregistered models

    • InvalidKeyError — missing or invalid key fields

    • PolymorphicUpdateError — polymorphic type conflicts

    • CompositeModelError — composite model handling failures

Thread safety

All adapters use mutex-based synchronization. Safe for concurrent use across threads.

Development

bin/setup             # Install dependencies
bundle exec rake      # Run specs + rubocop
bundle exec rspec     # Run all specs
bundle exec rspec spec/lutaml/store/database_store_spec.rb  # Single spec file
bundle exec rubocop   # Lint

Ruby >= 3.1 required.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/lutaml/lutaml-store.

License

BSD-2-Clause. See the LICENSE file for details.

Copyright Ribose.