SolidLog

Self-hosted log management for Rails applications using SQLite, PostgreSQL, or MySQL

SolidLog is a modular, Rails-native log ingestion and viewing system that eliminates the need for paid log viewers like Datadog, Splunk, or ELK for the majority of Rails applications. Store logs in your choice of database, search with full-text search, and view in a Mission Control-style UI.

Most Rails applications don't need expensive, complex logging infrastructure. SolidLog provides:

• Zero external dependencies: Everything runs in your Rails app
• Modular architecture: Use only the components you need
• Database flexibility: SQLite, PostgreSQL, or MySQL with adapter-specific optimizations
• Persistent storage: Logs survive container restarts and deployments - all data lives in your database
• HTTP ingestion: Send structured logs from any service via simple HTTP POST
• Full-text search: Database-native FTS (SQLite FTS5, PostgreSQL tsvector, MySQL FULLTEXT)
• Request/job correlation: Trace requests and background jobs across your system
• Mission Control UI: Clean, familiar interface for browsing and filtering logs
• Field promotion: Auto-detect frequently used fields and optimize queries
• Cost: Free, self-hosted, no per-GB pricing

This repository contains three gems that work together:

solid_log/
├── solid_log-core/       # Database models, adapters, parser, services
├── solid_log-service/    # Standalone service for high-volume ingestion
├── solid_log/            # Main gem: core + Rails engine with web UI
├── demo/                 # Demo Rails app showing all 3 gems working together
├── docs/                 # Comprehensive documentation
├── Rakefile              # Run tests for all gems
└── README.md             # This file

Foundation layer - Database models, adapters, services, and background jobs

• Database schema and migrations
• ActiveRecord models (Entry, RawEntry, Token, Field, FacetCache)
• Database adapters (SQLite, PostgreSQL, MySQL)
• DirectLogger: High-performance, batched logging for parent app (50,000+ logs/sec)
• Parser for structured JSON logs
• Core services (SearchService, RetentionService, FieldAnalyzer, BatchParsingService, etc.)
• Background jobs with conditional ActiveJob inheritance (ParseJob, RetentionJob, CacheCleanupJob, FieldAnalysisJob)
• Puma plugin for inline parsing (no separate service needed)
• Railtie for automatic log silencing
• Anti-recursion prevention

Use this gem when: Building custom logging integrations, running inline processing, or using SolidLog without the UI

See solid_log-core README

Service layer - HTTP ingestion API and standalone service runner

• HTTP ingestion API with bearer token authentication
• Built-in scheduler for running core jobs
• Standalone service process (no Rails needed)
• Health check endpoints

Note: Background jobs (ParseJob, RetentionJob, CacheCleanupJob, FieldAnalysisJob) are now in solid_log-core and work with any ActiveJob backend (Solid Queue, Sidekiq, etc.) or can run inline via Puma plugin.

Use this gem when: Running a dedicated log ingestion service separate from your main app

See solid_log-service README

All-in-one gem - Includes core + Rails engine with web interface

• All features from solid_log-core (models, adapters, services, jobs)
• Mission Control-style dashboard
• Log streams with filtering (level, app, env, time range, search)
• Timeline views for request/job correlation
• Field management interface
• Token management
• Live tail support
• ActionCable integration for real-time updates

Use this gem when: You want the complete SolidLog experience (recommended for most apps)

See solid_log README

The fastest way to see SolidLog in action is to run the demo app:

cd demo
bundle install
mkdir -p storage
touch storage/development.sqlite3 storage/development_log.sqlite3
sqlite3 storage/development_log.sqlite3 < db/log_structure.sql
bin/rails server

Visit http://localhost:3000 and click "Generate Sample Logs" to create test data.

See demo/README.md for detailed demo app documentation.

For step-by-step integration instructions, see QUICKSTART.md.

Quick overview:

1. Add gems to your Gemfile:

   gem "solid_log-core", path: "vendor/gems/solid_log-core"
   gem "solid_log-service", path: "vendor/gems/solid_log-service"
   gem "solid_log", path: "vendor/gems/solid_log"

2. Configure multi-database in config/database.yml:

   production:
     primary:
       adapter: sqlite3
       database: storage/production.sqlite3
     log:
       adapter: sqlite3
       database: storage/production_log.sqlite3
       migrations_paths: db/log_migrate

3. Run migrations:

   rails db:migrate

4. Mount the UI in config/routes.rb:

   mount SolidLog::UI::Engine => "/admin/logs"

5. Create an API token:

   rails solid_log:create_token["Production API"]

6. Start sending logs via HTTP POST to the ingestion endpoint

• HTTP Ingestion API: Bearer token authentication, single or batch inserts
• Two-table architecture: Fast writes (raw table) + optimized queries (entries table)
• Multi-database support: SQLite, PostgreSQL, and MySQL adapters with database-specific optimizations
• Full-text search: Database-native FTS (SQLite FTS5, PostgreSQL tsvector, MySQL FULLTEXT)
• Structured logging: Automatic parsing of JSON logs (Lograge format)
• Multi-level filtering: Filter by level, app, environment, time range, or custom fields

• Request correlation: View all logs for a request_id in timeline format
• Job correlation: Track background job execution across workers
• Field registry: Dynamic tracking of all JSON fields with usage stats
• Field promotion: Auto-detect hot fields and promote to indexed columns
• Facet caching: 5-minute cache for filter options to reduce DB load
• Retention policies: Configurable retention with longer periods for errors (30/90 days default)
• Health dashboard: Monitor ingestion rate, parse backlog, error rate, database size

• Mission Control-style dashboard: Familiar Rails UI with stats and health metrics
• Streams view: Filter, search, and browse logs with multiple facets
• Timeline views: Chronological visualization of correlated events
• Field management: View field registry, promote/demote fields
• Token management: Create and manage API tokens for ingestion

SolidLog uses a modular, three-gem architecture with a two-table storage pattern and flexible processing options:

┌─────────────────────────────────────────────────────────┐
│                   Your Rails App                        │
│  ┌──────────────────────────────────────────────────┐  │
│  │  solid_log (mounted at /admin/logs)           │  │
│  │  - Dashboard, Streams, Timeline, Field Mgmt      │  │
│  └────────────────────┬─────────────────────────────┘  │
│                       │                                 │
│  ┌────────────────────▼─────────────────────────────┐  │
│  │  solid_log-core (models, services, jobs)         │  │
│  │  - Entry, RawEntry, Token, Field models          │  │
│  │  - Database adapters, Parser, Services           │  │
│  │  - Background Jobs (ParseJob, RetentionJob...)   │  │
│  │  - Puma Plugin (inline processing option)        │  │
│  │  - OR Solid Queue Jobs (scheduled processing)    │  │
│  └────────────────────┬─────────────────────────────┘  │
└────────────────────────┼──────────────────────────────┘
                         │
                         ▼
              ┌──────────────────────┐
              │   Log Database       │
              │  (:log connection)   │
              ├──────────────────────┤
              │ solid_log_raw        │  ← Fast writes (JSON blobs)
              │ solid_log_entries    │  ← Parsed, indexed, queryable
              │ solid_log_entries_fts│  ← Full-text search
              │ solid_log_fields     │  ← Field registry
              │ solid_log_tokens     │  ← API authentication
              │ solid_log_facet_cache│  ← Performance optimization
              └──────────────────────┘

Optional: Separate Service for High Volume
┌──────────────────────────────────────────┐
│  solid_log-service (standalone process)  │
│  - HTTP Ingestion API                    │
│  - Built-in scheduler runs core jobs     │
│  - No Rails/ActiveJob needed             │
└──────────────┬───────────────────────────┘
               │
               └──> Same Log Database

Data Flow:

1. Logs arrive via DirectLogger (parent app) or HTTP POST (external) → solid_log_raw (append-only, fast)
2. Processing options (choose one):
   • Puma Plugin: Background thread polls for unparsed logs (simplest)
   • Solid Queue: Scheduled recurring job processes batches (scalable)
   • Service: Dedicated process with HTTP API (maximum scale)
3. Parser claims unparsed rows → processes JSON → inserts into solid_log_entries
4. FTS triggers automatically sync full-text search index
5. UI queries solid_log_entries with filters, search, correlation

Benefits:

• Fast ingestion (raw inserts don't block on parsing)
• CPU-intensive parsing doesn't block writes
• Audit trail preserved (raw entries never modified)
• Optimized queries on parsed data
• Flexible processing (inline, scheduled, or dedicated service)
• Jobs in core work with any ActiveJob backend

See docs/ARCHITECTURE.md for detailed design documentation.

Comprehensive guides are available in the repository:

• QUICKSTART.md - Get started in 15 minutes
• demo/README.md - Demo app setup and usage
• docs/API.md - HTTP API reference
• docs/ARCHITECTURE.md - System design and internals
• docs/DATABASE_ADAPTERS.md - SQLite, PostgreSQL, MySQL adapters
• docs/DEPLOYMENT.md - Production deployment guide
• docs/RECURSIVE_LOGGING_PREVENTION.md - How SolidLog prevents logging itself

Individual gem documentation:

• solid_log-core/README.md
• solid_log-service/README.md
• solid_log/README.md

This is a monorepo containing three gems. Each gem has its own test suite.

# Run all tests for all gems
rake test

# Run tests for individual gems
rake test:core
rake test:service
rake test:ui

# Or run tests directly in each gem
cd solid_log-core && bundle exec rake test
cd solid_log-service && bundle exec rake test
cd solid_log && bundle exec rake test

The test suite achieves a 10/10 quality rating with comprehensive coverage:

• 352 tests, 1,131 assertions
• 100% passing across all 3 gems
• Substantive testing: Tests verify actual behavior, not just types
• Edge cases: Boundary conditions, concurrency, error handling
• Security: Token cryptography, authentication, hash validation
• Data integrity: Database constraints, validation

See TEST_QUALITY_REPORT.md for detailed test quality analysis.

solid_log/
├── solid_log-core/
│   ├── lib/                  # Core models, services, adapters
│   ├── test/                 # Test suite (113 tests)
│   ├── solid_log-core.gemspec
│   └── README.md
│
├── solid_log-service/
│   ├── app/
│   │   ├── controllers/      # API controllers
│   │   └── jobs/             # Background jobs
│   ├── lib/                  # Service initialization
│   ├── test/                 # Test suite (127 tests)
│   ├── solid_log-service.gemspec
│   └── README.md
│
├── solid_log/
│   ├── app/
│   │   ├── controllers/      # UI controllers
│   │   ├── views/            # ERB templates
│   │   ├── assets/           # JavaScript, CSS
│   │   ├── helpers/          # View helpers
│   │   └── channels/         # ActionCable channels
│   ├── config/               # Routes, engine config
│   ├── test/                 # Test suite (112 tests)
│   ├── solid_log.gemspec
│   └── README.md
│
├── demo/                     # Full Rails app demonstrating all 3 gems
│   ├── app/controllers/      # Demo controller (log generator)
│   ├── config/               # Rails config, routes
│   ├── db/                   # Log database structure
│   └── README.md
│
├── docs/                     # Comprehensive documentation
│   ├── ARCHITECTURE.md
│   ├── API.md
│   ├── DATABASE_ADAPTERS.md
│   ├── DEPLOYMENT.md
│   └── RECURSIVE_LOGGING_PREVENTION.md
│
├── Rakefile                  # Top-level test runner
├── README.md                 # This file
├── QUICKSTART.md             # Integration guide
├── TEST_QUALITY_REPORT.md    # Test quality analysis
├── MIT-LICENSE
└── .gitignore

Bug reports and pull requests are welcome on GitHub.

Before submitting a PR:

1. Run the full test suite: rake test
2. Ensure all tests pass
3. Add tests for new functionality
4. Update relevant documentation

See individual gem READMEs for gem-specific development notes.

SolidLog stores all logs in your database, ensuring they persist across container restarts, deployments, and rollbacks. As long as your database persists (via volume mounts, managed database services, or SQLite volume persistence), your logs remain available.

SolidLog offers flexible integration options for different scales and architectures:

Run everything in your Rails app with inline processing:

• Add solid_log-core + solid_log to your Gemfile
• Enable the Puma plugin for background parsing
• No separate service or job queue needed
• Background thread polls for unparsed logs every 10 seconds

Setup:

# config/puma.rb
plugin :solid_log if ENV["SOLIDLOG_PUMA_PLUGIN_ENABLED"]

# config/initializers/solid_log.rb
SolidLog.configure do |config|
  config.inline_parsing_enabled = true
  config.parse_interval = 10  # seconds
  config.parser_batch_size = 200
end

Best for: Small to medium Rails apps (<1M logs/day), simple deployments, minimal infrastructure

Use Rails background jobs for processing:

• Add solid_log-core + solid_log to your Gemfile
• Schedule jobs in config/recurring.yml (or use any ActiveJob backend)
• Jobs scale with worker count
• Works with Solid Queue, Sidekiq, Resque, etc.

Setup:

# config/recurring.yml
production:
  solidlog_parse:
    class: SolidLog::Core::Jobs::ParseJob
    schedule: every 10 seconds
    args:
      - batch_size: 200

  solidlog_retention:
    class: SolidLog::Core::Jobs::RetentionJob
    schedule: every day at 2am
    args:
      - retention_days: 30
      - error_retention_days: 90
      - max_entries: 100000

Best for: Medium to large apps, existing job infrastructure, horizontal scaling

Run a separate service process for high-volume ingestion:

• Service app: solid_log-core + solid_log-service (ingestion + parsing)
• Main app: solid_log-core + solid_log (viewing only)
• Service exposes HTTP API for ingestion
• Built-in scheduler (no Rails or ActiveJob needed in service)
• Both apps connect to the same log database

Best for: Very high volume (>10M logs/day), multiple apps, microservices, scaling ingestion independently

Use only solid_log-core for custom implementations:

• Build your own ingestion pipeline
• Use the parser and models directly
• Create custom UI or reporting

Best for: Advanced use cases, non-standard workflows

See docs/DEPLOYMENT.md for production deployment guides including:

• Detailed setup for each integration option
• Kamal configuration
• Database tuning (SQLite WAL mode, PostgreSQL, MySQL)
• Multi-process setup
• Scaling considerations
• Monitoring and alerting

SolidLog is designed for high performance with real-world benchmarks:

• File-based SQLite with WAL: 16,882 logs/sec with crash safety, 56,660 logs/sec without
• Batching: 9x faster than individual inserts
• Crash safety: Eager flush for error/fatal logs (prevents losing crash context)
• PostgreSQL: ~30,000+ logs/sec estimated (2x faster than SQLite)

• SQLite: 5,000-10,000 logs/second
• PostgreSQL: 20,000-50,000+ logs/second
• Batching: Send up to 100 logs per request for best performance

• Parsing: 5,000+ logs/second per worker
• Search: Sub-second FTS queries on millions of entries
• Facet caching: Filter options cached for 5 minutes to reduce load
• Concurrent reads: WAL mode (SQLite) or native (PostgreSQL/MySQL)
• Optimized indexes: Level, app, env, timestamps, correlation IDs

Scaling:

• SQLite handles 100M+ log entries efficiently
• PostgreSQL recommended for >1M logs/day or high-traffic apps
• Enable WAL mode for SQLite - 243% faster for crash-safe logging
• Run multiple parser workers for high ingestion loads
• Use retention policies to manage database size
• Promote hot fields for faster queries

See solid_log-core/BENCHMARK_RESULTS.md for detailed benchmarks and configuration recommendations.

The gem is available as open source under the terms of the MIT License.

Inspired by:

• mission_control-jobs - UI design
• Lograge - Structured logging
• Solid Queue - SQLite-backed Rails services
• Litestream - SQLite replication (recommended for backups)