Huginn

Performant, elegant ActiveRecord datatables and tolerant search.
Huginn is the raven of Odin that represents thought and remembrance — the mate of Muninn.

🇺🇸 English · 🇧🇷 Português

Huginn is a lightweight query layer for Rails that turns a raw datatable request into a lean count, a paginated subset and one preload — instead of a massive JOIN materialized in memory. It also ships a PostgreSQL tolerant-search builder (pg_trgm similarity via the % operator with unaccent and ILIKE fallback, plus a :full_text lexeme strategy) that is tolerant to typos and accents — and that can lean on GIN indexes when present.

• Two-phase execution — association filters/orders/range become reflection-secured subqueries, then a lean count and preload only on the paginated subset.
• Lean counts — the base relation never joins (COUNT(*) over a stripped relation); association matches run as pk subqueries instead.
• SQL injection safe ordering/filtering — every column reference is resolved through Arel reflection, never string-interpolated.
• Tolerant search — pg_trgm similarity (% operator) OR unaccent+ILIKE (typo/acent tolerant) and :full_text over lexemes (morphology-aware); strategies are search_strategy-selectable or combined with OR, backed by GIN indexes when present.
• Rails conventions — works with ActionController::Parameters, Railtie auto-includes both concerns (toggleable), zero boilerplate.

The root Gemfile keeps only the tooling (rspec, appraisal, pry) — each supported Rails series lives in its own Appraisal. Use these to run the suite:

bundle install
bundle exec appraisal install        # generates gemfiles/*.gemfile + resolves
bundle exec appraisal rspec          # runs the full matrix (Rails 7.1/7.2/8.0)
bundle exec appraisal rails-8.0 rspec   # or a single series
bundle exec rake matrix              # alias for the full matrix

A bare bundle exec rspec needs an active environment: export BUNDLE_GEMFILE=gemfiles/rails_8.0.gemfile.

The suite is verified against Rails 7.1, 7.2 and 8.0 across supported Rubies via Appraisal. Run the full matrix locally:

bundle exec appraisal install
bundle exec appraisal rspec

The gemfiles/*.gemfile are generated by Appraisal (committed); their .lock files are not — each CI cell resolves for its own Ruby/Rails pair.

gem "huginn_datatable", require: "huginn"

# config/initializers/huginn.rb
Huginn.configure do |config|
  # :pg_trgm   (recommended) — trigram similarity (%) OR unaccent+ILIKE
  # :full_text — PostgreSQL full text search over lexemes
  #              (to_tsvector @@ plainto_tsquery)
  # :unaccent               — unaccent + ILIKE only
  # :simple                 — plain LIKE
  # Pass an Array to combine strategies with OR, e.g.
  # config.search_strategy = [:pg_trgm, :full_text]
  config.search_strategy = :pg_trgm

  # Unaccent function used around search terms/columns. Defaults to the pg
  # builtin UNACCENT(). Point it at the IMMUTABLE wrapper created by
  # `rails g huginn:trigram_indexes` ("public.f_unaccent") so GIN trigram
  # indexes can actually be used.
  config.unaccent_function = "unaccent"

  # Full text search (strategy :full_text). The dictionary and the IMMUTABLE
  # tsvector wrapper (`rails g huginn:fts_indexes`) pinned to that dictionary;
  # they only apply when the wrapper exists, otherwise search degrades to the
  # unaccent fallback.
  # config.fts_dictionary = "portuguese"
  # config.fts_function   = "public.f_tsvector"

  config.pagy_items = 10         # default page size
  config.pagy_max_items = 500    # hard cap for per_page
end

Threshold: under :pg_trgm the similarity cutoff is the PostgreSQL GUC pg_trgm.similarity_threshold (default 0.3), not a gem-level setting. Tune it with SET pg_trgm.similarity_threshold = 0.4 / SELECT set_limit(0.4) on the database.

Strategy tradeoffs: :pg_trgm tolerates typos and substrings but has no notion of morphology. :full_text matches morphological variants (e.g. "correndo"/"correr" → same stem) but is blind to typos, and shines on longer text columns — for name/keyword lookups it mostly overlaps with trigram. Combining both ([:pg_trgm, :full_text]) unions the two result sets and requires both index sets (rails g huginn:trigram_indexes + rails g huginn:fts_indexes).

By default the Railtie includes Huginn::Datatable and Huginn::Searchable into every ActiveRecord::Base model. You do not need include statements unless you opt in selectively:

Huginn.configure { |c| c.auto_include_datatable = false; c.auto_include_searchable = false }

class Plano < ApplicationRecord
  # Datatable + Searchable are auto-included via Railtie
end

result = Plano.datatable(
  params,
  allowed_paths: [:grupo, { operadora: [:pessoa] }], # associations filters/orders may use
  includes: [{ operadora: { pessoa: [:endereco, :contatos] } }] # preloaded on the page only
)

result[:total_count] # Integer (lean COUNT(*))
result[:data]        # ActiveRecord::Relation (paged + preloaded)

Supported params:

filters also work on array columns (uuid[], text[], string[]...) — own, aliased or association-scoped — with the same payload:

Person.datatable({ filters: { tag_ids: id } })                          # contains id        (tag_ids @> ARRAY[id])
Person.datatable({ filters: [{ tag_ids: a }, { tag_ids: b }] })          # contains a OR b    (tag_ids && ARRAY[a, b])
Person.datatable({ filters: { tag_ids: "null" } })                       # empty array

Several values match any of them by default (like IN). To require all of them:

class Person < ApplicationRecord
  huginn_array_match :tag_ids, :all
  # or, with an alias:
  huginn_attributes tags: { column: "people.tag_ids", array: :all }
end

Values are cast by the element type (an invalid UUID is dropped, never sent to the database) and the array literal is built from quoted values only. Array columns are left out of the default search columns.

Any column or association.column reference is validated and mapped to its real reflected table:

Plano.datatable({ orders: [{ "operadora.pessoa.nome" => "asc" }] }, allowed_paths: [{ operadora: :pessoa }])

Association filters/range, ordering and search use reflection-resolved subqueries (see the "allowlist of associations" section below). The base relation stays singular and join-free, so the count is a plain COUNT(*).

To protect the schema and keep the query lean, the datatable does not materialize left_joins to filter/order by associations. Instead:

• Filters/ranges over association columns become pk IN (SELECT DISTINCT pk …) subqueries — the main relation is never multiplied;
• Ordering by an association column uses a correlated scalar subquery (ORDER BY (SELECT … ORDER BY col ASC LIMIT 1)), which is deterministic even for has_many (smallest value);
• Only authorized associations may be referenced. Pass allowed_paths: with the associations the caller may use in the query:

result = Plano.datatable(
  params,
  allowed_paths: [:grupo, { operadora: :pessoa }],  # associations filters/orders may use
  includes:      [{ operadora: { pessoa: [:endereco, :contatos] } }] # preload only the page
)

• Deny-all by default: without allowed_paths:, no association is authorized for filtering/ordering — only columns of the table itself.
• allowed_paths: accepts the same shapes Rails knows (:symbol, "string", nested Hash, mixed Array). Table names ("companies") are recognized as the matching association (:company).
• includes: stays independent of allowed_paths: — it only controls the preload of the paginated page.

Public APIs should not expose the database schema. Declare a mapping of public names to real columns/tables with huginn_attributes:

class User < ApplicationRecord
  # Public API name -> real column/table (association name or table name)
  huginn_attributes(
    name:         "users.name",
    email:        "users.email",
    created_at:   "users.created_at",
    company_name: "companies.name"   # association column, resolved via subquery
  )
end

• Callers then filter/order/range only by the aliases: { filters: { company_name: "Acme Corp" } }, { orders: [{ company_name: "asc" }] }.
• Strict by default: fields outside the mapping are silently rejected (they never reach SQL and are never answered). The schema stays hidden from API consumers.
• Scoped aliases (companies.name) resolve through the association only if it is authorized in allowed_paths: (the same allowlist applies to aliases).
• Without huginn_attributes, the model falls back to plain/reflected columns (name, company.name).
• Without allowed_paths:, association-scoped aliases are denied; only plain columns are usable.
• huginn_attributes({ ... }, strict: false) keeps alias translation but also accepts raw columns.

# Default: searches every :string / :text column of the model.
Person.search("kayky")            # typo/accent tolerant, case-insensitive

# Override which columns (including through associations) are searched:
class Person < ApplicationRecord
  searchable_columns :name, company: [:name, :cnpj]
end

Person.search("globex")                            # matches company.name via a pk subquery
Person.search("kayky", distinct: false)            # disable the implicit DISTINCT

searchable_columns columns: [] is honored: an empty array means "search only the declared associations" — it no longer falls back to the model's own :string/:text columns:

class Person < ApplicationRecord
  searchable_columns columns: [], company: [:name, :cnpj]  # only company.name/cnpj are searched
end

Huginn::Datatable reuses Huginn::Searchable.search automatically when the model responds to search.

Under :pg_trgm, each searchable column produces an indexable predicate:

(UNACCENT(col) % UNACCENT('term')) OR (UNACCENT(col) ILIKE UNACCENT('%term%'))

Both branches are supported by a GIN trigram index, so the planner can run a BitmapOr on it. Because the pg builtin unaccent() is STABLE (not IMMUTABLE) on PostgreSQL 13+, it cannot be used directly in an index expression — you need an IMMUTABLE wrapper plus an expression index on it:

CREATE OR REPLACE FUNCTION public.f_unaccent(text)
RETURNS text AS $$
  SELECT public.unaccent('public.unaccent', $1);
$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;

CREATE INDEX index_people_name_trgm ON people USING gin (public.f_unaccent(name) gin_trgm_ops);

Then point the gem at the wrapper:

Huginn.configure { |c| c.unaccent_function = "public.f_unaccent" }

Or scaffold the migration for every Searchable model (wrapper + indexes included):

rails g huginn:trigram_indexes                     # all Searchable models
rails g huginn:trigram_indexes Person Product      # specific models

The index is optional — without it the search still returns correct results (via a sequential scan), and the same indexes also accelerate the :unaccent (ILIKE) and :simple (LIKE) strategies.

Notes:

• Requires the pg_trgm and unaccent extensions.
• Only helpful for search terms of 3 or more characters — trigrams need that many to match. Shorter terms always scan.
• To see it working, run a Person.search("term").explain with the wrapper configured and a gin_trgm_ops index present.

Under :full_text, each searchable column produces a lexeme predicate backed by a GIN tsvector index:

public.f_tsvector(col) @@ plainto_tsquery('portuguese'::regconfig, 'term')

Because to_tsvector() is STABLE (not IMMUTABLE), it cannot be indexed directly — you need an IMMUTABLE wrapper that pins the dictionary plus an expression index on it:

CREATE OR REPLACE FUNCTION public.f_tsvector(text)
RETURNS tsvector AS $$
  SELECT to_tsvector('portuguese'::regconfig, $1);
$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;

CREATE INDEX index_people_name_fts ON people USING gin (public.f_tsvector(name));

Then point the gem at the wrapper and dictionary:

Huginn.configure do |c|
  c.search_strategy = :full_text
  c.fts_dictionary  = "portuguese"
  c.fts_function    = "public.f_tsvector"
end

Or scaffold the migration for every Searchable model (wrapper + GIN indexes included), picking a dictionary via --dictionary:

rails g huginn:fts_indexes                               # all Searchable models
rails g huginn:fts_indexes Person Product                # specific models
rails g huginn:fts_indexes --dictionary english          # pin a different dictionary

The index is optional — without the wrapper/index the search degrades gracefully to the unaccent+ILIKE fallback and stays correct (via sequential scan).

Notes:

• Requires the full text search support bundled with PostgreSQL (no extra extension).
• :full_text has no typo tolerance — combine it with :pg_trgm (search_strategy: [:pg_trgm, :full_text]) and generate both index sets (huginn:trigram_indexes + huginn:fts_indexes).
• To see it working, run a Person.search("term").explain with the wrapper configured and a GIN tsvector index present.

phase 1  build the relation          subqueries (pk IN … / ORDER BY (SELECT …)) + search + filters + order   (no data in memory)
phase 2  count                       SELECT COUNT(*) ... (base relation is join-free)
          paginate                   offset / limit
          preload                    SELECT ... WHERE id IN (subset)        (2nd lightweight query)

For a Plano datatable with deep includes:, this is exactly 2 extra queries on the small page instead of one enormous JOIN.

lib/huginn.rb                       entry, Huginn.configure, Huginn.instrument
lib/huginn/configuration.rb         search_strategy, unaccent_function, fts_*, pagy_*
lib/huginn/railtie.rb               auto-includes concerns into ActiveRecord
lib/huginn/datatable.rb             Huginn::Datatable (aggregator)
lib/huginn/datatable/datatable.rb   the datatable Concern
lib/huginn/datatable/validator.rb   column/association validation + Arel resolution
lib/huginn/datatable/association_path.rb   resolution of association chains + correlated subqueries
lib/huginn/datatable/allowed_paths.rb      `allowed_paths:` allowlist expansion/authorization
lib/huginn/datatable/filter_normalizer.rb  functional param normalization
lib/huginn/datatable/paginator.rb    lean count, pagination, isolated preload
lib/huginn/searchable.rb            Huginn::Searchable (aggregator)
lib/huginn/searchable/searchable.rb the search Concern + DSL
lib/huginn/searchable/query.rb      tolerant search builder (subqueries + OR)
lib/huginn/searchable/fuzzy.rb      pg_trgm / full_text / unaccent / simple predicates
lib/generators/...                  `huginn:trigram_indexes` + `huginn:fts_indexes` (GIN index migrations)

Huginn.instrument wraps ActiveSupport::Notifications events under the huginn namespace (e.g. datatable.call.huginn). Subscribe with ActiveSupport::Notifications.subscribe(/\.huginn/).

MIT