Kotoshu: Semantic Spell Checking for Ruby

Status

Kotoshu is v1.0.8. The wave-2 benchmark program is closed: across 16 languages, kotoshu matches or beats the strongest open word-level baselines — Hunspell and SymSpell over the same published frequency lists — on nonword top-1, and is the only engine with realword signal in most of them (Arabic 26.5 vs 0.0, German 17.0 vs 0.0, Spanish 14.0 vs 0.0). Every number is a committed frozen report; the full matrix: Wave-2 results — 16 languages, 16 wins

New in 1.0.7 / 1.0.8

Cross-script retrieval — type the romanization, get the native word: mrhb → مرحبا, privet → привет, kalemera → καλημέρα (ar + ru + el; more following). Measured: 78.9% top-1 on 2,000 romanization pairs where SymSpell scores 2.4% — a capability class no baseline lane can enter.
Vowelless-script normalization — vocalized Arabic and Hebrew input (مُحَمَّد, נִקּוּד) now suggests and corrects correctly.
Variant-pure Chinese — zh-Hant-HK / zh-Hant-TW / zh-Hans-CN each rank their own vocabulary; no script- or region-mixed lists.
Cleaner slates — suggestions no longer repeat the same word from multiple strategies.

What works (carried from 0.3)

Two-stage resource model — Kotoshu.setup(:en) then Kotoshu.correct?("hello"). The hot path is cache-only and raises ResourceNotSetupError on miss; downloads are never implicit.
Kotoshu.check(text, language: "en") / Kotoshu.suggest("helo") — full document check and suggestions
Kotoshu.spellchecker_for(lang) — per-language checker resolved from the cache (strict: is a Kotoshu.setup option)
kotoshu check FILE CLI with these flags:
- --language LANG|auto — any set-up language code, script variants included (sr-Latn) (default: auto)
- --format text|json|sarif
- --offline — use only cached resources, never download
- --strict — exit 3 if any optional resource (frequency, model) can’t load
- --interactive — review each error after the check
- --verbose
kotoshu setup LANGUAGE [LANGUAGE …] — pre-warm spelling + frequency + ONNX caches for offline use (fetch is kept as a hidden deprecated alias)
Local-source setup: kotoshu setup en --aff path/to.en.aff --dic path/to.en.dic or kotoshu setup en --from /path/to/dict/dir/
Exit codes: 0 clean, 1 errors found, 2 usage error, 3 resource setup failed
SHA-256 integrity verification (manifest-based, with graceful degradation when manifest is absent)
Offline mode via KOTOSHU_OFFLINE=1 or --offline
XDG base directory layout — caches in $XDG_CACHE_HOME/kotoshu/, config in $XDG_CONFIG_HOME/kotoshu/, data in $XDG_LOCAL_HOME/kotoshu/ (overridable via KOTOSHU_CACHE_PATH, KOTOSHU_CONFIG_PATH, KOTOSHU_DATA_PATH)

Planned for 0.4+

--output (file output redirection)
ONNX semantic reranking as default path
≥30 language modules wired
Grammar rule packs
CJK and RTL language support

See the 0.2 cut plan, the 0.3 tasks under TODO.impl/, and the vision for the path to 1.0.

Purpose

Kotoshu 「言修」 is a pure-Ruby spell checker that aims to work for every language by dynamically downloading the right combination of dictionary, frequency data, and embedding model on demand.

The current release pairs a Ruby port of the Hunspell algorithm (traditional morphological lookup + affix rules) with optional FastText word embeddings converted to ONNX for context-aware reranking.

Note	The semantic (ONNX) path is an optional feature. `gem install kotoshu` works without `onnxruntime`; install it separately (`gem install onnxruntime`) to enable context-aware reranking. Set `KOTOSHU_NO_ONNX=1` to opt back out.

Features

Note	The list below describes the design vision. See Status for exactly what works in 0.2 and what is planned for 0.3+.

Multi-language support with automatic detection
Semantic error detection using word embeddings (opt-in via Ruby API in 0.2)
Interactive review mode with full navigation (planned for 0.3)
Batch processing for CI/CD (JSON in 0.2; SARIF planned for 0.3)
Fast ONNX inference via ONNX Runtime
Support for Markdown, AsciiDoc, and plain text
Multiple analysis models (Hunspell, FastText, Hybrid) (Hunspell path only in 0.2)

Architecture

Kotoshu is built on a modern, semantic architecture:

Architecture overview

╔═══════════════════════════════════════════════════════════════════╗
║                    Kotoshu Semantic Architecture                ║
╠═══════════════════════════════════════════════════════════════════╣
║                                                                   ║
║  ┌─────────────────────────────────────────────────────────────┐  ║
║  │                     Interface Layer                          │  ║
║  │  ┌─────────────────────┐  ┌─────────────────────────────┐   │  ║
║  │  │   CLI (Thor)        │  │      Ruby API               │   │  ║
║  │  │   lib/kotoshu/cli/  │  │   Kotoshu module methods    │   │  ║
║  │  └──────────┬──────────┘  └───────────┬─────────────────┘   │  ║
║  │             │  Auto Language Detect   │                     │  ║
║  └─────────────┼──────────────────────────┼─────────────────────┘  ║
║                │                          │                        ║
║                ▼                          ▼                        ║
║  ┌─────────────────────────────────────────────────────────────┐  ║
║  │                   Analysis Layer                             │  ║
║  │  ┌──────────────┐  ┌─────────────┐  ┌───────────────────┐  │  ║
║  │  │    Hunspell  │  │  FastText   │  │  Hybrid (Best!)   │  │  ║
║  │  │  Dictionary  │  │  Embeddings │  │  Combined         │  │  ║
║  │  │  (Traditional)│  │  (ONNX)    │  │  Approach         │  │  ║
║  │  └──────────────┘  └─────────────┘  └───────────────────┘  │  ║
║  └───────────────────────────┬─────────────────────────────────┘  ║
║                              │                                    ║
║  ┌───────────────────────────▼─────────────────────────────────┐  ║
║  │                  Model Layer (ONNX)                          │  ║
║  │  ┌──────────────────────────────────────────────────────┐  │  ║
║  │  │  ONNX Runtime → Fast Embedding Lookup                 │  │  ║
║  │  │  Semantic Similarity → Context-Aware Suggestions      │  │  ║
║  │  │  Nearest Neighbor Search → Smart Corrections          │  │  ║
║  │  └──────────────────────────────────────────────────────┘  │  ║
║  └─────────────────────────────────────────────────────────────┘  ║
║                                                                   ║
╚═══════════════════════════════════════════════════════════════════╝

Key Components

Kotoshu::Models::OnnxModel: ONNX-based word embedding model for fast semantic similarity and nearest neighbor search.
Kotoshu::Analyzers::SemanticAnalyzer: Unified semantic error detection using word embeddings (no artificial spelling/grammar split).
Kotoshu::Language::LanguageIdentifier: Automatic language detection using FastText LID model (127 languages).
Kotoshu::Cli::InteractiveReviewer: Interactive CLI for error review with full navigation (forward, backward, jump, skip, accept).
Kotoshu::Dictionary::Hunspell: Traditional Hunspell dictionary backend for morphological analysis and affix rules.

Why ONNX?

ONNX Runtime provides:

Performance: C++ implementation, 10-100x faster than pure Ruby
Portability: Works on CPU, GPU, TPU, mobile devices
Optimization: Automatic graph optimization and quantization
Interoperability: Models can be trained in Python, deployed in Ruby

Kotoshu uses FastText models converted to ONNX format for semantic spell checking.

Semantic Analysis

Unlike traditional spell checkers that only check dictionary membership and edit distance, Kotoshu uses semantic similarity to:

Detect contextually appropriate corrections ("desert" vs "dessert")
Handle out-of-vocabulary words via subword embeddings
Provide ranked suggestions based on semantic similarity
Support compound words and morphological variations

Example 1. Usage example

Kotoshu.setup(:en, want: %i[spelling model])  # one-time per language

# Traditional: knows "helo" is wrong and lists edit-distance candidates
Kotoshu.suggest("helo").to_words
# => ["hello", "help", "held", "hell", "hole"]

# Semantic: reranks candidates by context similarity
model = Kotoshu::Models::OnnxModel.from_github("en")
analyzer = Kotoshu::Analyzers::SemanticAnalyzer.new(model)
analyzer.suggest_corrections("helo", context: "I said helo to the world").map(&:word)
# => ["hello"]  # "hello" makes more sense in greeting context

Note	The semantic path requires the optional `onnxruntime` gem. See Requirements.

Confidence cascade

When the traditional (dictionary + edit distance) strategies are already confident about a word, the semantic rerank adds latency and model load without changing the answer. The confidence cascade formalizes the hybrid split: dictionary verdict first, ONNX rerank only for uncertain candidates.

The semantic_cascade_threshold configuration option controls it:

Threshold	Behavior
`1.0` (default)	Always rerank — identical to the classic pipeline. This is the sentinel: the composite can legitimately score exactly `1.0`, so `1.0` never skips.
`0.9`	Rerank only when the top traditional candidate scores below `0.9`; otherwise skip the ONNX rerank for that word.
`0.0`	Never rerank — the semantic path is bypassed entirely.

# Tune via environment variable
KOTOSHU_SEMANTIC_CASCADE_THRESHOLD=0.9 kotoshu check document.txt

# Or via the Ruby API
Kotoshu.configuration.semantic_cascade_threshold = 0.9

Each skip is counted in the metrics registry (Kotoshu::Metrics.stats[:semantic_cascade_skips], also exported in StatsD and Prometheus formats) and logged under debug/verbose output.

Tuning guidance will come from the eval harness once the corpus benchmark lands; until then, 0.8–0.9 is a reasonable range for typo-heavy text.

Multi-Language Support

Kotoshu serves every language staged in the dictionaries manifest — 36 full-feature codes with per-language modules (script-aware tokenizer, national keyboard grid, engine-verified specimens) and the script-fallback basic tier for the rest; see the Language Support Matrix for the breakdown. Semantic reranking adds an ONNX model where the models registry pins one (50+ languages), for example:

de - German (Deutsch)
en - English
es - Spanish (Español)
fr - French (Français)
pt - Portuguese (Português)
ru - Russian (Русский)

Automatic language detection is enabled by default:

Example 2. Usage example

# Language auto-detected from document content
kotoshu check document.txt
# Detected: en (95% confidence)
# Analyzing document.txt (language: en)...

# Explicit language specification
kotoshu check document.txt --language de

ONNX Models

Kotoshu uses FastText crawl vectors converted to ONNX format:

Source: FastText Crawl Vectors
Format: ONNX with optimized runtime
Vocabulary: 2 million words per language (full coverage)
Dimension: 300-dimensional word vectors
Size: ~2.4GB per language

FastText File Formats

FastText provides two file formats. Kotoshu uses the .vec format for ONNX conversion.

Aspect	`.vec` (Text)	`.bin` (Binary)
Content	Word vectors only (pre-computed embeddings)	Full FastText model (trained model)
Structure	Text: one word + 300 floats per line	Binary: complete model with matrices
File Size	~1.3GB compressed (~2.4GB uncompressed)	~1.8GB compressed (~4.8GB uncompressed)
Train New Words	✗ No (static lookup only)	✓ Yes (can train/OOV with subword info)
Subword Embeddings	✗ No	✓ Yes (n-gram character embeddings)
ONNX Converter	✓ Supported (what we use)	✗ Not supported
Use Case	Simple word vector lookup for spell checking	Full FastText functionality (training, OOV)

Kotoshu uses .vec files because:

Simpler extraction: Just word → vector mapping
No subword complexity needed: Dictionary-based spell checking doesn’t require OOV generation
Smaller ONNX models: ~2.4GB vs ~4.8GB
Faster conversion: Direct serialization to ONNX

Example 3. Model management

# Set up a language with spelling + ONNX semantic model
kotoshu setup en --want spelling,model

# List what's set up in the cache
kotoshu setup --list

# Re-validate cached resources
kotoshu cache validate

Note	FastText `.vec` → ONNX conversion is done upstream in the `kotoshu/models-fasttext-onnx' repo. The CLI downloads pre-converted artifacts; users do not run conversion locally.

Model tiers

ONNX models come in three tiers, published by models-fasttext-onnx in a resource registry under stable ids (kotoshu://models/{lang}/{tier}):

Tier	Approx. size	What it is
`full`	~120 MB	Unquantized, ~100k-word vocabulary. The same artifact and cache layout as before tiers existed.
`fluency`	~15-18 MB	int8-quantized, reduced vocabulary (~50k-60k words).
`mini`	~3 MB	int8-quantized, 10k-word vocabulary.

Tiered downloads are verified: the model bytes are fetched from the registry entry’s primary URL, falling back to its mirror, with the vocab sibling downloaded alongside; the SHA-256 is checked against the registry entry before the model is accepted into the cache.

Example 4. Picking a tier

# One-time setup at a specific tier
Kotoshu.setup(:en, want: %i[model], tier: :fluency)

# Resolve stays cache-only: the requested tier must already be cached,
# otherwise ResourceNotSetupError is raised (no download, no fallback
# to another tier)
Kotoshu.resolve(language: :en, want: %i[spelling model], tier: :fluency)

# "Is this tier cached?" (tier: :any asks whether any tier is cached)
Kotoshu.setup?(:en, resource: :model, tier: :mini)

kotoshu setup en --model --tier mini

tier: :any on resolve maps to the single cached tier. It resolves when exactly one tier is cached and raises otherwise: with no tier cached it raises ResourceNotSetupError, and with several it raises Kotoshu::ResourceResolutionError listing them. That listing uses the fixed order mini, fluency, full — a reporting order for deterministic error messages, not a fallback chain; tiers are never silently substituted for one another.

The default tier is fluency (~15 MB, near-lossless; owner decision 2026-09-04) — caches written under the old full default keep working via the tier-less legacy bridge. Override it per call with tier:, or set it globally via the model_tier configuration option (environment variable KOTOSHU_MODEL_TIER; one of full, fluency, mini):

KOTOSHU_MODEL_TIER=mini kotoshu setup en --model

Typo Retrieval (opt-in)

The typo-retrieval layer adds a 0.5 MB character bi-encoder to the suggestion pipeline: it retrieves the twenty vocabulary entries nearest a misspelling’s character shape, and the language’s FULL tier rescores them exactly. On the frozen real-pair benchmarks it lifts top-5 recall over the full tier alone (+6.3 points on English, +10 to +29 on de/es/ru).

Setup fetches everything with one flag:

kotoshu setup en --typo

The layer arms only when you opt in with KOTOSHU_TYPO_RETRIEVAL=1; without it, suggestions are byte-identical to the base pipeline. A prebuilt 26 MB retrieval matrix is served by the model registry, so arming is a download (about half a second) instead of a 25-second derivation. The matrix rows pair with the language’s full-tier vocabulary, so keep the full tier cached (setup fetches it).

Interactive Mode

Note	Interactive mode shipped in 0.3.0. It is navigation-only — the session records which suggestions the user accepted but does not rewrite the source file yet.

kotoshu check README.md --interactive

Features in 0.3:

Navigate: [n] / Enter next, [p] previous, [l] list
Accept: [1-9] record suggestion N for the current error
Skip: [s] skip the current error
Quit: [q] exit the review loop

Batch Processing

For CI/CD and automation, Kotoshu supports JSON and SARIF output in 0.3; --output file redirection is planned for 0.4+.

Example 5. JSON output for CI/CD

# JSON output to stdout (supported in 0.3)
kotoshu check README.md --format json

# SARIF 2.1.0 output (supported in 0.3)
kotoshu check README.md --format sarif

# Exit code for CI
kotoshu check README.md
echo $?  # 0 if no errors, 1 if errors found

Ignoring Words Inline

Words that are intentionally "wrong" - product names, code identifiers quoted in prose, dialect - can be suppressed in place instead of being added to the personal dictionary:

kotoshu:disable-line                 suppress everything on this line
kotoshu:disable-next-line            suppress the following line
kotoshu:disable-next-line foo bar    suppress only these words
kotoshu:disable-file / enable-file   block suppression (nestable)

Each format recognizes the directive in its own comment syntax, and the directive line itself is never spellchecked:

Format	Form	Trailing directive
Plain text	a bare line: `kotoshu:disable-line`	not supported
Markdown	`<!-- kotoshu:disable-next-line -→`	supported
AsciiDoc	`// kotoshu:disable-next-line`	supported

The Ruby API picks this up automatically: Kotoshu.spellchecker_for("en").check(text) moves suppressed words to result.suppressed_errors (each marked suppressed: true and suppressed_by: "inline") so only real errors stay in result.errors. Pass --show-suppressed to kotoshu check to list them.

CI Baselines

Freeze existing spelling debt so CI fails only on NEW errors:

# Record current findings: one entry per file+word with a count,
# stable across reformatting
kotoshu baseline init README.md docs/*.adoc

# CI: errors covered by the baseline pass, new errors fail
kotoshu check README.md --baseline .kotoshu-baseline.json

A summary line reports baseline entries that have gone stale (the error no longer exists), so the debt shrinks visibly; refresh by re-running kotoshu baseline init. JSON output adds suppressedCount / suppressedErrors plus a baseline block, and SARIF marks baseline-suppressed results with a suppressions entry (justification: "baseline").

Git pre-commit Hook

The gem repository ships a pre-commit hook definition (.pre-commit-hooks.yaml):

repos:
  - repo: https://github.com/kotoshu/kotoshu
    rev: v0.7.0   # pin to a released tag
    hooks:
      - id: kotoshu

Note	The hook uses `language: system`: it runs the `kotoshu` executable from PATH, so it honestly requires a Ruby environment with the gem installed (`gem install kotoshu`, then `kotoshu setup en`). A WebAssembly client (in development) is the future zero-Ruby path.

Document Formats

Kotoshu supports structured documents with AST parsing:

Plain text: Line-based error detection
Markdown: AST-based using Kramdown parser
AsciiDoc: AST-based using Asciidoctor parser

Structured documents preserve node paths for precise error location.

Analysis Models

Note	In 0.2, the CLI runs the Hunspell traditional path only. The `--model` flag and FastText/Hybrid paths are planned for 0.3+. The Ruby API can opt into the semantic path today via `Kotoshu::Models::OnnxModel` (auto-available when `onnxruntime` is installed).

Kotoshu is designed to support three analysis models:

Table 1. Dictionary backend comparison

Model	Description	Best For
hunspell	Traditional dictionary-based with morphological rules	Fast checking, compound words, languages with complex morphology
fasttext	Pure semantic embeddings via ONNX	Context awareness, out-of-vocabulary words, semantic similarity
hybrid	Hunspell candidates + FastText reranking (recommended)	Maximum accuracy, best of both worlds

Example 6. Intended usage (0.3+)

# Fast dictionary-based checking (default in 0.2)
kotoshu check document.txt                # 0.2: Hunspell path

# Semantic / hybrid paths: planned for 0.3
# kotoshu check document.txt --model fasttext
# kotoshu check document.txt --model hybrid

Installation

Add this line to your application’s Gemfile:

gem 'kotoshu'

And then execute:

bundle install

Or install it yourself as:

gem install kotoshu

Note	`onnxruntime` is an optional dependency. Install it separately (`gem install onnxruntime`) to enable semantic analysis; the traditional Hunspell path works without it.

Quick Start

# One-time per language: download spelling dictionary from
# github.com/kotoshu/dictionaries (idempotent, ~5 MB)
kotoshu setup en

# Then check files instantly, cache-only
kotoshu check README.md

Or skip the explicit setup — the CLI will prompt interactively the first time you check a file in a non-cached language (TTY only; in non-TTY or KOTOSHU_OFFLINE=1 mode it exits with code 3).

Command-line usage

# Check a file (uses --language, or auto-detects from content)
kotoshu check README.md

# Explicit language
kotoshu check README.md --language en

# JSON output for programmatic use
kotoshu check README.md --format json

# Offline mode — use only cached dictionaries, never download
kotoshu check README.md --offline

# Check stdin
echo "helo wrld" | kotoshu check

Exit codes: 0 (no errors), 1 (errors found), 2 (usage error), 3 (language not set up — run kotoshu setup LANG, or run kotoshu check in a TTY to be prompted).

Ruby API usage

require 'kotoshu'

# Stage 1: set up the language once (downloads from github.com/kotoshu/dictionaries)
Kotoshu.setup(:en)

# Stage 2: hot-path checks are cache-only and never touch the network
Kotoshu.correct?("hello")  # => true
Kotoshu.correct?("helo")   # => false

# Suggestions return a SuggestionSet; call #to_words for an Array
Kotoshu.suggest("helo").to_words  # => ["hello", "help", "held", ...]

# Check a document
result = Kotoshu.check("Hello wrold")
result.errors.map(&:word)  # => ["wrold"]

# Each error carries position + suggestions
result = Kotoshu.check_file("README.md")
result.errors.each do |error|
  puts "#{error.word} at offset #{error.position}: #{error.top_suggestions(3).join(', ')}"
end

# Semantic analysis is optional — requires the onnxruntime gem
# (gem install onnxruntime). Skip this block if you only want Hunspell.
if Kotoshu::Models::OnnxModel::ONNX_LOADED
  Kotoshu.setup(:en, want: %i[spelling model])
  model = Kotoshu::Models::OnnxModel.from_github('en')
  analyzer = Kotoshu::Analyzers::SemanticAnalyzer.new(model)
  analyzer.analyze(Kotoshu.check("Hello wrold"))
end

Note	The library API is strict: calls like `Kotoshu.correct?` raise `Kotoshu::ResourceNotSetupError` until you’ve run `Kotoshu.setup`. This prevents surprise downloads on metered networks. The CLI (`kotoshu check`) intercepts the error and prompts to download interactively.

Requirements

Ruby 3.1+
onnxruntime gem (optional — enables semantic spell checking; install separately with gem install onnxruntime)
Python 3 + fasttext (optional, only if you want to convert .vec → .onnx upstream)

Resource Caching and Language Support

Kotoshu uses a sophisticated multi-layer caching system to manage dictionaries, frequency lists, and embedding models. Resources are downloaded explicitly via Kotoshu.setup (or kotoshu setup) and cached under the XDG base directory layout (~/.cache/kotoshu/ by default; override via KOTOSHU_CACHE_PATH, KOTOSHU_CONFIG_PATH, KOTOSHU_DATA_PATH, or the XDG_*_HOME vars).

Cache Architecture

Cache System Class Diagram

┌────────────────────────────────────────────────────────────────────────────┐
│                              BaseCache (Abstract)                        │
│  ┌────────────────────────────────────────────────────────────────────┐   │
│  │ Common: download, metadata, validation, stats, TTL management     │   │
│  └────────────────────────────────────────────────────────────────────┘   │
└────────────────────┬───────────────────┬────────────────────┬──────────────┘
                     │                   │                    │
        ┌────────────▼────────┐  ┌──────▼──────┐  ┌───────▼─────────┐
        │   LanguageCache     │  │ModelCache   │  │ FrequencyCache  │
        │  (Dictionaries)     │  │ (Embeddings)│  │  (Kelly Lists)  │
        └─────────────────────┘  └─────────────┘  └─────────────────┘
                     │                   │                    │
        ┌────────────▼────────┐  ┌──────▼──────┐  ┌───────▼─────────┐
        │ ~/.cache/kotoshu/  │  │~/.cache/    │  │ ~/.cache/kotoshu/│
        │   languages/       │  │  kotoshu/   │  │frequency-lists/ │
        │                    │  │  models/    │  │                 │
        └─────────────────────┘  └─────────────┘  └─────────────────┘

Cache Types

LanguageCache (Dictionaries)

Manages Hunspell dictionaries and grammar rules for spell checking.

Cache Path: ~/.cache/kotoshu/languages/{code}/
TTL: 7 days (604,800 seconds)
Source: kotoshu/dictionaries
Resources per language:
- spelling/**: Hunspell dictionary (index.dic, index.aff)
- grammar/*: Grammar rules (rules.yaml) - *future
- frequency/*: Frequency data - *deprecated, use FrequencyCache

Usage

# Access via cache
cache = Kotoshu::Cache::LanguageCache.new
dict = cache.get_spelling('en')

# Result:
# {
#   aff_path: "~/.cache/kotoshu/languages/en/spelling/index.aff",
#   dic_path: "~/.cache/kotoshu/languages/en/spelling/index.dic",
#   cached: true,
#   metadata: { ... }
# }

FrequencyCache (Kelly Project)

Manages Kelly Project frequency lists for intelligent suggestion ranking.

Cache Path: ~/.cache/kotoshu/frequency-lists/{code}/
TTL: 7 days (604,800 seconds)
Source: kotoshu/frequency-list-kelly
Format: JSON with tiered word frequency data

Kelly Frequency Data Structure

{
  "metadata": {
    "language": "en",
    "source": "Kelly Project (University of Leeds)",
    "total_words_analyzed": 1500000
  },
  "tiers": {
    "top_50": {
      "words": ["the", "be", "to", "of", "and", ...],
      "info": "Most common 50 words"
    },
    "top_200": {
      "words": ["will", "my", "one", "all", ...],
      "info": "Most common 200 words"
    },
    "top_1000": {
      "words": ["however", "although", ...],
      "info": "Most common 1000 words"
    }
  }
}

Usage

# Access via cache
cache = Kotoshu::Cache::FrequencyCache.new
freq_data = cache.get('en', force_download: true)

# Result:
# {
#   frequency_path: "~/.cache/kotoshu/frequency-lists/en/frequency.json",
#   tiers: {
#     top_50: Set<...>,
#     top_200: Set<...>,
#     top_1000: Set<...>
#   },
#   metadata: { ... }
# }

# Integrated into EditDistanceStrategy
strategy = Kotoshu::Suggestions::Strategies::EditDistanceStrategy.new(
  language_code: 'en'
)
strategy.frequency_bonus('the')   # => 200 (top 50)
strategy.frequency_bonus('hello') # => 100 (top 200)
strategy.frequency_bonus('xyz')   # => 0 (not in lists)

ModelCache (Embedding Models)

Manages FastText and ONNX embedding models for semantic spell checking.

Cache Path: ~/.cache/kotoshu/models/{code}/models/{type}/
TTL: 30 days (2,592,000 seconds)
Sources:
- FastText (.vec): Facebook CDN (dl.fbaipublicfiles.com)
- ONNX (.onnx): Converted locally from FastText models
Supported Types:
- fasttext: FastText word vectors (.vec files, 300D) - Downloaded from Facebook CDN
- onnx: ONNX-converted models (.onnx files) - Auto-converted from FastText

Note	ONNX models are automatically converted from FastText models on first use. The conversion uses `lib/kotoshu/scripts/fasttext_to_onnx.py` and requires Python 3 with `numpy` and `onnx` packages installed.

Table 2. Model Files by Language

Language	`FastText File`
ONNX File

CLI Cache Management

Kotoshu provides CLI commands for managing cached resources:

# List all cached resources
kotoshu cache list

# List specific cache type
kotoshu cache list language
kotoshu cache list model
kotoshu cache list frequency

# Show cache statistics
kotoshu cache status

# Show detailed status (verbose)
kotoshu cache status --verbose

# Download a resource
kotoshu cache download language en
kotoshu cache download model en:fasttext
kotoshu cache download frequency en

# Get information about a resource
kotoshu cache info language en
kotoshu cache info model en:fasttext
kotoshu cache info frequency en

# Purge cached data
kotoshu cache purge all
kotoshu cache purge language en
kotoshu cache purge frequency

# Clean expired entries
kotoshu cache clean

Cache Statistics

Each cache type tracks statistics:

Hits: Number of cache hits (resource found locally)
Misses: Number of cache misses (had to download)
Hit Rate: Percentage of cache hits
Size: Total disk space used
Cached Resources: Number of resources cached

$ kotoshu cache status
======================================================================
Kotoshu Cache Status
======================================================================

Language Cache:
  Directory: /Users/username/.cache/kotoshu/languages
  Resources cached: 2
  Size: 2.45 MB
  Hits: 15, Misses: 2
  Hit rate: 88.2%

Frequency Cache:
  Directory: /Users/username/.cache/kotoshu/frequency-lists
  Resources cached: 1
  Size: 815.84 KB
  Hits: 42, Misses: 1
  Hit rate: 97.7%

Model Cache:
  Directory: /Users/username/.cache/kotoshu/models
  Resources cached: 0
  Size: 0 B
  Hits: 0, Misses: 0
  Hit rate: 0.0%

Total:
  Total size: 3.26 MB
  Overall hit rate: 93.5%
======================================================================

Shell Completion

Kotoshu ships tab-completion scripts for bash, zsh, and fish. The scripts complete subcommands for kotoshu <TAB> and language codes for kotoshu setup <TAB> (the language list is read dynamically via kotoshu completions languages, so newly registered languages appear without re-running the install step).

Install for your shell:

# bash (system-wide)
kotoshu completions bash > /etc/bash_completion.d/kotoshu

# bash (single user)
kotoshu completions bash > ~/.local/share/bash-completion/completions/kotoshu

# zsh — drop into any directory on your $fpath
kotoshu completions zsh > "${fpath[1]}/_kotoshu"

# fish
kotoshu completions fish > ~/.config/fish/completions/kotoshu.fish

To see the supported language codes without installing a completion script:

kotoshu completions languages

Language Support Matrix

Kotoshu serves languages in tiers. Every language staged in the kotoshu/dictionaries manifest downloads through kotoshu setup and checks through the Hunspell engine. Region variants fold onto their base dictionary (en-GB resolves en, es-MX resolves es); script variants keep a dictionary of their own (sr-Latn stages the Latin Serbian dictionary alongside the Cyrillic sr).

Table 3. Language Support Matrix

Tier	Languages	What it carries

| Full feature (35 languages, 36 codes) | ar bg ca cs da de el en es et fa fr he hr hu id it ko lt lv nb ne nl nn pl pt ro ru sk sl sr sr-Latn sv tr uk vi | Per-language module: staged dictionary + Hunspell affix rules, script-aware tokenizer, national keyboard grid for typo-proximity suggestions, engine-verified specimens | | Basic (26 languages) | br cy eo eu fo fur fy ga gd gl hy hyw ia ie is ka la lb ltg mk mn nds oc rw tk tlh | Staged dictionary + script-fallback word extraction and the generic keyboard | | Kelly frequency (8 languages) | ar el en it no ru sv zh | Top-1000 frequency list wired into suggestion ranking, on top of either tier | |=========================================+

ONNX semantic models (opt-in, --model) resolve per the registry the models repo pins — 50+ languages across the tiers, in full/fluency/mini sizes.

Table 4. Dictionary Sources

Language	Word Count	License	Source

Staged dictionaries for the other tiers carry their provenance in the dictionaries repo manifest (LICENSE and source per file; LibreOffice / wooorm-dictionaries derivations dominate).

Table 5. Kelly Frequency Lists

Language	Size	Coverage

Note	Kelly frequency lists provide the top 1000 most common words from the Kelly Project (University of Leeds & University of Gothenburg). Languages not listed here require external frequency data sources.

Programmatic Usage

Using Language Cache

require 'kotoshu/cache/language_cache'

cache = Kotoshu::Cache::LanguageCache.new

# Get spelling dictionary
dict = cache.get_spelling('en')
puts "Dictionary: #{dict[:dic_path]}"
puts "Words: #{File.readlines(dict[:dic_path]).count}"

# Get available languages
cache.available_languages  # => ["de", "en", "es", "fr", "pt", "ru"]

# Check if resource is cached
cache.available?('en:spelling')  # => true

# Get language info
info = cache.language_info('en')
puts "Language: #{info[:name]}"
puts "Words: #{info[:word_count]}"
puts "License: #{info[:license]}"

Using Frequency Cache

require 'kotoshu/cache/frequency_cache'

cache = Kotoshu::Cache::FrequencyCache.new

# Get frequency data
freq_data = cache.get('en')

# Access frequency tiers
top_50 = freq_data[:tiers][:top_50]
top_50.include?('the')  # => true
top_50.include?('hello')  # => true (in top 200)

# Get available languages
cache.available_languages  # => ["ar", "zh", "en", "el", "it", "no", "ru", "sv"]

Integration with Suggestion Strategies

require 'kotoshu/suggestions/strategies/edit_distance_strategy'

# Frequency bonuses automatically applied
strategy = Kotoshu::Suggestions::Strategies::EditDistanceStrategy.new(
  language_code: 'en'
)

# Suggestions are ranked by frequency
suggestions = strategy.suggest('helo', max_results: 5)
# => [
#   { word: "hello", score: 1200 },  # High frequency word
#   { word: "help", score: 1150 },   # Medium frequency word
#   ...
# ]

Cache TTL and Expiration

All cached resources have a Time-To-Live (TTL) and automatically expire:

LanguageCache: 7 days (dictionaries change infrequently)
FrequencyCache: 7 days (frequency lists are stable)
ModelCache: 30 days (models are large and change rarely)

Expired resources are automatically re-downloaded on next access.

cache = Kotoshu::Cache::FrequencyCache.new

# Force re-download (ignores cache)
freq_data = cache.get('en', force_download: true)

# Clean expired entries manually
cache.clean

Manual Cache Management

cache = Kotoshu::Cache::LanguageCache.new

# Clear specific resource
cache.clear('en:spelling')

# Clear all resources
cache.clear_all

# Check if resource exists
cache.available?('en:spelling')  # => true after download

# Get statistics
stats = cache.stats
puts "Hit rate: #{stats[:hit_rate] * 100}%"
puts "Size: #{stats[:size_bytes]} bytes"

GitHub Repository Structure

The kotoshu/dictionaries repository follows this structure:

kotoshu/dictionaries/
├── en/
│   ├── spelling/
│   │   ├── index.dic          # Hunspell dictionary
│   │   ├── index.aff          # Hunspell affix rules
│   │   └── metadata.json      # Version info
│   ├── grammar/
│   │   └── rules.yaml         # Grammar rules (future)
│   └── models/
│       ├── fasttext/
│       │   └── cc.en.300.vec  # FastText vectors
│       └── onnx/
│           └── fasttext.en.onnx # ONNX model
├── de/
│   └── ... (same structure)
└── README.md

kotoshu/frequency-list-kelly/
├── data/
│   ├── en.json               # Kelly frequency data
│   ├── ru.json
│   └── ...
└── README.md

Adding New Languages

To add support for a new language:

Dictionary: Add Hunspell dictionary to kotoshu/dictionaries/{code}/spelling/
Frequency: Add Kelly frequency data to kotoshu/frequency-list-kelly/data/{code}.json
Register: Add to AVAILABLE_LANGUAGES in LanguageCache
Test: Run integration tests to verify

See CONTRIBUTING.adoc for detailed guidelines.

Model Repository

ONNX models are hosted at: kotoshu/dictionaries

Download and setup:

# Preferred: let kotoshu fetch and verify the model
kotoshu setup en --want spelling,model

# Manual clone (advanced; bypasses manifest verification)
git clone https://github.com/kotoshu/dictionaries.git ~/src/kotoshu/dictionaries

Writing a Document Plugin

Kotoshu ships only Kotoshu::Documents::PlainTextDocument for the plain-text case. Format-specific parsers (Markdown, AsciiDoc, reStructuredText, LaTeX, …) live in plugins — see the kotoshu-document-plugin-boundary design note. Kotoshu auto-discovers plugin files via Gem.find_files("kotoshu_plugin/document/*.rb") — install the gem and the parser registers itself on first use.

A plugin has three pieces:

Parser class — responds to .from_string(text, language_code:) (and optionally .from_file(path, language_code:)). Returns a Kotoshu::Documents::Document whose text_nodes carry proper `Documents::SourceRange`s.
Plugin file — placed at lib/kotoshu_plugin/document/<format>.rb in the gem. The file body calls Kotoshu::Documents.register(format, ParserClass) at load time.
Gem dependency — kotoshu (this gem) and whatever format-parsing library the parser wraps (e.g. kramdown for Markdown, asciidoctor for AsciiDoc).

Example skeleton (my_markdown_parser gem):

# lib/my_markdown_parser/document_parser.rb
module MyMarkdownParser
  class DocumentParser
    # Build one or more TextNodes from the Markdown AST, each
    # carrying a SourceRange that points at the original source.
    def self.from_string(text, language_code: nil)
      # ... walk the AST, collect (text, source_offset) per text run,
      # build TextNodes with SourceRanges ...
      Kotoshu::Documents::Document.new(
        text_nodes: nodes,
        source: text,
        format: :markdown,
        language_code: language_code
      )
    end
  end
end

# lib/kotoshu_plugin/document/markdown.rb
require "kotoshu"
require "my_markdown_parser/document_parser"
Kotoshu::Documents.register(:markdown, MyMarkdownParser::DocumentParser)

After installing the gem, callers get Markdown parsing for free:

require "kotoshu"

doc = Kotoshu::Documents.parse(source, format: :markdown,
                              language_code: "en")
errors = Kotoshu::Spellchecker.new(
  resource_bundle: Kotoshu::ResourceManager.resolve(language: "en")
).check(doc.flattened_text)
errors.each do |error|
  puts "#{error.source_range}: #{error.original} → #{error.recommended_suggestion}"
end

The structure-aware contract: the analyzer scans doc.flattened_text (concatenation of every node’s text), but every error carries a source_range from the original source. An editor can use that range to highlight the user’s actual markup, not the stripped text. For example, on the source

I'm an **friend** of Tom

a grammar error "an friend" → "a friend" carries a source_range pointing at the original an friend range, not the flattened "an friend" range.

Plugin gems should publish to RubyGems with a runtime dependency on kotoshu >= 0.5.0. Format-specific parsers are the plugin’s responsibility — kotoshu never owns document parsing.

API Documentation

The YARD API reference for the Ruby API builds in CI on every change to lib/ (the docs workflow); the generated HTML is attached to each run as the yard-docs artifact. A hosted copy is pending — it will live either on this repository’s GitHub Pages or as an API subpath on kotoshu.github.io.

To build and browse it locally:

bundle install
bundle exec yard doc
open doc/index.html

License

BSD 2-Clause — see the LICENSE file for details.

Bundled dictionaries and frequency lists carry their own licenses; see the per-language license files in kotoshu/dictionaries.

Contributing

Contributions are welcome! Please see CONTRIBUTING.adoc for guidelines.

Acknowledgments

FastText: Facebook Research
ONNX Runtime: Microsoft
Hunspell: László Németh