TokenEstimator is a Rails gem that allows you to count tokens in Excel, CSV, PDF, TXT, Markdown, and input text files using different tokenizers.

Training loop, LoRA, and optimization utilities for fine-tuning HuggingFace transformer models using torch-rb and transformers-rb. Supports full fine-tuning, LoRA adapters, learning rate scheduling, callbacks, and model serialization via safetensors.

Vector Search backed by your vector search DB of choice.