Project

toon-parser

0.0
The project is in a healthy, maintained state
toon-parserafshmini/toon-parserHomepageDocumentationSource CodeBug TrackerWiki
Parse TOON files and handle TOON-formatted API requests/responses. TOON is a compact, human-readable data format designed to reduce token usage in LLM applications.
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
 Popularity
Downloads
1,648
Stars
0
Forks
0
Watchers
0
 Releases
Current version
0.1.0
Total releases
1
First release
2025-11-20
Latest release
2025-11-20
 Development
Primary Language
Ruby
Licenses
MIT
Average date of last 50 commits
2025-11-20
Reverse Dependencies
0

 Dependencies

Development

rake
~> 13.0
rspec
~> 3.0
rubocop
~> 1.21
 Project Readme

ToonParser

Ruby Version License GitHub

A Ruby gem for parsing and serializing TOON (Token-Oriented Object Notation) format. TOON is a compact, human-readable data format designed to reduce token usage in LLM applications by 30-60% compared to JSON.

Table of Contents

  • Features
  • Installation
  • Quick Start
  • Usage
    • Basic Parsing
    • Basic Serialization
    • File Operations
    • Rails Controller Integration
    • API Integration
    • Advanced Usage
  • TOON Format Overview
  • Examples
  • Development
  • Contributing
  • License

Features

✨ Key Features:

  • 🚀 Fast parsing and serialization of TOON format
  • 🎯 Rails integration - Use TOON in controllers instead of JSON
  • 📦 File operations - Read and write TOON files
  • 🌐 API helpers - Built-in HTTP client for TOON APIs
  • 🔄 Round-trip conversion - Parse and serialize without data loss
  • 🎨 Type inference - Automatic detection of numbers, booleans, and null
  • 📝 Human-readable - Compact yet readable format
  • 🔌 Optional Rails dependency - Works standalone or with Rails

Installation

Add this line to your application's Gemfile:

gem 'toon-parser'

And then execute:

$ bundle install

Or install it yourself as:

$ gem install toon-parser

Quick Start

require 'toon-parser'

# Parse TOON string
toon = "users[2]{id,name}:\n  1,Alice\n  2,Bob"
data = ToonParser.parse(toon)
# => {"users" => [{"id" => 1, "name" => "Alice"}, {"id" => 2, "name" => "Bob"}]}

# Serialize to TOON
data = { "users" => [{ "id" => 1, "name" => "Alice" }] }
toon = ToonParser.serialize(data)
# => "users[1]{id,name}:\n  1,Alice"

Usage

Basic Parsing

Parse a TOON string into Ruby objects:

require 'toon-parser'

toon = <<~TOON
  users[2]{id,name}:
    1,Alice
    2,Bob
TOON

data = ToonParser.parse(toon)
# => {"users" => [{"id" => 1, "name" => "Alice"}, {"id" => 2, "name" => "Bob"}]}

Basic Serialization

Convert Ruby objects to TOON format:

data = {
  "users" => [
    { "id" => 1, "name" => "Alice" },
    { "id" => 2, "name" => "Bob" }
  ]
}

toon = ToonParser.serialize(data)
# => "users[2]{id,name}:\n  1,Alice\n  2,Bob"

File Operations

Parse a TOON file:

data = ToonParser.parse_file("data.toon")

Serialize to a TOON file:

ToonParser.serialize_to_file(data, "output.toon")

Rails Controller Integration

Use TOON format in your Rails controllers instead of JSON. The gem automatically integrates with Rails when available.

Basic Controller Usage

# app/controllers/api/users_controller.rb
class Api::UsersController < ApplicationController
  before_action :set_toon_format

  def index
    @users = User.all
    render toon: { 
      users: @users.map { |u| { id: u.id, name: u.name, email: u.email } } 
    }
  end

  def show
    @user = User.find(params[:id])
    render toon: { 
      user: { id: @user.id, name: @user.name, email: @user.email } 
    }
  end

  def create
    # TOON request body is automatically parsed if Content-Type is application/toon
    user_data = parse_toon_request
    @user = User.create(user_data)
    
    if @user.persisted?
      render toon: { user: @user }, status: :created
    else
      render toon: { errors: @user.errors.full_messages }, status: :unprocessable_entity
    end
  end

  private

  def set_toon_format
    request.format = :toon if request.format == :html
  end
end

Setting Default Format

Set TOON as the default format for API controllers:

# app/controllers/api/base_controller.rb
class Api::BaseController < ApplicationController
  before_action :set_toon_format

  private

  def set_toon_format
    request.format = :toon if request.format == :html
  end
end

Responding to Multiple Formats

Support both TOON and JSON:

class Api::UsersController < ApplicationController
  def index
    @users = User.all
    data = { users: @users.map { |u| { id: u.id, name: u.name } } }
    
    respond_to do |format|
      format.toon { render toon: data }
      format.json { render json: data }
    end
  end
end

Parsing TOON Request Bodies

The gem automatically registers a parameter parser for TOON request bodies. When a request has Content-Type: application/toon, Rails will automatically parse the body. You can also manually parse:

class Api::UsersController < ApplicationController
  def create
    # Option 1: Automatic parsing (if Content-Type is application/toon)
    # The body is automatically parsed into params
    
    # Option 2: Manual parsing
    user_data = parse_toon_request
    @user = User.create(user_data)
    render toon: { user: @user }
  end
end

Note: The automatic parameter parser only works if the request has Content-Type: application/toon header. For manual parsing, use the parse_toon_request helper method.

Routes Configuration

Configure routes to accept TOON format:

# config/routes.rb
Rails.application.routes.draw do
  namespace :api do
    resources :users, defaults: { format: :toon }
  end
end

Available Controller Helpers

  • render_toon(data, options = {}) - Render TOON response
  • parse_toon_request - Parse TOON request body
  • set_toon_format - Set TOON as default format
  • respond_to_toon(data, options = {}) - Support multiple formats

API Integration

Use the API helper for TOON-formatted API requests and responses:

# Create an API client
api = ToonParser.api("https://api.example.com", headers: { "Authorization" => "Bearer token" })

# GET request
users = api.get("/users")

# POST request with TOON body
new_user = api.post("/users", data: { "name" => "Alice", "email" => "alice@example.com" })

# PUT request
updated = api.put("/users/1", data: { "name" => "Alice Updated" })

# PATCH request
patched = api.patch("/users/1", data: { "name" => "Alice Patched" })

# DELETE request
api.delete("/users/1")

Advanced Usage

Nested Objects

toon = <<~TOON
  user:
    id: 1
    name: Alice
    profile:
      age: 30
      city: New York
TOON

data = ToonParser.parse(toon)
# => {"user" => {"id" => 1, "name" => "Alice", "profile" => {"age" => 30, "city" => "New York"}}}

Arrays of Primitives

toon = <<~TOON
  numbers[3]:
    1
    2
    3
TOON

data = ToonParser.parse(toon)
# => {"numbers" => [1, 2, 3]}

Boolean and Null Values

toon = <<~TOON
  data:
    active: true
    deleted: false
    value: null
TOON

data = ToonParser.parse(toon)
# => {"data" => {"active" => true, "deleted" => false, "value" => nil}}

Floating Point Numbers

toon = <<~TOON
  prices[2]:
    19.99
    29.50
TOON

data = ToonParser.parse(toon)
# => {"prices" => [19.99, 29.50]}

TOON Format Overview

TOON uses a compact syntax that eliminates unnecessary brackets, quotes, and whitespace:

JSON:

{
  "users": [
    { "id": 1, "name": "Alice" },
    { "id": 2, "name": "Bob" }
  ]
}

TOON:

users[2]{id,name}:
  1,Alice
  2,Bob

Key Features

  • Array notation: [size] indicates array length
  • Object schema: {field1,field2} defines object structure
  • Tabular data: Rows are comma-separated values
  • Type inference: Numbers, booleans, and null are automatically detected
  • Indentation: Used for nesting and structure
  • Compact: 30-60% smaller than JSON for typical data structures

Examples

See the examples directory for more usage examples:

Development

After checking out the repo, run:

$ bundle install

To run the test suite:

$ bundle exec rspec

Or run tests directly:

$ rspec spec/

To install this gem onto your local machine, run:

$ bundle exec rake install

To release a new version:

  1. Update the version number in lib/toon-parser/version.rb
  2. Run bundle exec rake release

This will create a git tag for the version, push git commits and the created tag, and push the .gem file to rubygems.org.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

  1. Fork the repository
  2. Create your feature branch (git checkout -b feature/amazing-feature)
  3. Commit your changes (git commit -m 'Add some amazing feature')
  4. Push to the branch (git push origin feature/amazing-feature)
  5. Open a Pull Request

Development Setup

# Clone the repository
git clone https://github.com/afshmini/toon-parser.git
cd toon-parser

# Install dependencies
bundle install

# Run tests
bundle exec rspec

# Run linter
bundle exec rubocop

License

The gem is available as open source under the terms of the MIT License.

Links

Acknowledgments