Dentaku

Dentaku is a parser and evaluator for a mathematical and logical formula language that allows run-time binding of values to variables referenced in the formulas. It is intended to safely evaluate untrusted expressions without opening security holes.

Dentaku supports Ruby 3.2 and newer, tracking Ruby's own maintenance policy: when a Ruby version reaches end-of-life, it may be dropped from the supported compatibility contract in the next major or minor release. Older Rubies may continue to work, but are not tested.

New code should target the lowest supported Ruby version, currently Ruby 3.2.

This is probably simplest to illustrate in code:

calculator = Dentaku::Calculator.new
calculator.evaluate('10 * 2')
#=> 20

Okay, not terribly exciting. But what if you want to have a reference to a variable, and evaluate it at run-time? Here's how that would look:

calculator.evaluate('kiwi + 5', kiwi: 2)
#=> 7

To enter a case sensitive mode, just pass an option to the calculator instance:

calculator.evaluate('Kiwi + 5', Kiwi: -2, kiwi: 2)
#=> 7
calculator = Dentaku::Calculator.new(case_sensitive: true)
calculator.evaluate('Kiwi + 5', Kiwi: -2, kiwi: 2)
#=> 3

You can also store the variable values in the calculator's memory and then evaluate expressions against those stored values:

calculator.store(peaches: 15)
calculator.evaluate('peaches - 5')
#=> 10
calculator.evaluate('peaches >= 15')
#=> true

For maximum CS geekery, bind is an alias of store.

Dentaku understands precedence order and using parentheses to group expressions to ensure proper evaluation:

calculator.evaluate('5 + 3 * 2')
#=> 11
calculator.evaluate('(5 + 3) * 2')
#=> 16

The evaluate method will return nil if there is an error in the formula. If this is not the desired behavior, use evaluate!, which will raise an exception.

calculator.evaluate('10 * x')
#=> nil
calculator.evaluate!('10 * x')
Dentaku::UnboundVariableError: Dentaku::UnboundVariableError

Dentaku has built-in functions (including if, not, min, max, sum, and round) and the ability to define custom functions (see below). Functions generally work like their counterparts in Excel:

calculator.evaluate('SUM(1, 1, 2, 3, 5, 8)')
#=> 20

calculator.evaluate('if (pears < 10, 10, 20)', pears: 5)
#=> 10
calculator.evaluate('if (pears < 10, 10, 20)', pears: 15)
#=> 20

round can be called with or without the number of decimal places:

calculator.evaluate('round(8.2)')
#=> 8
calculator.evaluate('round(8.2759, 2)')
#=> 8.28

round follows rounding rules, while roundup and rounddown are ceil and floor, respectively.

If you're too lazy to be building calculator objects, there's a shortcut just for you:

Dentaku('plums * 1.5', plums: 2)
#=> 3.0

The flexibility and safety of Dentaku don't come without a price. Tokenizing a string, parsing to an AST, and then evaluating that AST are about 2 orders of magnitude slower than doing the same math in pure Ruby!

The good news is that most of the time is spent in the tokenization and parsing phases, so if performance is a concern, you can enable AST caching:

Dentaku.enable_ast_cache!

Caching can also be controlled per calculator, overriding the module-level setting:

calculator = Dentaku::Calculator.new(cache_ast: true)

After this, Dentaku will cache the AST of each formula that it evaluates, so subsequent evaluations (even with different values for variables) will be much faster -- closer to 4x native Ruby speed. As usual, these benchmarks should be considered rough estimates, and you should measure with representative formulas from your application. Also, if new formulas are constantly introduced to your application, AST caching will consume more memory with each new formula.

Dentaku treats formulas as pure: a subexpression may be evaluated zero, one, or more than one time (for example, a guard position may be evaluated once during dependency analysis and again during evaluation), so custom functions should not rely on side effects or call counts. Functions that fall outside this contract -- reading external state, performing I/O, or returning different values across calls -- must be registered with volatile: true (see EXTERNAL FUNCTIONS below). Volatile functions are never invoked during dependency analysis; in guard positions this means no branch can be pruned, so all branches count as dependencies and evaluate! requires variables from every branch to be bound.

Logical constructs short-circuit. IF and CASE only evaluate the branch that is taken, and AND / OR resolve as soon as one operand decides the result. An unbound variable raises Dentaku::UnboundVariableError only when its value is actually needed:

calculator.evaluate!('a OR b', a: true)   #=> true
calculator.evaluate!('a OR b', a: false)  #=> raises Dentaku::UnboundVariableError

Two methods answer two different questions about a formula's inputs:

calculator.dependencies(expression, context) reports the identifiers a formula still needs, given what is already known. It is resolution-aware: guard positions (IF predicates, CASE switches, AND / OR operands) whose own dependencies are already satisfied are evaluated -- executing any (non-volatile) functions they contain -- in order to prune branches that cannot be taken.

calculator.identifiers(expression) reports every identifier the formula could ever reference, regardless of branching. It is purely syntactic: nothing is evaluated, no functions run (volatile or not), and stored variables are not subtracted. Use it to statically list a formula's possible inputs, e.g. to tell users what values they may need to supply.

calculator.dependencies('IF(x > 5, y, z)', x: 7)  #=> ["y"]
calculator.identifiers('IF(x > 5, y, z)')         #=> ["x", "y", "z"]

Because unused operands are never checked at evaluation time, a misspelled identifier on a branch that is not taken will not be reported; use identifiers for static validation.

Context keys prefixed with a double underscore (e.g. __evaluation_mode) are reserved for internal use.

Literal keywords are case-insensitive, all following are valid: null, TRUE, False.

Date/time literals take an optional time (separated by a space, T or |), optional fractional seconds, and an optional timezone (Z or ±HH:MM). Pass raw_date_literals: false to disable them.

String literals don't support escape sequences.

Operator, function, and keyword names are case-insensitive, you can use AND, and, and And; SUM, sum, and Sum; CASE, case, and Case.

Every method from Ruby's Math module is available as a function. All angles are in radians.

AND, OR, XOR and NOT are available both as operators and functions. All arguments must be logical or null; XOR and NOT treat null as false.

NOT binds looser than comparisons and tighter than AND, OR and XOR, so NOT a > 5 AND b means (NOT (a > 5)) AND b. AND, OR and XOR share a precedence and group left to right: a OR b AND c means (a OR b) AND c. NOT directly followed by parentheses is the function, so NOT (a) = b means NOT(a) = b; write !(a) = b for the operator.

IF(condition, when_true, when_false) returns one of two branches based on a boolean condition:

IF(1 > 2, 123, 456) => 456

IF(2 > 1, "big", "small") => "big"

SWITCH(value, candidate1, result1, candidate2, result2, ..., default) compares value against each candidate and returns the matching result, or the trailing default (nil if omitted and nothing matches):

SWITCH("banana", "apple", 1, "banana", 2, 3) => 2

SWITCH(4, 1, "a", 2, "b", "c") => "c"

CASE provides similar behaviour using keyword form. If no branch matches and there is no ELSE, it raises an error:

CASE "banana" WHEN "apple" THEN 1 WHEN "banana" THEN 2 ELSE 3 END => 2

CASE 4 WHEN 1 THEN "a" WHEN 2 THEN "b" ELSE "c" END => "c"

FIND and SUBSTITUTE also accept a regular expression from a variable.

Member access accepts string or symbol keys (case-insensitively unless the calculator is case-sensitive), and raises if the key is missing or the value is not a hash. Member names must look like identifiers, so a key such as 'a.b' or '2fa' needs brackets: users[0]["a.b"]. There is no whitespace before the dot, and a dotted name on its own, such as user.age, is still a single nested variable.

Both [] and .name are nil-safe: accessing into NULL gives NULL, so FILTER(users, u, u.age > 99)[0].name returns NULL when nothing matches.

If your formulas rely on one another, they may need to be resolved in a particular order. For example:

calc = Dentaku::Calculator.new
calc.store(monthly_income: 50)
need_to_compute = {
  income_taxes: "annual_income / 5",
  annual_income: "monthly_income * 12"
}

In the example, annual_income needs to be computed (and stored) before income_taxes.

Dentaku provides two methods to help resolve formulas in order:

Pass a (string) expression to Dependencies and get back a list of variables (as :symbols) that are required for the expression. Dependencies also takes into account variables already (explicitly) stored into the calculator.

calc.dependencies("monthly_income * 12")
#=> []
# (since monthly_income is in memory)

calc.dependencies("annual_income / 5")
#=> [:annual_income]

Have Dentaku figure out the order in which your formulas need to be evaluated.

Pass in a hash of {eventual_variable_name: "expression"} to solve! and have Dentaku resolve dependencies (using TSort) for you.

Raises TSort::Cyclic when a valid expression order cannot be found.

calc = Dentaku::Calculator.new
calc.store(monthly_income: 50)
need_to_compute = {
  income_taxes:  "annual_income / 5",
  annual_income: "monthly_income * 12"
}
calc.solve!(need_to_compute)
#=> {annual_income: 600, income_taxes: 120}

calc.solve!(
  make_money: "have_money",
  have_money: "make_money"
}
#=> raises TSort::Cyclic

solve! will also raise an exception if any of the formulas in the set cannot be evaluated (e.g. raise ZeroDivisionError). The non-bang solve method will find as many solutions as possible and return the symbol :undefined for the problem formulas.

If your expressions grow long or complex, you may add inline comments for future reference. This is particularly useful if you save your expressions in a model.

calculator.evaluate('kiwi + 5 /* This is a comment */', kiwi: 2)
#=> 7

Comments can be single or multi-line. The following are also valid.

/*
 * This is a multi-line comment
 */

/*
 This is another type of multi-line comment
 */

I don't know everything, so I might not have implemented all the functions you need. Please implement your favorites and send a pull request! Okay, so maybe that's not feasible because:

1. You can't be bothered to share
2. You can't wait for me to respond to a pull request, you need it NOW()
3. The formula is the secret sauce for your startup

Whatever your reasons, Dentaku supports adding functions at runtime. To add a function, you'll need to specify a name, a return type, and a lambda that accepts all function arguments and returns the result value.

Here's an example of adding a function named POW that implements exponentiation.

> c = Dentaku::Calculator.new
> c.add_function(:pow, :numeric, ->(mantissa, exponent) { mantissa ** exponent })
> c.evaluate('POW(3,2)')
#=> 9
> c.evaluate('POW(2,3)')
#=> 8

Here's an example of adding a variadic function:

> c = Dentaku::Calculator.new
> c.add_function(:max, :numeric, ->(*args) { args.max })
> c.evaluate 'MAX(8,6,7,5,3,0,9)'
#=> 9

(However both of these are already built-in -- the ^ operator and the MAX function)

Functions can be added individually using Calculator#add_function, or en masse using Calculator#add_functions.

The second argument is the function's declared return type. It is used at parse time to decide whether a call to the function is a valid operand of an arithmetic operator -- only :numeric, :integer, :array, and nil are accepted there. Declaring anything else means the function cannot be used in arithmetic:

> c = Dentaku::Calculator.new
> c.add_function(:label, :string, ->(x) { "item#{x}" })
> c.evaluate!('label(1)')
#=> "item1"
> c.evaluate!("label(1) = 'item1'")     # comparison: type is not consulted
#=> true
> c.evaluate!('label(1) + 1')           # arithmetic: rejected at parse time
#=> Dentaku::ParseError: Dentaku::AST::Addition requires operands that are
#   numeric or compatible types, not string

The type is a declaration, not a guarantee: Dentaku does not coerce the return value and does not check it at runtime. Declaring :numeric for a function that actually returns a String defers the failure to whatever consumes it:

> c.add_function(:mislabeled, :numeric, ->(x) { "item#{x}" })
> c.evaluate!('mislabeled(1) + 1')
#=> Dentaku::ArgumentError: String input 'item1' is not coercible to numeric

So the type only has an observable effect when the result feeds an arithmetic operator. A function used standalone, in comparisons, or as a function argument behaves the same whatever type is declared -- which is why an incorrect type can go unnoticed for a long time. Declare the type your lambda actually returns.

Dentaku assumes registered functions are pure (see DEPENDENCY ANALYSIS AND SHORT-CIRCUITING above). A function that reads external state, performs I/O, or returns different values across calls must be declared volatile so that dependency analysis never executes it:

> c = Dentaku::Calculator.new
> c.add_function(:user_level, :numeric, -> { Current.user.level }, volatile: true)
> c.identifiers('IF(user_level() > 50, high_rate, low_rate)')
#=> ["high_rate", "low_rate"]

A volatile function in a guard position prevents branch pruning: both branches count as dependencies, and evaluate! requires variables from every branch (Dentaku cannot know in advance which branch will be taken). The predicate itself is then evaluated exactly once per evaluate! call, at evaluation time. In add_functions, pass volatility as a fifth tuple element: [:user_level, :numeric, -> { ... }, nil, true].

Every function can be aliased by synonyms. For example, it can be useful if your application is multilingual.

Dentaku.aliases = {
  round: ['rrrrround!', 'округлить']
}

Dentaku('rrrrround!(8.2) + округлить(8.4)') # the same as round(8.2) + round(8.4)
# 16

Also, if you need thread-safe aliases you can pass them to Dentaku::Calculator initializer:

aliases = {
  round: ['rrrrround!', 'округлить']
}
c = Dentaku::Calculator.new(aliases: aliases)
c.evaluate('rrrrround!(8.2) + округлить(8.4)')
# 16

Big thanks to ElkStone Basements for allowing me to extract and open source this code. Thanks also to all the contributors!

(The MIT License)

Copyright © 2012-2022 Solomon White

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the ‘Software’), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED ‘AS IS’, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.