Efficient Pure Ruby Unicode Normalization (eprun)

(pronounced ee-prune)

This library was originally developed by Martin J. Dürst and Ayumu Nojima, a professor and graduate student at Aoyama Gakuin university in Japan. In 2013, Dürst presented eprun, a fast and efficient pure-ruby Unicode normalization algorithm, at the International Unicode Conference 37 in Santa Clara, California. A textual version of the presentation can be found here.

Shorty after @camertron met Dürst at the conference, @camertron packaged up eprun into a Rubygem, published it to rubygems.org, and replaced the original normalization algorithm in TwitterCLDR with eprun.

Dürst, an active member of the Ruby community back in those days, was eventually able to upstream a version of eprun into Ruby itself. The String#unicode_normalize method uses a modified version of eprun to perform normalization work.

The version of eprun built into Ruby only supports a single version of the Unicode Character Database (UCD), which evolves over time. TwitterCLDR uses a specific version of the UCD, and needs eprun to use the same one for consistency and compatibility. Other libraries dependent on eprun may also have the same need. For this reason, eprun supports the last 5 UCD versions.

First, require the library:

require "eprun"

Eprun's normalization API consists of two methods. To normalize a string:

Eprun.normalize("abc")

Eprun.normalize accepts the normalization form as the second argument. The valid normalization forms are :nfc (the default), :nfd, :nfkc, and :nfkd. For more information on Unicode normalization forms, see the Unicode Standard Annex #15.

The second API method tests if a string is already normalized:

Eprun.normalized?("abc")

The currently supported UCD versions can be inspected via the Eprun::UCD_VERSIONS constant, which is an array of strings of the form "1.2.3."

To change the version eprun will use during normalization operations, set Eprun.current_version.

Eprun.current_version = "17.0.0"

# normalizes using UCD v17
Eprun.normalize("abc")

There's also a .with_version method that changes the current version for the duration of the given block:

Eprun.with_version("17.0.0") do
  # normalizes using UCD v17
  Eprun.normalize("abc")
end

Run tests by executing:

bundle exec rake

To download the necessary Unicode files, first run:

bundle exec rake update_data

Then, to re-generate the normalization tables, regexes, etc, run:

bundle exec rake generate_tables

Licensed under the terms of the original eprun library, which is licensed under the same terms as Ruby.