0.0
The project is in a healthy, maintained state
Sketch data structures for Ruby
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
 Project Readme

DataSketches

DataSketches - sketch data structures - for Ruby

Build Status

Installation

Add this line to your application’s Gemfile:

gem 'datasketches'

Sketch Families

Distinct counting

  • CPC sketch
  • HyperLogLog sketch
  • Theta sketch

Most frequent

  • Frequent item sketch

Quantiles and histograms

  • KLL sketch

Sampling

  • VarOpt sketch

CPC Sketch

Create a sketch

sketch = DataSketches::CpcSketch.new

Add data

sketch.update(1)
sketch.update(2.0)
sketch.update("three")

Estimate the count

sketch.estimate

Save a sketch

data = sketch.serialize

Load a sketch

sketch = DataSketches::CpcSketch.deserialize(data)

Get the union

u = DataSketches::CpcUnion.new(14)
u.update(sketch1)
u.update(sketch2)
u.result

HyperLogLog Sketch

Create a sketch

sketch = DataSketches::HllSketch.new(14)

Add data

sketch.update(1)
sketch.update(2.0)
sketch.update("three")

Estimate the count

sketch.estimate

Save a sketch

data = sketch.serialize_updatable
# or
data = sketch.serialize_compact

Load a sketch

sketch = DataSketches::HllSketch.deserialize(data)

Get the union

u = DataSketches::HllUnion.new(14)
u.update(sketch1)
u.update(sketch2)
u.result

Theta Sketch

Create a sketch

sketch = DataSketches::UpdateThetaSketch.new

Add data

sketch.update(1)
sketch.update(2.0)
sketch.update("three")

Estimate the count

sketch.estimate

Save a sketch

data = sketch.serialize

Load a sketch

sketch = DataSketches::UpdateThetaSketch.deserialize(data)

Get the union

u = DataSketches::ThetaUnion.new
u.update(sketch1)
u.update(sketch2)
u.result

Get the intersection

i = DataSketches::ThetaIntersection.new
i.update(sketch1)
i.update(sketch2)
i.result

Compute A not B

d = DataSketches::ThetaANotB.new
d.compute(a, b)

Frequent Item Sketch

Create a sketch

sketch = DataSketches::FrequentStringsSketch.new(64)

Add data

sketch.update("a")
sketch.update("b")
sketch.update("c")

Estimate the frequency of an item

sketch.estimate("a")

Save a sketch

data = sketch.serialize

Load a sketch

sketch = DataSketches::FrequentStringsSketch.deserialize(data)

KLL Sketch

Create a sketch

sketch = DataSketches::KllIntsSketch.new
# or
sketch = DataSketches::KllFloatsSketch.new

Add data

sketch.update(1)
sketch.update(2)
sketch.update(3)

Get quantiles

sketch.quantile(0.5)
sketch.quantiles([0.25, 0.5, 0.75])
sketch.quantiles(3)

Get the minimum and maximum values from the stream

sketch.min_value
sketch.max_value

Save a sketch

data = sketch.serialize

Load a sketch

sketch = DataSketches::KllIntsSketch.deserialize(data)

Merge sketches

sketch.merge(sketch2)

VarOpt Sketch

Create a sketch

sketch = DataSketches::VarOptSketch.new(14)

Add data

sketch.update(1)
sketch.update(2.0)
sketch.update("three")

Sample data

sketch.samples

Credits

This library is modeled after the DataSketches Python API.

History

View the changelog

Contributing

Everyone is encouraged to help improve this project. Here are a few ways you can help:

To get started with development:

git clone --recursive https://github.com/ankane/datasketches.git
cd datasketches
bundle install
bundle exec rake compile
bundle exec rake test