No release in over 3 years
Low commit activity in last 3 years
A/B test analysis library for Ruby
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
 Dependencies

Development

~> 5.10
~> 12.1
~> 0.50

Runtime

 Project Readme

ABAnalyzer¶ ↑

Build Status

ABAnalyzer is a Ruby library that will perform testing to determine if there is a statistical difference in categorical data (typically called an A/B Test). By default, it uses a G-Test for independence, but a Chi-Square test for independence can also be used.

Installation¶ ↑

Simply run:

gem install abanalyzer

Basic Usage¶ ↑

The simplest test (which uses a gtest):

require 'abanalyzer'

values = {}
values[:agroup] = { :opened => 100, :unopened => 300 }
values[:bgroup] = { :opened => 50, :unopened => 350 }

tester = ABAnalyzer::ABTest.new values
# Are the two different?  Returns true or false (at 0.05 level of significance)
puts tester.different?

Multiple Categories¶ ↑

You can use the ABAnalyzer module to test for differences in more than two categories. For instance, to test accross three:

require 'abanalyzer'

values = {}
values[:agroup] = { :male => 200, :female => 250 }
values[:bgroup] = { :male => 150, :female => 300}
values[:cgroup] = { :male => 50, :female => 50 }

tester = ABAnalyzer::ABTest.new values
# Are the two different?  Returns true or false (at 0.05 level of significance)
puts tester.different?

Tests Available¶ ↑

You can get the actual p-value for either a Chi-Square test for independence or a G-Test for independence.

...
tester = ABAnalyzer::ABTest.new values
puts tester.chisquare_p
puts tester.gtest_p

You can additionally get the actual score for either a Chi-Square test for independence or a G-Test for independence.

...
tester = ABAnalyzer::ABTest.new values
puts tester.chisquare_score
puts tester.gtest_score

Sample Size Calculations¶ ↑

Let's say you want to determine how large your sample size needs to be for an A/B test. Let's say your baseline is 10%, and you want to be able to determine if there's at least a 10% relative lift (1% absolute) to 11%. Let's assume you want a power of 0.8 and a significance level of 0.05 (that is, an 80% chance of that you'll succeed in recognizing a difference when there is one, and a 5% chance of a false negative).

...
ABAnalyzer.calculate_size(0.1, 0.11, 0.05, 0.8)
 => 14751

This means that you will need at least 14,751 people in each group sample. You can see this same example with R at on the 37 signals blog.

Confidence Intervals¶ ↑

You can also get a confidence interval. Let's say you have the results of a test where there were 711 successes out of 4000 trials. To get a 95% confidence interval of the “true” value of the conversion rate, use:

...
ABAnalyzer.confidence_interval(711, 4000, 0.95)
 => [0.1659025512617185, 0.1895974487382815]

This means (roughly) that if you ran this experiment over and over, 95% of the time the resulting proportion would be between 17% and 19%.

You can also determine what the relative confidence intervals would be. Let's say that your old conversion rate was 13%, and you wanted to know what sort of relative lift you could get.

...
ABAnalyzer.relative_confidence_interval(711, 4000, 0.13, 0.95)
 => [0.27617347124398833, 0.45844191337139606]

This means (roughly) that if you ran this experiment over and over, 95% of the time the resulting proportion would be a relative lift of between 28% and 46%. Go buy yourself a beer!

Running Tests¶ ↑

Testing can be run by using:

bundle exec rake