ActiveRecord::Sharding

Simple Database Sharding in ActiveRecord.

ActiveRecord object distributed multiple databases by modulo. Modulo target is id, generated by sequencer database.

Support database

MySQL

Installation

Add this line to your application's Gemfile:

gem 'activerecord-sharding'

And then execute:

$ bundle

Or install it yourself as:

$ gem install activerecord-sharding

Usage

Add database connections to your application's config/database.yml:

default: &default
  adapter: mysql2
  encoding: utf8
  pool: 5
  username: root
  password:
  host: localhost

user_sequencer:
  <<: *default
  database: user_001
  host: localhost

user_001:
  <<: *default
  database: user_001
  host: localhost

user_002:
  <<: *default
  database: user_002
  host: localhost

user_003:
  <<: *default
  database: user_003
  host: localhost

Add this example your application's config/initializers/active_record_sharding.rb:

ActiveRecord::Sharding.configure do |config|
  config.define_sequencer(:user) do |sequencer|
    sequencer.register_connection(:user_sequencer)
    sequencer.register_table_name('user_id')
  end

  config.define_cluster(:user) do |cluster|
    cluster.register_connection(:user_001)
    cluster.register_connection(:user_002)
    cluster.register_connection(:user_003)
  end
end

define user cluster config
define user sequencer config

Model

app/model/user.rb

class User < ActiveRecord::Base
  include ActiveRecord::Sharding::Model
  use_sharding :user, :modulo # shard name, algorithm
  define_sharding_key :id

  include ActiveRecord::Sharding::Sequencer
  use_sequencer :user

  before_put do |attributes|
    attributes[:id] = next_sequence_id unless attributes[:id]
  end
end

Create sequencer databases

$ rake active_record:sharding:sequencer:setup

Create cluster dtabases

$ rake active_record:sharding:setup

and, migrations all cluster databases.

in applications

Create

using #put! method.

user = User.put! name: 'foobar'

if transaction to shard database and nest other database transactions.

User.put!(name: 'foobar') do |new_user| # transaction for user shard database
  OTHER_MODEL.transaction do
    OTHER_MODEL.create!(user_id: new_user.id)
  end
end

returns User new object.

Select Query

sharding_key = user.id
User.shard_for(sharding_key).where(name: 'foorbar')

sharding_key is your define_syarding_key.(example is User Object id)

#shard_for is returns User class.

for all shards query

User.all_shards.flat_map { |model| model.find_by(name: 'foorbar') }.compact

for all shards query in parallel

User.parallel.flat_map { |model| model.find_by(name: 'foorbar') }.compact

Association/Relation

if use database association/relation in sharding databases.

Please, don't use ActiveRecord standard associations/relation features(has_may, has_one, belongs_to... etc).because, it using ActiveRecord::Base.connection(not sharding databases conneciton).

Please, manually add association/relation methods.

Bad sample

class User < ActiveRecord::Base
  has_many :items # connect to not sharding databases(default database)

  include ActiveRecord::Sharding::Model
  use_sharding :user, :modulo
  define_sharding_key :id
  # (snip)
end

Manually add method

class User < ActiveRecord::Base
  def items
    return [] unless id
    Item.shard_for(id).where(user_id: id).all
  end

  include ActiveRecord::Sharding::Model
  use_sharding :user, :modulo
  define_sharding_key :id
  # (snip)
end

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/hirocaster/activerecord-sharding.