* 0XBAD broadcasty distributed shared memory key/value cache
the whole thing is horribly simple, using FNV32 hashing to build a table
(without collision detection - so if 2 keys collide they are overwritten every time)
[box A] -
my $v = $cache->find($key) ||
$cache->store_and_broadcast($key,slow_subroutine(),0,12345);
[box B] - daemon receives the broadcast message; adds it into the local cache
[box C] - daemon receives the broadcast message; adds it into the local cache
[box D] - udp is lost
in this case, when a request for $key hits [box D] it will do the same
$cache->store_and_broadcast() and rebroadcast the item.
* install
ruby: gem install xbad
perl: cd perl && perl Makefile.PL && make && make test && make install
* usage
my $CACHE = BAD->new(0xbeef,16,128)
creates new shared memory segment with id 0xBEEF
the segment size is 2^16 * 128 bytes, and it can hold at most 2^16 items
my $value = $CACHE->find($key) - looks up
my $value = $CACHE->store($key,$x_value,$expire_after);
my $value =
$CACHE->store_and_broadcast($key,$x_value,$expire_after,$broadcast_port);
if $expire_after is 0, the item never expires
ruby:
b = XBAD.new(0xBAD,16,128)
b.store(key,value,expire_after)
value = b.find(key)
* broadcasting
store_and_broadcast sends udp broadcast to the specified port
* fixed size
every item in the cache is with size sp->item_size (specified on
pool creation) and the shared memory segment is created when you
specify exactly how many items between 2^16 and 2^32 you want to
have in the pool.
my $b = BAD->new(0xBEEF,17,128);
# will create 2^17 items with size 128 bytes
this is very ineficient, but for caching small-ish items it works fine.
* speed
perl:
(key: aaaaaaaaaaaaaaaabbbbbbbbbbbbbbb value 'a' x 100)
native hash store: 4644160.73/s
0xBAD store: 3062807.05/s
memcached store: 24514.01/s
native hash get: 3476691.62/s
0xBAD get: 2153601.64/s
memcached get: 25216.77/s
* native hash - perl's $hash{$key} = $value and $value = $hash{$key}
ruby:
hash set 3173242.4 i/s - 15841482 in 4.993171s
xbad set 3760115.4 i/s - 18829228 in 5.008079s
hash get 4008048.8 i/s - 20056464 in 5.004359s
xbad get 1844151.9 i/s - 8363250 in 5.010949s
hash mis 4262632.4 i/s - 21317955 in 5.001340s
xbad mis 4441230.5 i/s - 22199103 in 4.999691s
memcache set 13164.1 i/s - 66689 in 5.070756s
memcache get 14114.6 i/s - 70788 in 5.026147s
memcache cset 12767.4 i/s - 64501 in 5.076763s
memcache cget 13474.4 i/s - 68202 in 5.082600s
hash clone set 437421.6 i/s - 2196828 in 5.031107s
hash clone get 446848.6 i/s - 2230452 in 4.996158s
xbad set-encoded 794074.4 i/s - 3953088 in 5.006689s
xbad get-encoded 372012.9 i/s - 1855488 in 5.035192s
xbad set-encoded-sn 558399.0 i/s - 2798724 in 5.023483s
xbad get-encoded-sn 372813.0 i/s - 1858272 in 5.028326s
* hash- is ruby's native hash hash[key] = value and value = hash[key]
* hash clone is hash[key] = value.clone and value = hash[key].clone
xbad - encoded is Sereal (https://github.com/Sereal/Sereal) encoded
xbad encoded-sn is Sereal encoded and snappy compressed
* locking
everything is protected by simple gcc atomic builtins
worst case is somebody locks an item and goes away (interrupted by signal
or something)
Project
xbad
xbad - shared memory key/value cache using shared memory and udp broadcasts for distribution
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
2026
Development
Dependencies
Development
>= 0
Project Readme