Project

xbad

0.0
No commit activity in last 3 years
No release in over 3 years
xbad - shared memory key/value cache using shared memory and udp broadcasts for distribution
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
 Dependencies

Development

 Project Readme
* 0XBAD broadcasty distributed shared memory key/value cache

  the whole thing is horribly simple, using FNV32 hashing to build a table 
  (without collision detection - so if 2 keys collide they are overwritten every time)

  [box A] - 
      my $v = $cache->find($key) || 
              $cache->store_and_broadcast($key,slow_subroutine(),0,12345);
              
  [box B] - daemon receives the broadcast message; adds it into the local cache
  [box C] - daemon receives the broadcast message; adds it into the local cache
  [box D] - udp is lost

  in this case, when a request for $key hits [box D] it will do the same 
  $cache->store_and_broadcast() and rebroadcast the item.

* install
  ruby: gem install xbad
  perl: cd perl && perl Makefile.PL && make && make test && make install

* usage
  my $CACHE = BAD->new(0xbeef,16,128)
    creates new shared memory segment with id 0xBEEF
    the segment size is 2^16 * 128 bytes, and it can hold at most 2^16 items

  my $value = $CACHE->find($key) - looks up
  my $value = $CACHE->store($key,$x_value,$expire_after);
  my $value = 
    $CACHE->store_and_broadcast($key,$x_value,$expire_after,$broadcast_port);

  if $expire_after is 0, the item never expires

  ruby: 
    b = XBAD.new(0xBAD,16,128)
    b.store(key,value,expire_after)
    value = b.find(key)

* broadcasting
  store_and_broadcast sends udp broadcast to the specified port
  
* fixed size
  every item in the cache is with size sp->item_size (specified on 
  pool creation) and the shared memory segment is created when you 
  specify exactly how many items between 2^16 and 2^32 you want to
  have in the pool.

  my $b = BAD->new(0xBEEF,17,128); 
      # will create 2^17 items with size 128 bytes

  this is very ineficient, but for caching small-ish items it works fine.

* speed

perl:
  (key: aaaaaaaaaaaaaaaabbbbbbbbbbbbbbb value 'a' x 100)
    native hash store: 4644160.73/s
          0xBAD store: 3062807.05/s
      memcached store: 24514.01/s

      native hash get: 3476691.62/s
            0xBAD get: 2153601.64/s
        memcached get: 25216.77/s

  * native hash - perl's $hash{$key} = $value and $value = $hash{$key}

ruby:
            hash set  3173242.4 i/s -   15841482 in   4.993171s
            xbad set  3760115.4 i/s -   18829228 in   5.008079s
            hash get  4008048.8 i/s -   20056464 in   5.004359s
            xbad get  1844151.9 i/s -    8363250 in   5.010949s
            hash mis  4262632.4 i/s -   21317955 in   5.001340s
            xbad mis  4441230.5 i/s -   22199103 in   4.999691s
        memcache set    13164.1 i/s -      66689 in   5.070756s
        memcache get    14114.6 i/s -      70788 in   5.026147s
       memcache cset    12767.4 i/s -      64501 in   5.076763s
       memcache cget    13474.4 i/s -      68202 in   5.082600s
      hash clone set   437421.6 i/s -    2196828 in   5.031107s
      hash clone get   446848.6 i/s -    2230452 in   4.996158s
    xbad set-encoded   794074.4 i/s -    3953088 in   5.006689s
    xbad get-encoded   372012.9 i/s -    1855488 in   5.035192s
 xbad set-encoded-sn   558399.0 i/s -    2798724 in   5.023483s
 xbad get-encoded-sn   372813.0 i/s -    1858272 in   5.028326s


  * hash- is ruby's native hash hash[key] = value and value = hash[key]
  * hash clone is hash[key] = value.clone and value = hash[key].clone
   xbad - encoded is Sereal (https://github.com/Sereal/Sereal) encoded
   xbad encoded-sn is Sereal encoded and snappy compressed

* locking
  everything is protected by simple gcc atomic builtins
  worst case is somebody locks an item and goes away (interrupted by signal
  or something)