Repository is archived
No commit activity in last 3 years
No release in over 3 years
Riak plugin for Fluent event collector
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
 Dependencies

Development

>= 0.9.2
>= 1.0.0
>= 0.5.4

Runtime

~> 0.10.7
~> 1.0.0
>= 2.1.3
 Project Readme

fluent-plugin-riak, a plugin for Fluentd

fluent-plugin-riak is a alternative for people who are not sufficient with mongo or webhdfs. Riak ( http://github.com/basho/riak ) is an open-source distributed KVS focused on availability. It also has a strong query system with secondary index (2i): see docs ( http://docs.basho.com/riak/latest/tutorials/querying/ ) for details.

Current status is still proof-of-concept: index setting and its configuration are to be decided. Also performance optimization is required. Another idea is in_tail_riak by using riak post-commit.

installation

$ sudo gem install fluent-plugin-riak

Notice: you need Riak configured using eleveldb as backend.

fluent.conf example

<match riak.**>
  type riak

  buffer_type memory
  flush_interval 10s
  retry_limit 5
  retry_wait 1s
  buffer_chunk_limit 256m
  buffer_queue_limit 8096

  # pb port
  nodes 127.0.0.1:8087
  #for cluster, define multiple machines
  #nodes 192.168.100.128:10018 129.168.100.128:10028 
</match>

  • key format -> 2013-02-

  • value format -> [records] in JSON

  • index:

  • year_int -> year

  • month_bin -> -

  • tag_bin -> tags

easy querying log

$ curl -X PUT http://localhost:8098/buckets/static/keys/browser.html -H 'Content-type: text/html' -d @browser.html
$ open http://localhost:8098/buckets/static/keys/browser.html

Pros

  • easy operations
  • high availability
  • horizontal scalability (esp. write performance)
  • good night sleep

Cons

  • no capped table, TTL objects

TODOs

  • refine browser.html query interface with cool features
  • rething index structures

License

Apache 2.0

Copyright Kota UENISHI

Many Thanks to fluent-plugin-mongo