NOTE: This project is no longer being maintained.

This gem provides a few input plugins for Fluentd which are designed to more efficiently poll large volumes of messages than other existing redis input plugins.

• redis_list_poller: Input plugin designed to fetch large volumes of messages
• redis_list_monitor: Input plugin designed to fetch queue size metrics

In additional to the standard stuff, the redis_list_poller input plugin also looks for a lock key in Redis to see whether it has been administratively disabled. This lock key follows the naming convention of redis:KEY_NAME:lock.

• Fluentd v0.14+

gem install fluent-plugin-redis_list_poller

<source>
  @type       redis_list_poller
  host        127.0.0.1
  port        6379
  path        nil
  password    nil
  db          0
  timeout     5.0
  driver      ruby

  key         redis_list_item
  command     lpop
  batch_size  100

  tag         redis.data

  poll_interval      0.01
  sleep_interval     5
  retry_interval    5

  <parse>
    @type json
  </parse>
</source>

These very simple benchmarks which give a rough ideas of the sorts of message rates one can expect from this input plugin.

They were run within a Vagrant box, with a local Redis, running on my Macbook Pro 15'. Fluentd is running on a single CPU core and is using the examples/standalone.conf.