Project

fluent-plugin-redis-store-wejick

0.0
Bugfix fork offluent-plugin-redis-store
No commit activity in last 3 years
No release in over 3 years
fluent-plugin-redis-store-wejickwejick/fluent-plugin-redis-storeHomepageDocumentationSource CodeWiki
Redis(zset/set/list/string/publish) output plugin for Fluentd...
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
 Popularity
Downloads
22,445
Stars
0
Forks
0
Watchers
1
 Releases
Current version
0.0.9
Total releases
9
First release
2016-08-02
Latest release
2016-12-17
 Development
Primary Language
Ruby
Licenses
Apache-2.0
Average date of last 50 commits
2015-02-08
Reverse Dependencies
0

 Dependencies

Development

rake
>= 0
test-unit
>= 0

Runtime

fluentd
>= 0
redis
>= 0
 Project Readme

Redis Output Plugin For fluentd

Fluentd output plugin to upload/publish event data to Redis storage.

Background

Forked from redis-store which forked from redisstore plugin I'm using this in high traffic production environment

Features

Supported Redis commands

Currently the plugin supports following Redis commands:

  • set by string type (of the plugin)
  • lpush/rpush by list type
  • sadd by set type
  • zadd by zset type
  • publish by publish type

Supported value format

key string for Redis storage

Redis commands require key and value.
For key, the plugin supports either way;

  1. Specify a fixed key.
    You can do this simply using key option in td-agent configuration file.

    type redis_store
key userdata

  2. Lookup a key string in every event data by a lookup path.
    If event data have structured data like

    { "user": { "name": "Kei" } }

    and you want to use each name of user, you can use key_path option.

    type redis_store
key_path user.name

    With the above data, Kei will be a key.

In addition, key_prefix and key_suffix are useful in some cases. Both are available either key and key_path

type redis_store
key_path user.name
key_prefix ouruser.
key_suffix .accesslog

With the previous data, key will be outuser.Kei.accesslog.

value data for Redis storage

To determine what value in every event data to be srtored, you have two options;

  1. Store extracted data in event data, by a lookup path with value_path option.
    It works like key_path.
  2. Store whole data.
    This is default behavior. To do it, simply omit value_path option.

Installation

/usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-redis-store

Configuration

Redis connection

Key Type Required? Default Description
host string Optional 127.0.0.1 host name of Redis server
port int Optional 6379 port number of Redis server
password string Optional password for Redis connection
path string Optional To connect via Unix socket, try '/tmp/redis.sock'
db int Optional 0 DB number of Redis
timeout float Optional 5.0 connection timeout in seconds

common options for storages

Key Type Default Description
key string Fixed key used to store(publish) in Redis
key_path string path to lookup for key in the event data
key_prefix string prefix of key
key_suffix string suffix of key
value_path string (whole event data) path to lookup for value in the event data
store_type string zset string/list/set/zset/publish
format_type string plain format type for value (plain/json/msgpack)
key_expire int -1 If set, the key will be expired in specified seconds
flush_interval time 1 Time interval which events will be flushed to Redis
Note: either key or key_path is required.

string storage specific options

Key Type Default Description
type string Fixed key used to store(publish) in Redis
No more options than common options.

list storage specific options

Key Type Default Description
order string asc asc: rpush, desc: lpush
prevent_duplicate (0,1) 0 Prevent duplicated value in one key (list)
string_tolow (0,1) 0 value to lowercase
string_unescape (0,1) 0 unescape value
string_unescape_twice (0,1) 0 if you need another unescape
escape_html_tag (0,1) 0 escape html tag
filter_html_tag (0,1) 0 replace <>'"& with empty character
value_length (0,1) list trimming doesn't honor ordering

set storage specific options

No more options than common options.

zset storage specific options

Key Type Default Description
score_path string (time of log event) path to lookup for score in the event data
value_expire int value expiration in seconds
prevent_duplicate (0,1) 0 Prevent duplicated value in one key (list)
string_tolow (0,1) 0 value to lowercase
string_unescape (0,1) 0 unescape value
string_unescape_twice (0,1) 0 in you need another unescape
escape_html_tag (0,1) 0 escape html tag
filter_html_tag (0,1) 0 replace <>'"& with empty character
value_length (0,1) z trimming doesn't honor ordering

If value_expire is set, the plugin assumes that the score in the SortedSet is based on timestamp and it deletes expired members every after new event data arrives.

publish storage specific options

No more options than common options.

Copyright

Copyright (c) 2013 moaikids
Copyright (c) 2014 HANAI Tohru
Copyright (c) 2016 Gian Giovani

License

Apache License, Version 2.0