fluent-plugin-querycombiner

This fluentd output plugin helps you to combine multiple queries.

This plugin is based on fluent-plugin-onlineuser written by Yuyang Lan.

• a running Redis

$ fluent-gem install fluent-plugin-querycombiner

Suppose you have the sequence of event messages like:

{
   'event_id':   '01234567',
   'status':     'event-start',
   'started_at': '2001-02-03T04:05:06Z',
}

and:

{
   'event_id':    '01234567',
   'status':      'event-finish',
   'finished_at': '2001-02-03T04:15:11Z',
}

Now you can combine these messages with this configuration:

<match event.**>
  type query_combiner
  tag combined.test

  # redis settings
  host            localhost
  port            6379
  db_index        0

  query_identify  event_id   # field to combine together
  query_ttl       3600       # messages time-to-live[sec]
  buffer_size     1000       # max queries to store in redis

  <catch>
    condition     status == 'event-start'
  </catch>

  <dump>
    condition     status == 'event-finish'
  </dump>

</match>

Combined results will be:

{
  "event_id":    "01234567",
  "status":      "event-finish",
  "started_at":  "2001-02-03T04:05:06Z",
  "finished_at": "2001-02-03T04:05:06Z"
}

If messages has the same fields, these are overwritten in the combination process. You can use replace sentence in <catch> and <dump> blocks to avoid overwriting such fields.

For example, you have some event messages like:

{
   'event_id': '01234567',
   'status':   'event-start',
   'time':     '2001-02-03T04:05:06Z',
}

and:

{
   'event_id': '01234567',
   'status':   'event-finish',
   'time':     '2001-02-03T04:15:11Z',
}

You can keep time fields which defined both event-start and event-finish by using replace sentence.

<match event.**>
  (...type, tag and redis configuration...)

  query_identify  event_id   # field to combine together
  query_ttl       3600       # messages time-to-live[sec]
  buffer_size     1000       # max queries to store in redis

  <catch>
    condition     status == 'event-start'
    replace       time => time_start
  </catch>

  <dump>
    condition     status == 'event-finish'
    replace       time => time_finish
  </dump>

</match>

Combined results will be:

{
  "event_id":     "01234567",
  "status":       "event-finish",
  "time_start":   "2001-02-03T04:05:06Z",
  "time_finish":  "2001-02-03T04:15:11Z"
}

You can also replace multiple fields joined by comma(,):

<catch>
  condition     status == 'event-start'
  replace       time => time_start, condition => condition_start
</catch>

In previous examples, messages with "status": "event-start" will be watched by plugin immediately.

Suppose some error events occur and you don't want to watch or combine these messages.

In this case <release> block will be useful.

For example, your error messages are such like:

{
  "event_id":  "01234567",
  "status":    "event-error",
  "time":      "2001-02-03T04:05:06Z"
}

Append this <release> block to the configuration and error events will not be watched or combined:

  <release>
    condition     status == 'event-error'
  </release>

You cannot use replace sentence in the <release> block.

Suppose your query_ttl is 600 (10 minutes) and almost events are finished within 10 minutes. But occasionally very-long events occur which finish about 1 hour. These very-long events send status: 'event-continue' messages every 5 minutes for keep-alive.

In this case you can use <prolong> block to reset expired time.

  <prolong>
    condition     status == 'event-continue'
  </prolong>

You cannot use replace sentence in the <prolong> block.

Also you cannot combine messages which defined <prolong> blocks.

If you combine events, time of the events will be lost except defined in <dump> block.

If you want record time of the event, you can define time sentence in <catch> and <dump> blocks.

For example, if you configure your fluentd configuration like below,

  <catch>
    condition     status == 'event-start'
    replace       time => time_start, condition => condition_start
    time          time-catch
  </catch>

you can record time in time-catch field in the result.

{
  "event_id":     "01234567",
  "status":       "event-finish",
  "time-catch":   1414715801.112015,
}

You can set time formats by time_format configuration.

The tag prefix for emitted event messages. By default it's query_combiner.

The basic information for connecting to Redis. By default it's redis://127.0.0.1:6379/0

How many times should the plugin retry when performing a redis operation before raising a error. By default it's 3.

The inactive expire time in seconds. By default it's 1800 (30 minutes).

The max queries to store in redis. By default it's 1000.

If you set this variable true, your pre-combined queries will not remove even after combined by <dump> block. Your pre-combined queries will remove only after their expire times set by query_ttl. Also your pre-combined queries will be prolonged if dumped.

By default it's false.

The interval time to delete expired or overflowed queries which configured by query_ttl and buffer_size. By default it's 10 [sec].

The key prefix for data stored in Redis. By default it's query_combiner:.

Indicates how to extract the query identity from event record. It can be set as a single field name or multiple field names join by comma (,).

The time format for recording time of the events. Default is $time which holds event time. You can also use Ruby's Time module. If you want write ISO8601 format (e.g. 2014-10-31T09:32:57+09:00), you can configure like below.

time_format     Time.at($time).iso8601

• Multi-query combination

Copyright:: Copyright (c) 2014- Takahiro Kamatani

License:: Apache License, Version 2.0