fluent-plugin-record-reformer

Fluentd plugin to add or replace fields of a event record

Requirements

Note that fluent-plugin-record-reformer supports both v0.14 API and v0.12 API in one gem.

Installation

Use RubyGems:

gem install fluent-plugin-record-reformer

Configuration

Example:

<match foo.**>
  type record_reformer
  remove_keys remove_me
  renew_record false
  enable_ruby false
  
  tag reformed.${tag_prefix[-2]}
  <record>
    hostname ${hostname}
    input_tag ${tag}
    last_tag ${tag_parts[-1]}
    message ${record['message']}, yay!
  </record>
</match>

Assume following input is coming (indented):

foo.bar {
  "remove_me":"bar",
  "not_remove_me":"bar",
  "message":"Hello world!"
}

then output becomes as below (indented):

reformed.foo {
  "not_remove_me":"bar",
  "hostname":"YOUR_HOSTNAME",
  "input_tag":"foo.bar",
  "last_tag":"bar",
  "message":"Hello world!, yay!",
}

Configuration (Classic Style)

Example:

<match foo.**>
  type record_reformer
  remove_keys remove_me
  renew_record false
  enable_ruby false
  tag reformed.${tag_prefix[-2]}
  
  hostname ${hostname}
  input_tag ${tag}
  last_tag ${tag_parts[-1]}
  message ${record['message']}, yay!
</match>

This results in same, but please note that following option parameters are reserved, so can not be used as a record key.

Option Parameters

output_tag (obsolete)

The output tag name. This option is deprecated. Use tag option instead
tag

The output tag name.
remove_keys

Specify record keys to be removed by a string separated by , (comma) like
```
  remove_keys message,foo
```
renew_record bool

renew_record true creates an output record newly without extending (merging) the input record fields. Default is false.
renew_time_key string

renew_time_key foo overwrites the time of events with a value of the record field foo if exists. The value of foo must be a unix time.
keep_keys

You may want to remain some record fields although you specify renew_record true. Then, specify record keys to be kept by a string separated by , (comma) like
```
  keep_keys message,foo
```
enable_ruby bool

Enable to use ruby codes in placeholders. See Placeholders section. Default is true (just for lower version compatibility).
auto_typecast bool

Automatically cast the field types. Default is false. NOTE: This option is effective only for field values comprised of a single placeholder.

Effective Examples:
```
  foo ${foo}
```
Non-Effective Examples:
```
  foo ${foo}${bar}
  foo ${foo}bar
  foo 1
```
Internally, this keeps the type of value if the value text is comprised of a single placeholder, otherwise, values are treated as strings.

When you need to cast field types manually, out_typecast and filter_typecast are available.

Placeholders

Following placeholders are available:

${record["key"]} Record value of key such as ${record["message"]} in the above example (available from v0.8.0).
- Originally, record placeholders were available as ${key} such as ${message}. This is still kept for the backward compatibility, but would be removed in the future.
${hostname} Hostname of the running machine
${tag} Input tag
${time} Time of the event
${tags[N]} (Obsolete. Use tag_parts) Input tag splitted by '.'
${tag_parts[N]} Input tag splitted by '.' indexed with N such as ${tag_parts[0]}, ${tag_parts[-1]}.

${tag_prefix[N]} Tag parts before and on the index N. For example,

  Input tag: prefix.test.tag.suffix
  
  ${tag_prefix[0]}  => prefix
  ${tag_prefix[1]}  => prefix.test
  ${tag_prefix[-2]} => prefix.test.tag
  ${tag_prefix[-1]} => prefix.test.tag.suffix

${tag_suffix[N]} Tag parts after and on the index N. For example,

  Input tag: prefix.test.tag.suffix

  ${tag_suffix[0]}  => prefix.test.tag.suffix
  ${tag_suffix[1]}  => test.tag.suffix
  ${tag_suffix[-2]} => tag.suffix
  ${tag_suffix[-1]} => suffix

It is also possible to write a ruby code in placeholders if you set enable_ruby true option, so you may write some codes as

${time.strftime('%Y-%m-%dT%H:%M:%S%z')}
${tag_parts.last}

but, please note that enabling ruby codes is not encouraged by security reasons and also in terms of the performance.

Relatives

Following plugins look similar:

fluent-plugin-record-modifier
fluent-plugin-format
fluent-plugin-add
filter_record_transformer is a Fluentd v0.12 built-in plugin which is based on record-reformer.

ChangeLog

See CHANGELOG.md for details.

Contributing

Fork it
Create your feature branch (git checkout -b my-new-feature)
Commit your changes (git commit -am 'Add some feature')
Push to the branch (git push origin my-new-feature)
Create new Pull Request