Repository is archived
No commit activity in last 3 years
No release in over 3 years
Fluentd parser plugin to parse key value pairs
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
 Dependencies

Development

~> 1.15.0
~> 11.1.2

Runtime

< 0.14.0, >= 0.10.0
 Project Readme

Key-Value Pairs Parser Plugin for Fluentd

This plugin is forked fluent-plugin-kv-parser.

Overview

This is a parser plugin for Fluentd. Learn more about parser plugins here.

This plugin allows you to parse inputs that look like key-value pairs. For example, if your text logs look like

"this_field=10000  that_field=hello time=2013-01-01T12:34:00"

It is parsed as

{"this_field":10000, "that_field":"hello"}

with the event's time being 2013-01-01T12:34:00

How to Install and Use

For Fluentd,

gem install fluent-plugin-kvp-parser

For Treasure Agent,

/usr/sbin/td-agent-gem install fluent-plugin-kvp-parser

Then, for parser-plugin enabled input plugins (including in_tail, in_tcp, in_udp and in_syslog), you can just write format kv

For example, using in_tcp with the following configuration:

<source>
  type tcp
  port 24225
  tag kv_log
  format kv
  time_key my_time
  types k1:integer,my_time:time
</source>
<match kv_log>
  type stdout
</match>

Running

echo 'my_time=2014-12-31T00:00:00 k1=1234 k2=hello' | nc localhost 24224

gives

2014-12-31 00:00:00 +0000 kv_log: {"k1":1234,"k2":"hello"}

Parameters

  • kv_delimiter: The delimiter for key-value pairs. By default \t\s (one or more whitespace/tabs).

    • kv_delimiter a splits on one or more "a"s
    • kv_delimiter ab splits on one or more "a"s or "b"s
  • kv_char: The string to split the key from the value. By default, it is "=".

  • time_key: The time key field among the key-value pairs to be used as the time for the event. If missing or unparsable, the current time is used.

  • types: The parameter to convert the values of key-value pairs. The syntax is <key_name>:<type_name>. For example, to convert the key "k1" into integer, write types k1:integer. For the time type, one can write <key_name>:time:<time_format> to convert the string into a time object. For example, to convert the string "my_time=12/31/2014 12:00:00", use my_time:time:%m/%d/%Y %H:%M:%S. This parameter is same as the one used for in_tail and others (see under the "types" section over there).