fluent-plugin-kestrel¶ ↑

fluentd input/output plugin for kestrel.

fluentd is an event collector system. see github.com/fluent/fluentd

kestrel is a simple, distributed message queue. see github.com/robey/kestrel

gem install fluent-plugin-kestrel

<match kestrel.**>
  type kestrel
  host localhost     # (required) kestrel host
  queue fluent       # (required) queue name of kestrel

  port 22133         # (optional) kestrel port. default 22133
  ttl  100           # (optional) ttl(sec). default=0 (never expire)

  # output format options
  output_include_time false           # (optional) default true
  output_include_tag  false           # (optional) default true
  remove_prefix       kestrel         # (optional) remove tag prefix.
                                      # ex) "kestrel.access" -> "access"
  field_separator COMMA               # (optional) default '\t'.
                                      # you can use SPACE and COMMA
  time_format  %Y-%m-%d %H-%M-%S      # (optional)

</match>

<source>
  type kestrel
  host localhost     # (required) kestrel host
  queue fluent       # (required) kestrel queue name
  tag kestrel.log    # (required) fluentd tag

  port 22133         # (optional) kestrel port. default 22133
  timeout 10         # (optional) default 10.
</source>

[time]\t[tag]\t[resource(JSON)]
2012-02-11T16:38:54Z apache.access   {"host":"::1","user":"-","method":"GET","path":"/test/}

field_separator COMMA

[time],[tag],[resource(JSON)]
2012-02-11T16:38:54Z,apache.access,{"host":"::1","user":"-","method":"GET","path":"/test/}

field_separator SPACE

[time] [tag] [resource(JSON)]
2012-02-11T16:38:54Z apache.access {"host":"::1","user":"-","method":"GET","path":"/test/}

output_include_tag false

[time]\t[resource(JSON)]
2012-02-11T16:38:54Z {"host":"::1","user":"-","method":"GET","path":"/test/}

output_include_time false

[tag]\t[resource(JSON)]
apache.access   {"host":"::1","user":"-","method":"GET","path":"/test/}

remove_prefix apache

[time]\t[tag(prefix removed)]\t[resource(JSON)]
2012-02-11T16:38:54Z access   {"host":"::1","user":"-","method":"GET","path":"/test/}

time_format %Y/%m/%d %H-%M-%S

[time(original format)]\t[tag]\t[resource(JSON)]
2012/02/11 16-38-54 apache.access   {"host":"::1","user":"-","method":"GET","path":"/test/}

you can see its data format like so:

$ telnet localhost 22133

get test

VALUE test 0 125
2012-02-11T16:38:54Z apache.access   {"host":"::1","user":"-","method":"GET","path":"/test/","code":"304","size":"-"}
END

• patches welcome!

Copyright

Copyright © 2012 Junichiro Takagi

License

Apache License, Version 2.0