No commit activity in last 3 years
No release in over 3 years
Flume Input/Output plugin for Fluentd event collector
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
 Dependencies

Development

>= 0.9.2
~> 3.1.0

Runtime

>= 0.10.43
~> 0.9.0
 Project Readme

Flume input/output plugin for Fluentd¶ ↑

Overview¶ ↑

This is a plugin for fluentd data collector. This plugin adds the Flume compatible interface to fluentd.

What’s Flume?¶ ↑

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data.

It uses Thrift, a cross-language RPC framework, to communicate between clients and servers.

What’s Flume plugin for fluent?¶ ↑

The Flume plugin for fluentd, which enables fluentd to talk the Flume protocol. Flume protocol is defined as follows, in Thrift-IDL format:

typedef i64 Timestamp

enum Priority {
  FATAL = 0,
  ERROR = 1,
  WARN = 2,
  INFO = 3,
  DEBUG = 4,
  TRACE = 5
}

enum EventStatus {
  ACK = 0,
  COMMITED = 1,
  ERR = 2
}

struct ThriftFlumeEvent {
  1: Timestamp timestamp,
  2: Priority priority,
  3: binary body,
  4: i64 nanos,
  5: string host,
  6: map<string,binary> fields
}

# Instead of using thrift's serialization, we just assume the contents are serialized already.
struct RawEvent {
  1: binary raw
}

service ThriftFlumeEventServer {
  oneway void append( 1:ThriftFlumeEvent evt ),
  oneway void rawAppend( 1:RawEvent evt),
  EventStatus ackedAppend( 1: ThriftFlumeEvent evt ),

  void close(),
}

A value that is stored in the ThriftFlumeEvent.fields map is used as fluentd ‘tag’. A key of the value enables be specified by users as configuration parameter.

How to use?¶ ↑

fluent-plugin-flume contains both input and output.

Flume Input¶ ↑

Please add the following configurations to fluent.conf.

# Flume input
<source>
  type flume
  port 56789
</source>

These options are supported.

  • port: port number (default: 56789)

  • bind: bind address (default: 0.0.0.0)

  • server_type: server architecture either in ‘simple’, ‘threaded’, ‘thread_pool’, (default: simple)

  • is_framed: use framed protocol or not (default: false)

  • tag_field: key name of fluentd ‘tag’ that is stored in ThriftFlumeEvent.fields (default: nil)

  • default_tag: default fluentd ‘tag’ (default: ‘category’)

  • add_prefix: prefix string, added to the tag (default: nil)

Use flume-ng-fluentd-sink to send events from flume-ng to fluentd.

Flume Output¶ ↑

Please add the following configurations to fluent.conf. This allows fluentd to output its logs into another Flume server. Note that fluentd conveys semi-structured data while Flume conveys unstructured data. Thus the plugin translates semi-structured data into JSON data by default and conveys it to Flume. The format can be adjusted via formatters.

# Flume output
<match *>
  type flume
  host flume-host.local
  port 56789
</match>

These options are supported.

  • host: host name or address (default: localhost)

  • port: port number (default: 35863)

  • timeout: thrift protocol timeout (default: 30)

  • remove_prefix: prefix string, removed from the tag (default: nil)

  • format: The format of the thrift body (default: json)

  • trim_nl: Trim new line from thrift body (default: true)

Contributors¶ ↑

Copyright

Copyright © 2012 - 2013 Treasure Data, Inc.

License

Apache License, Version 2.0