No commit activity in last 3 years
No release in over 3 years
Fluentd input/output plugin to handle Facebook scribed thrift protocol
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
 Dependencies

Development

>= 0
~> 3.0.2

Runtime

~> 0.8.0
 Project Readme

Scribe input plugin for Fluentd¶ ↑

Overview¶ ↑

This is a plugin for fluentd data collector. This plugin adds the Scribe compatible interface to fluentd.

What’s Scribe?¶ ↑

Scribe is a server for aggregating log data streamed in real time from a large number of servers, developed at Facebook.

It uses Thrift, a cross-language RPC framework, to communicate between clients and servers.

What’s Scribe plugin for fluent?¶ ↑

The Scribe plugin for fluentd, which enables fluentd to talk the Scribe protocol. Scribe protocol is defined as follows, in Thrift-IDL format:

enum ResultCode
{
  OK,
  TRY_LATER
}

struct LogEntry
{
  1:  string category,
  2:  string message
}

service scribe extends fb303.FacebookService
{
  ResultCode Log(1: list<LogEntry> messages);
}

The category field is used as fluentd ‘tag’.

How to use?¶ ↑

fluent-plugin-scribe contains both input and output.

Scribe Input¶ ↑

Please add the following configurations to fluent.conf. This allows your Scribe client to import logs through port 1463.

# Scribe input
<source>
  type scribe
  port 1463
</source>

These options are supported.

  • port: port number (default: 1463)

  • bind: bind address (default: 0.0.0.0)

  • server_type: server architecture either in ‘simple’, ‘threaded’, ‘thread_pool’, ‘nonblocking’ (default: nonblocking)

  • is_framed: use framed protocol or not (default: true)

  • add_prefix: prefix string, added to the tag (default: nil)

  • msg_format: format of the messages either in ‘text’, ‘json’, ‘url_param’ (default: text)

Scribe Output¶ ↑

Please add the following configurations to fluent.conf. This allows fluentd to output its logs into another Scribe server. Note that fluentd conveys semi-structured data while Scribe conveys unstructured data, thus, ‘field_ref’ parameter is prepared to specify which field will be transferred.

# Scribe output
<match *>
  type scribe
  host scribed-host.local
  port 1463
  field_ref message
</match>

These options are supported.

  • host: host name or address (default: localhost)

  • port: port number (default: 1463)

  • field_ref: field name which sent as scribe log message (default: message)

  • timeout: thrift protocol timeout (default: 30)

  • format_to_json: if true/yes, format entire record as json, and send as message (default: false)

  • remove_prefix: prefix string, removed from the tag (default: nil)

For Developers¶ ↑

To run fluentd with this plugin on chaging,

$ bundle # (or 'bundle update')
$ bundle exec fluentd -v -v -v -c example.conf

Then please execute the sample client.

$ bundle exec bin/fluent-scribe-remote

Contributors¶ ↑

Copyright

Copyright © 2011 Treasure Data, Inc.

License

Apache License, Version 2.0