fluent-plugin-pull_forward

Fluentd input/output plugin to forward data, by pulling/request-based transportation, over HTTPS.

We can do with pull_forward:

• transfer data into hosts in firewall by pulling
• protect transferring route by HTTPS and basic authentication
• fetch Fluentd events as JSON by HTTPS from any processes

Configure output plugin to transfer fluentd events to another fluentd nodes.

<match dummy>
  type pull_forward
  
  bind 0.0.0.0 ## default
  port 24280   ## default
  
  buffer_path    /home/myname/tmp/fluentd_event.buffer
  flush_interval 1m   ## default 1h
  
  self_hostname      ${hostname}
  cert_auto_generate yes
  # or
  # "cert_file_path PATH", "private_key_path PATH" and "private_key_passphrase ..."
  
  <user>
    username tagomoris
    password foobar
  </user>
  <user>
    username repeatedly
    password booo
  </user>
</match>

PullForwardOutput uses PullPoolBuffer plugin. DO NOT MODIFY buffer_type. It uses buffer file, so buffer_path is required, and Not so short values are welcome for flush_interval because PullPoolBuffer make a file per flushes (and these are not removed until fetches of cluent/in_pull_forward).

PullForward always requires SSL and basic authentication. SSL options and <user> sections are also required.

Configure input plugin to fetch fluentd events from another fluentd nodes.

<source>
  type pull_forward
  
  fetch_interval 10s
  timeout 10s
  
  <server>
    host host1.on.internet.example.com
    port 24280 ## default
    username tagomoris
    password foobar
  </server>
  <server>
    host host2.on.internet.example.com
    username tagomoris
    password foobar
  </server>
</source>

PullForwardInput can fetch events from many nodes of <server>.

We can fluentd events from PullForwardOutput by normal HTTPS.

$ curl -k -s --user tagomoris:foobar https://localhost:24280/
[
  [ "test.foo", 1406915165, { "pos": 8, "hoge": 1 } ],
  [ "test.foo", 1406915168, { "pos": 9, "hoge": 1 } ],
  [ "test.foo", 1406915173, { "pos": 0, "hoge": 0 } ]
]

• TESTS!

• Copyright (c) 2014- TAGOMORI Satoshi (tagomoris)
• License
  • Apache License, Version 2.0