OpenStack Storage Service (swift) output plugin for Fluent event collector¶ ↑

Overview¶ ↑

swift output plugin buffers event logs in local file and upload it to Swift periodically.

This plugin splits files exactly by using the time of event logs (not the time when the logs are received). For example, a log ‘2011-01-02 message B’ is reached, and then another log ‘2011-01-03 message B’ is reached in this order, the former one is stored in “20110102.gz” file, and latter one in “20110103.gz” file.

Requirements¶ ↑

| fluent-plugin-swift  | fluentd | ruby |
|-------------------|---------|------|
| >= 0.0.4 | >= v0.14.0 | >= 2.1 |
|  < 0.0.4 | >= v0.12.0 | >= 1.9 |

Installation¶ ↑

Simply use RubyGems:

gem install fluent-plugin-swift

Configuration¶ ↑

v1.0 style¶ ↑

With fluentd v1.0 and fluent-plugin-swift >=v0.0.4, use new buffer configuration to dynamic parameters.

<match pattern>
  @type swift

  auth_url https://your.swift.proxy/auth/v1.0 _or_ https://your.keystone/v3.0
  auth_user test:tester
  auth_api_key testing
  project_name test-project
  domain_name "Default"
  auth_region RegionOne
  swift_container CONTAINER_NAME
  ssl_verify false

  path logs/
  swift_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
  <buffer tag,time>
    @type file
    path /var/log/fluent/swift
    timekey 3600 # 1 hour partition
    timekey_wait 10m
    timekey_use_utc true # use utc
  </buffer>
  <format>
    @type json
  </format>

</match>

Use OpenStack environment variables to configure parameters dynamically

<match pattern>
  @type swift

  auth_url "#{ENV['OS_AUTH_URL']}"
  project_name "#{ENV['OS_PROJECT_NAME']}"
  auth_user "#{ENV['OS_USERNAME']}"
  auth_api_key "#{ENV['OS_PASSWORD']}"
  domain_name  "#{ENV['OS_PROJECT_DOMAIN_NAME']}"
  auth_region "#{ENV['OS_REGION_NAME']}"
  swift_container CONTAINER_NAME
  ssl_verify false

  path logs/
  swift_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
  <buffer tag,time>
    @type file
    path /var/log/fluent/swift
    timekey 3600 # 1 hour partition
    timekey_wait 10m
    timekey_use_utc true # use utc
  </buffer>
  <format>
    @type json
  </format>

</match>

auth_url: Authentication URL. If not set in conf, use env OS_AUTH_URL
auth_user: Authentication User Name. if you use TempAuth, auth_user is ACCOUNT:USER . If not set in conf, use env OS_USERNAME
auth_tenant (optional, for keystone v2): Authentication Tenant. if you use TempAuth, this isn’t required.
project_name (keystone v3): Authentication Project. If not set in conf, use env OS_PROJECT_NAME
domain_name (keystone v3): Authentication Domain. If not set in conf, use env OS_PROJECT_DOMAIN_NAME
auth_api_key: Authentication Key (Password). If not set in conf, use env OS_PASSWORD
auth_region: Authentication Region. Optional, not required if there is only one region available. If not set in conf, use env OS_REGION_NAME
swift_account (optional): Account name. if this isn’t provided, use default Account.
swift_container: Container name.
swift_object_key_format: The format of Swift object keys. You can use several built-in variables:

%{path}
%{time_slice}
%{index}
%{file_extension}

to decide keys dynamically.

%{path} is exactly the value of path configured in the configuration file. E.g., “logs/” in the example configuration above. %{time_slice} is the time-slice in text that are formatted with time_slice_format. %{index} is the sequential number starts from 0, increments when multiple files are uploaded to Swift in the same time slice. %{file_extention} is always “gz” for now.

The default format is “%{path}%{time_slice}_%{index}.%{file_extension}”.

For instance, using the example configuration above, actual object keys on Swift will be something like:

"logs/20130111-22_0.gz"
"logs/20130111-23_0.gz"
"logs/20130111-23_1.gz"
"logs/20130112-00_0.gz"

With the configuration:

swift_object_key_format %{path}/events/ts=%{time_slice}/events_%{index}.%{file_extension}
path log
time_slice_format %Y%m%d-%H

You get:

"log/events/ts=20130111-22/events_0.gz"
"log/events/ts=20130111-23/events_0.gz"
"log/events/ts=20130111-23/events_1.gz"
"log/events/ts=20130112-00/events_0.gz"

The fluent-mixin-config-placeholders mixin is also incorporated, so additional variables such as %{hostname}, %{uuid}, etc. can be used in the swift_object_key_format. This could prove useful in preventing filename conflicts when writing from multiple servers.

swift_object_key_format %{path}/events/ts=%{time_slice}/events_%{index}-%{hostname}.%{file_extension}

store_as: archive format on Swift. You can use serveral format:

gzip (default)
json
text
lzo (Need lzop command)

auto_create_container: Create Swift container if it does not exists. Default is true.
path: path prefix of the files on Swift. Default is “” (no prefix).
buffer_path (required): path prefix of the files to buffer logs.
time_slice_format: Format of the time used as the file name. Default is ‘%Y%m%d’. Use ‘%Y%m%d%H’ to split files hourly.
time_slice_wait: The time to wait old logs. Default is 10 minutes. Specify larger value if old logs may reache.
utc: Use UTC instead of local time.

Copyright¶ ↑

This software is based on fluent-plugin-s3 ( github.com/fluent/fluent-plugin-s3 ), written by Sadayuki Furuhashi, licensed by Apache License, Version 2.0.

License: Apache License, Version 2.0

fluent-plugin-swift

Development