Project

fluent-plugin-tag-normaliser

0.17
Repository is archived
Low commit activity in last 3 years
fluent-plugin-tag-normaliserbanzaicloud/fluent-plugin-tag-normaliserHomepageDocumentationSource CodeWiki
Tag-normaliser is a `fluentd` plugin to help re-tag logs with Kubernetes metadata. It uses special placeholders to change tag.
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
 Popularity
Downloads
7,039,705
Stars
0
Forks
0
Watchers
1
 Releases
Current version
0.1.3
Total releases
3
First release
2019-10-14
Latest release
2024-03-11
 Development
Primary Language
Ruby
Licenses
Apache-2.0
Average date of last 50 commits
2020-06-12
Reverse Dependencies
0

 Dependencies

Development

bundler
~> 1.14
rake
~> 12.0
test-unit
~> 3.0

Runtime

fluentd
>= 0.14.10, < 2
 Project Readme

⚠️ Repository moved

New location: https://github.com/kube-logging/fluent-plugin-tag-normaliser

fluent-plugin-tag-normaliser

Tag-normaliser is a fluentd plugin to help re-tag logs with Kubernetes metadata coming from fluent-bit kubernetes filter.

Installation

RubyGems

$ gem install fluent-plugin-tag-normaliser

Bundler

Add following line to your Gemfile:

gem "fluent-plugin-tag-normaliser"

And then execute:

$ bundle

Configuration

You only need to specify the format option.

<match example.tag**>
  @type tag_normaliser
  format cluster1.${namespace_name}.${pod_name}.${labels.app}
</match>
Parameter Description Default
key_prefix Prefix used to access record attributes kubernetes
format Format to rewrite tag to. You can access record with ${record_name} expression. Nested attributes available via . separator: ${labels.app} ""
unknown Fallback value for missing record attribute "unknown"
sticky_tags Sticky tags will match only one record from an event stream. The same tag will be treated the same way true

Available fluent-bit provided kubernetes attributes

Parameter Description Example
pod_name Pod name understood-butterfly-nginx-logging-demo-7dcdcfdcd7-h7p9n
container_name Container name inside the Pod nginx-logging-demo
namespace_name Namespace name default
pod_id Kubernetes UUID for Pod 1f50d309-45a6-11e9-b795-025000000001
labels Kubernetes Pod labels. This is a nested map. You can access nested attributes via . {"app":"nginx-logging-demo", "pod-template-hash":"7dcdcfdcd7" }
host Node hostname the Pod runs on docker-desktop
docker_id Docker UUID of the container 3a38148aa37aa3...

Example

Configuration

<match example.tag**>
  @type tag_normaliser
  format cluster1.${namespace_name}.${pod_name}.${labels.app}
</match>

Input

tag = "kubernetes.var.log.containers.nginx-s6fvr_default_nginx-889831b33146bb9ec28a5700442b53c6cdf4c445fb5b71c0c48cfa27aaa00212.log"

{  
   "log":"10.1.0.1 - - [13/Mar/2019:15:42:31 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"kube-probe/1.13\" \"-\"\n",
   "kubernetes":{
      "pod_name":"understood-butterfly-nginx-logging-demo-7dcdcfdcd7-h7p9n",
      "namespace_name":"default",
      "pod_id":"1f50d309-45a6-11e9-b795-025000000001",
      "labels":{  
         "app":"nginx-logging-demo",
         "pod-template-hash":"7dcdcfdcd7",
         "release":"understood-butterfly"
      },
      "host":"docker-desktop",
      "container_name":"nginx-logging-demo",
      "docker_id":"3a38148aa37aa30e6e2df96af95cbda7a47b0428689bb4152413f4be25532fda"
   }
}

Output

tag = "cluster1.default.understood-butterfly-nginx-logging-demo-7dcdcfdcd7-h7p9n.nginx-logging-demo",

You can use the plugin without Kubernetes labels just set key_prefix to empty string.

Copyright

  • Copyright(c) 2019- Banzai Cloud
  • License
    • Apache License, Version 2.0