fluent-plugin-sumologic-radiant

A modernized and actively maintained Fluentd output plugin for sending logs and metrics to Sumo Logic via the HTTP Collector API.

This is a fork of the original fluent-plugin-sumologic_output by Sumo Logic Inc. This version includes:

• ✅ Ruby 3.x support (requires Ruby 3.0+)
• ✅ Modern dependencies (Fluentd 1.16+, latest gems)
• ✅ Better performance (using oj for JSON, net-http-persistent for connections)
• ✅ Enhanced security (TLS 1.2+ by default, custom SSL certificates, updated dependencies)
• ✅ Bug fixes from original plugin (see Fixed Issues below)
• ✅ Enhanced debugging with better error messages and logging
• ✅ Active maintenance and vulnerability management
• ✅ Comprehensive test coverage

gem install fluent-plugin-sumologic-radiant

Add to your Gemfile:

gem "fluent-plugin-sumologic-radiant"

Then run:

bundle install

td-agent-gem install fluent-plugin-sumologic-radiant

The plugin is registered as @type sumologic_radiant.

📁 Example Configurations: See the examples/ directory for complete configuration examples:

• basic.conf - Simple configuration for getting started
• kubernetes.conf - Kubernetes logs with metadata
• metrics.conf - Metrics in various formats
• ssl-custom-certs.conf - Custom SSL certificates
• advanced.conf - Comprehensive production configuration

<match **>
  @type sumologic_radiant
  endpoint https://[SumoEndpoint]/receiver/v1/http/[UniqueHTTPCollectorCode]
  log_format json
  source_category my_category
  source_name my_source
  source_host my_host
</match>

<match **>
  @type sumologic_radiant

  # HTTP Collector endpoint (required)
  endpoint https://collectors.sumologic.com/receiver/v1/http/XXXXX

  # Source metadata
  source_category production/app/logs
  source_name ${tag}
  source_host ${hostname}

  # Log formatting
  log_format json          # text, json, json_merge, or fields (default: json)
  log_key message          # Field to use as log message (when log_format is fields)
  open_timeout 60
  add_timestamp true
  timestamp_key timestamp

  # Compression
  compress true            # Enable gzip compression (default: false)
  compress_encoding gzip   # gzip or deflate (default: gzip)

  # Custom fields
  custom_fields department=engineering,application=myapp
  custom_dimensions cluster=prod,region=us-east

  # Proxy configuration
  proxy_uri http://proxy.example.com:8080

  # SSL/TLS
  verify_ssl true          # Verify SSL certificates (default: true)

  # Performance tuning
  disable_cookies true     # Disable cookie handling (default: false)

  # Metadata extraction
  sumo_metadata_key _sumo_metadata  # Extract metadata from this field
</match>

To send metrics to Sumo Logic (requires a metrics HTTP source):

<match metrics.**>
  @type sumologic_radiant
  endpoint https://collectors.sumologic.com/receiver/v1/http/XXXXX
  data_type metrics
  metric_data_format graphite  # graphite, carbon2, or prometheus (default: graphite)
  source_category production/metrics

  # Optional: Add dimensions to all metrics
  custom_dimensions cluster=prod,service=api
</match>

Graphite format:

metric.path value timestamp

Carbon2 format:

metric=value field1=value1 field2=value2 timestamp

Prometheus format:

# TYPE metric_name gauge
metric_name{label1="value1"} value timestamp

You can override source metadata on a per-record basis using the _sumo_metadata field:

{
  "message" => "Application error",
  "_sumo_metadata" => {
    "category" => "errors/application",
    "host" => "webserver-01",
    "source" => "app.log"
  }
}

Enable this feature by setting sumo_metadata_key:

<match **>
  @type sumologic_radiant
  endpoint https://collectors.sumologic.com/receiver/v1/http/XXXXX
  sumo_metadata_key _sumo_metadata
</match>

Note: Issue #50 from the original plugin has been resolved in modern Fluentd versions.

The recommended approach for accessing Kubernetes metadata (e.g., namespace, pod name, container name) is to use Fluentd's buffer chunk keys with the $ accessor syntax:

<match kubernetes.**>
  @type sumologic_radiant
  endpoint https://collectors.sumologic.com/receiver/v1/http/XXXXX

  # Use Kubernetes metadata in source fields
  source_name ${$.kubernetes.container_name}
  source_category ${$.kubernetes.namespace_name}/${$.kubernetes.pod_name}

  # Configure buffer to enable placeholder extraction
  <buffer $.kubernetes.namespace_name, $.kubernetes.pod_name, $.kubernetes.container_name>
    @type memory
    flush_interval 5s
  </buffer>
</match>

You can also include Kubernetes metadata as custom fields:

<match kubernetes.**>
  @type sumologic_radiant
  endpoint https://collectors.sumologic.com/receiver/v1/http/XXXXX

  custom_fields namespace=${$.kubernetes.namespace_name},pod=${$.kubernetes.pod_name},container=${$.kubernetes.container_name}

  <buffer $.kubernetes.namespace_name, $.kubernetes.pod_name, $.kubernetes.container_name>
    @type memory
  </buffer>
</match>

Modern Fluentd (1.16+) supports extracting values from nested record fields using the $.field.subfield syntax when declared in buffer chunk keys. This replaces the deprecated ${record['field']} pattern from older versions.

This plugin is designed as a drop-in replacement for the original fluent-plugin-sumologic_output. To migrate:

1. Update your Gemfile or installation:

   # Old
   # gem "fluent-plugin-sumologic_output"
   
   # New
   gem "fluent-plugin-sumologic-radiant"

2. Update your Fluentd configuration:

   <match **>
     # Old
     # @type sumologic
   
     # New
     @type sumologic_radiant
   
     # ... rest of configuration remains the same
   </match>

3. Verify Ruby version: Ensure you're running Ruby 3.0 or newer.

• Ruby 2.x is no longer supported - Ruby 3.0+ is required
• TLS 1.0/1.1 disabled by default - TLS 1.2+ is enforced
• Dependency changes: Uses oj instead of yajl, net-http-persistent instead of httpclient

This modernized version fixes several issues reported in the original plugin:

Problem: When using log_format=text without the correct log_key, logs would silently stop being sent.

Fix: Added comprehensive warnings at both startup and runtime:

• Startup warning when log_format is set to text or fields
• Runtime warning showing available record keys when log_key is missing
• Enhanced debug logging to track processed vs. dropped records

Problem: The original plugin using httpclient would spam logs with "Unknown key: MAX-AGE" and "Unknown key: SameSite" warnings.

Fix: Replaced httpclient with net-http-persistent, which handles modern cookie attributes properly and eliminates these noisy warnings entirely.

Problem: No support for custom CA certificates or client certificate authentication.

Fix: Added full SSL/TLS customization:

• ca_file - Custom CA certificate file
• ca_path - Custom CA certificate directory
• client_cert / client_key - Mutual TLS support

These options allow secure connections with internal PKI, self-signed certificates, or enterprise certificate requirements.

Added detailed debug logging throughout the plugin:

• Chunk processing statistics (processed, dropped, sent counts)
• SSL configuration details
• Connection initialization messages
• Per-batch send tracking with retry information

Sends the entire record as plain text.

Sends the record as JSON:

{"field1": "value1", "field2": "value2"}

Merges a specific field with the record:

<match **>
  @type sumologic_radiant
  log_format json_merge
  log_key log
</match>

Extracts a specific field as the message:

<match **>
  @type sumologic_radiant
  log_format fields
  log_key message
</match>

• Ruby 3.0 or newer
• Bundler 2.0+
• Git

git clone https://github.com/gnanirahulnutakki/fluent-plugin-sumologic-radiant.git
cd fluent-plugin-sumologic-radiant
bundle install

bundle exec rspec

bundle exec rubocop

bundle exec rake build

The gem will be created in the pkg/ directory.

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

Please ensure:

• Tests pass (bundle exec rspec)
• Code passes linting (bundle exec rubocop)
• New features include tests
• Documentation is updated

Copyright 2025 G. Rahul Nutakki Copyright 2016-2024 Sumo Logic Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

This project is a derivative work of fluent-plugin-sumologic_output by Sumo Logic Inc. See NOTICE for full attribution details.

• Issues: GitHub Issues
• Documentation: README.md
• Original Plugin: fluent-plugin-sumologic_output

See CHANGELOG.md for version history.