No release in over 3 years
Low commit activity in last 3 years
This gem is a Logstash output plugin to deliver the log or metrics to Sumo Logic cloud service. Go to https://github.com/SumoLogic/logstash-output-sumologic for getting help, reporting issues, etc.
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
 Dependencies

Development

Runtime

 Project Readme

Logstash Sumo Logic Output Plugin

Build Status Gem Version Docker Pulls

This is an output plugin for Logstash. It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.

Getting Started

This guide is for the users just want to download the binary and make the plugin work. For the developer, please refer to the Developer Guide

1. Create a Sumo Logic HTTP source

Create a Sumo Logic free account if you currently don't have one.

Create a HTTP source in your account and get the URL for this source. It should be something like: https://events.sumologic.net/receiver/v1/http/XXXXXXXXXX

2. Install LogStash on your machine

Following this instruction to download and install LogStash. This plugin requires Logstash 2.3 or higher to work.

3. Install latest Logstash Sumo Logic Output plugin from RubyGems

bin/logstash-plugin install logstash-output-sumologic

4. Start Logstash and send log

In the Logstash home, running:

bin/logstash -e "input{stdin{}}output{sumologic{url=>'<URL from step 1>'}}"

This will send any input from console to Sumo Logic cloud service.

5. Try out samples

Send Log lines

Set the URL got from step 1 as environment variable:

export sumo_url=https://events.sumologic.net/receiver/v1/http/XXXXXXXXXX

Launch sample with:

bin/logstash -f samples/log.conf

The input from console will be sent to Sumo Logic cloud service as log lines.

Send Metrics

Set the URL got from step 1 as environment variable:

export sumo_url=https://events.sumologic.net/receiver/v1/http/XXXXXXXXXX

Install plugins-filters-metrics plugin

Launch sample with:

bin/logstash -f samples/metrics.conf

Mocked events will be sent to Sumo Logic server as 1 minute and 15 minutes rate metrics.

6. Get result from Sumo Logic web app

Logon to Sumo Logic web app and run

What's Next

  • Try it with different input/filter/codec plugins
  • Start LogStash as a service/daemon in your production environment
  • Report any issue or idea through Git Hub

Parameters of Plugin

Parameter Type Required? Default value Description
url string Yes HTTP Source URL
source_category string No Logstash Source category to appear when searching in Sumo Logic by _sourceCategory. Using empty string if want keep source category of the HTTP source.
source_name string No logstash-output-sumologic Source name to appear when searching in Sumo Logic by _sourceName. Using empty string if want keep source name of the HTTP source.
source_host string No machine name Source host to appear when searching in Sumo Logic by _sourceHost. Using empty string if want keep source host of the HTTP source.
extra_headers hash No Extra fields need to be send in HTTP headers.
compress boolean No false Enable or disable compression.
compress_encoding string No "deflate" Encoding method of comressing, can only be "deflate" or "gzip".
interval number No 0 The maximum time for waiting before sending the message pile, in seconds.
pile_max number No 102400 The maximum size of message pile, in bytes.
queue_max number No 4096 The maximum message piles can be hold in memory.
sender_max number No 100 The maximum HTTP senders working in parallel.
format string No "%{@timestamp} %{host} %{message}" For log only, the formatter of log lines. Use %{@json} as the placeholder for whole event json.
json_mapping hash No Override the structure of {@json} tag with the given key value pairs.
For example:
json_mapping => { "foo" => "%{@timestamp}" "bar" => "%{message}" }
will create messages as:
{"foo":"2016-07-27T18:37:59.460Z","bar":"hello world"}
{"foo":"2016-07-27T18:38:01.222Z","bar":"bye!"}
metrics hash No If defined, the event will be sent as metrics. Keys will be the metrics name and values will be the metrics value.
metrics_format string No "cabon2" Metrics format, can only be "graphite" or "carbon2".
metrics_name string No "*" Define the metric name looking, the placeholder "*" will be replaced with the actual metric name.
intrinsic_tags hash No For carbon2 format only, send extra intrinsic key-value pairs other than metric (which is the metric name).
meta_tags hash No For carbon2 format only, send metadata key-value pairs.
fields_as_metrics boolean No false If true, all fields in logstash event with number value will be sent as a metrics (with filtering by fields_include and fields_exclude ; the metics parameter is ignored.
fields_include array No all fields Working with fields_as_metrics parameter, only the fields which full name matching these RegEx pattern(s) will be included in metrics.
fields_exclude array No none Working with fields_as_metrics parameter, the fields which full name matching these RegEx pattern(s) will be ignored in metrics.
sleep_before_requeue number No 30 The message failed to send to server will be retried after (x) seconds. Not retried if negative number set
stats_category string No Logstash.stats The source category this plugin's stats will be sent with when stats_enabled is true
stats_enabled boolean No false If true, stats of this plugin will be sent as metrics
stats_interval number No 60 The stats will be sent every (x) seconds

This plugin is based on logstash-mixin-http_client thus also supports all HTTP layer parameters like proxy, authentication, timeout etc.

Trouble Shooting

Enable plugin logging

Logstash is using log4j2 framework for logging. Starting with 5.0, each individual plugin can configure the logging strategy. Here is a sample log4j2.properties to print plugin log to console and a rotating file.

Optimize throughput

The throughput can be tuning with following parameters:

  • Messages will be piled before sending if both interval and pile_max are larger than 0. (e.g. multiple messages will sent in single HTTP request); The maximum size of pile is defined in pile_max and if there is no more message comes in, piled message will be sent out every interval seconds. A higher number of these parameters normally means more messages will be piled together so overall reduce the overhead in transmission and benefit for compressing efficiency; but it may make a larger latency because messages may be hold in plugin for longer before sending;
  • Message piles will be cached before sending in a memory queue. The maximum piles can stay in queue is defined with queue_max. A larger setting may be helpful if input is blocked by the plugin consuming speed, but may also consume more RAM (which can be set in JVM options)
  • The plugin will use up to sender_max HTTP senders in parallel for talking to Sumo Logic server. This number is also limited by the max TCP connections
  • Depends on the content pattern, adjusting compress/compress_encoding for balancing between the CPU consumption and package size

On the other side, this version is marked as thread safe so if necessary, multiple plugins can work in parallel as workers

Monitor throughput in metrics

If your Sumo Logic account supports metrics feature, you can enable the stats monitor of this plugin with configuring stats_enabled to true. For every stats_interval seconds, a batch of metrics data points will be sent to Sumo Logic with source category specified by stats_category parameter:

Metric Description
total_input_events Total number of events handled from the plugin startup
total_input_bytes Total bytes of inputs after encoded to payload
total_metrics_datapoints Total metrics data points generated from input
total_log_lines Total log lines generated from input
total_output_requests Total number of HTTP requests sent to Sumo Logic server
total_output_bytes Total bytes of payloads sent to Sumo Logic server
total_output_bytes_compressed Total bytes of payloads sent to Sumo Logic server (after compressing)
total_response_times Total number of responses acknowledged by Sumo Logic server
total_response_success Total number of accepted(200) acknowledged by Sumo Logic server

NOTE: The data points will consume DPM quota

TLS 1.2 Requirement

Sumo Logic only accepts connections from clients using TLS version 1.2 or greater. To utilize the content of this repo, ensure that it's running in an execution environment that is configured to use TLS 1.2 or greater.