Logstash Output Plugin for Azure Data Explorer (Kusto)

This is a plugin for Logstash.

It is fully free and open source. The license is Apache 2.0.

This Azure Data Explorer (ADX) Logstash plugin enables you to process events from Logstash into an Azure Data Explorer database for later analysis.

This connector forwards data to Azure Data Explorer, Azure Synapse Data Explorer and Real time analytics in Fabric

• Logstash version 8.7+. Installation instructions
• Azure Data Explorer cluster with a database. Read Create a cluster and database for more information.
• AAD Application credentials with permission to ingest data into Azure Data Explorer. Read Creating an AAD Application for more information.

To make the Azure Data Explorer plugin available in your Logstash environment, run the following command:

bin/logstash-plugin install logstash-output-kusto

Perform configuration before sending events from Logstash to Azure Data Explorer. The following example shows the minimum you need to provide. It should be enough for most use-cases:

output {
	kusto {
            path => "/tmp/kusto/%{+YYYY-MM-dd-HH-mm}.txt"
            ingest_url => "https://ingest-<cluster-name>.kusto.windows.net/"
            app_id => "<application id>"
            app_key => "<application key/secret>"
            app_tenant => "<tenant id>"
            database => "<database name>"
            table => "<target table>"
            json_mapping => "<mapping name>"
            proxy_host => "<proxy host>"
            proxy_port => <proxy port>
            proxy_protocol => <"http"|"https">              
	}
}

More information about configuring Logstash can be found in the logstash configuration guide

Queued ingestion remains the default and is recommended for high-throughput workloads. For lower ingestion latency, enable streaming mode:

output {
  kusto {
    ingestion_mode => "streaming"
    streaming_max_request_bytes => 1048576
    streaming_temp_directory => "/var/lib/logstash/kusto-streaming"
    ingest_url => "https://ingest-<cluster-name>.kusto.windows.net/"
    app_id => "<application id>"
    app_key => "<application key/secret>"
    app_tenant => "<tenant id>"
    database => "<database name>"
    table => "<target table>"
    json_mapping => "<mapping name>"
  }
}

Streaming ingestion must be enabled on the target Kusto table. The connector groups encoded events into requests of at most 1 MiB by default, measured in bytes. It never splits one event: an event above the configured target is sent intact, and the Kusto client decides whether to stream it or fall back to queued ingestion.

Before acknowledging a Logstash batch, the connector writes all requests into an atomic local spool batch. Accepted requests are removed, transient failures apply backpressure and retry with interruptible exponential delays, and final non-success statuses are quarantined for investigation without replaying the entire request.

Streaming delivery is at least once. A process or host failure after Kusto accepts a request but before the local spool file is removed can cause that request to be recovered and ingested again. The stable source identifier is used for request tracking; it does not provide Kusto-side row deduplication.

The SDK's Queued result means the request was transferred to queued ingestion, not that ingestion into the table has completed. Continue monitoring Kusto ingestion failures for later mapping, schema, encoding-policy, or row-size failures.

The default spool is a destination-specific directory under Logstash path.data. Streaming mode defaults to private 0700 directories and 0600 files, rejects symlinked or untrusted recovery files, and allows only one active plugin instance per spool. When setting a custom directory, place it on persistent local storage with trusted parent directories. Group- or world-writable dir_mode and file_mode values are rejected.

Note : LS_JAVA_OPTS can be used to set proxy parameters as well (using export or SET options)

export  LS_JAVA_OPTS="-Dhttp.proxyHost=1.2.34 -Dhttp.proxyPort=8989 -Dhttps.proxyHost=1.2.3.4 -Dhttps.proxyPort=8989"

• Openjdk 8 64bit (https://www.openlogic.com/openjdk-downloads)
• JRuby 9.2 or higher, defined with openjdk 8 64bit
• Logstash, defined with openjdk 8 64bit

It is reccomened to use the bundled jdk and jruby with logstash to avoid compatibility issues.

To fully build the gem, run:

bundle install
lock_jars
gem build

All contributions are welcome: ideas, patches, documentation, bug reports, and complaints. Programming is not a required skill. It is more important to the community that you are able to contribute. For more information about contributing, see the CONTRIBUTING file.