The project is in a healthy, maintained state
This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program. Also, this plugin inherit of elasticsearch input plugin, but added tracking_column like jdbc input plugin.
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
 Dependencies

Development

Runtime

 Project Readme

Logstash input plugin

GitHub. This is a plugin for Logstash.

RubyGems repository logstash-input-elastic_jdbc

It is fully free and fully open source.

How install

sudo /usr/share/logstash/bin/logstash-plugin install logstash-input-elastic_jdbc

Documentation

This plugin inherit of elasticsearch(ES) input plugin, and added a tracking_column using in jdbc input plugin for make a query to obtain the updates values.

Sample :

  input{
       elastic_jdbc{
         hosts => "localhost"
         index => "documents"
         tracking_column => "last_update"
         query => '{"query":{"range":{"created":{"gte":"2021-08-13T00:17:58+00:00"}}}}'
         last_run_metadata_path => "/opt/logstash/last_run/elastic_jdbc_documents"
       }
  }
  filter {
  }
  output{
    stdout{}
  }

In the sample before, we read from ES cluster, documents index, where documents hits have last_update field as a date type field (recommend use Ingest pipelines), then we look for all documents that have a field value last_update greater than the value stored in /opt/logstash/last_run/elastic jdbc_documents" .

Required parameters:

  • hosts: ES cluster url
  • index: ES index
  • tracking_column: Date field to tracking in ES index
  • last_run_metadata_path : File path where stored the last value from last hist readed from ES index. By the default have the date 1960-01-01

Optional parameters:

  • All logstash-input-elasticsearch parameters can use in this plugins.
  • query: By the default we use a bool query where we get a hits with tracking column greater that last value stored in last_run_metadata_path. You can insert a query, but keep in mind that your query always be appended with the default query ( if you don't need search by tracking column, please use logstash-input-elasticsearch plugin).

Sample, for this query parameter query => '{"query":{"range":{"created":{"gte":"2021-08-13T00:17:58+00:00"}}}}', the final query using this plugin would be:

    "query":{
       "bool":{
          "must":[
               {"range": {"last_update":{"gt": "date_time_value_stored"}}},
               {"range":{"abonado_date":{"gte": "2021-08-13T00:17:58+00:00"}}}
          ]
       }
    }, 
    "sort": [{"last_update"=>{:order=>"asc"}}]
   }

Note: If you insert a sort statement inside the query, we always overwrite it with the sort statement value that shown above.

Contributing

All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.

Programming is not a required skill. Whatever you've seen about open source and maintainers or community members saying "send patches or die" - you will not see that here.

It is more important to the community that you are able to contribute.

For more information about contributing, see the CONTRIBUTING file.