Fluent plugin to ensure @timestamp is in correct format for elasticsearch
gem install fluent-plugin-elasticsearch-timestamp-check
The purpose of this filter is to make sure the @timestamp field exists in the record which is necessary for the record to be indexed properly by elasticsearch.
@timestampfield already exists, it will ensure the format is correct by parse and convert to format '%Y-%m-%dT%H:%M:%S%z'. As of version 0.2.4, it will support epoch second / epoch millis format as a valid timestamp value. If such value is detected, it will be converted to iso8601 format for easier consumption of elasticsearch when dynamic mapping is used.
By default, it will check whether fields named
syslog_timestampexists, if so it will parse that field and conver it to format '%Y-%m-%dT%H:%M:%S.%L%z' then store it in
@timestampfield. In addition, a field
fluent_converted_timestampis added to the object with the same value.
(>=0.3.0) the list of fields can be overriden by setting the
timestamp_fieldsparameter. It accepts a list of strings, the default is set to:
['@timestamp', 'timestamp', 'time', 'syslog_timestamp']
If none of the above field exists, it will insert current event time in '%Y-%m-%dT%H:%M:%S.%L%z' format as the
@timestampfield. A field
fluent_added_timestampis added to the object with same value.
(>=0.2.6) Subsecond Precision
subsecond_precision controls the subsecond precision during the conversion.
Default value is set to
subsecond_precision sample values are:
and more high precision is also supported.
<filter **> type elasticsearch_timestamp_check subsecond_precision 3 </filter>