To Csv filter plugin for Embulk

Convert a record to CSV. This plugin is the filter-plugin just ported from embulk-formatter-csv.

Overview

Plugin type: filter

Configuration

cf. http://www.embulk.org/docs/built-in.html#csv-formatter-plugin

name	type	description	required?
column_name	string	Column name used when converting to single value	`"payload"` by default
delimiter	string	Delimiter character such as , for CSV, `"\t"` for TSV, `"	"` or any single-byte character
quote	string	The character surrounding a quoted value	`"` by default
quote_policy	enum	Policy for quote ( `ALL`, `MINIMAL`, `NONE`) (see below)	`MINIMAL` by default
escape	string	Escape character to escape quote character	same with quote default (*1)
header_line	boolean	If true, write the header line with column name at the first line	`false` by default
null_string	string	Expression of `NULL` values	empty by default
newline	enum	Newline character ( `CRLF`, `LF` or `CR`)	`CRLF` by default
newline_in_field	enum	Newline character in each field ( `CRLF`, `LF`, `CR`)	`LF` by default
charset	enum	Character encoding (eg. `ISO-8859-1`, `UTF-8`)	`UTF-8` by default
default_timezone	string	Time zone of timestamp columns. This can be overwritten for each column using `column_options`	`UTC` by default
column_options	hash	See bellow	optional

(*1): if quote_policy is NONE, quote option is ignored, and default escape is \.

The quote_policy option is used to determine field type to quote.

name	description
`ALL`	Quote all fields
`MINIMAL`	Only quote those fields which contain delimiter, quote or any of the characters in lineterminator
`NONE`	Never quote fields. When the delimiter occurs in field, escape with escape char

The column_options option is a map whose keys are name of columns, and values are configuration with following parameters:

name	type	description	required?
timezone	string	Time zone if type of this column is timestamp. If not set, default_timezone is used.	optional
format	string	Timestamp format if type of this column is timestamp.	`%Y-%m-%d %H:%M:%S.%6N %z` by default

Why does this need?

Some output plugin cannot use a formatter plugin, because they are not inherited FileOutputPlugin, but sometimes they need formatters. In that case, this plugin is useful. For example, embulk-output-bigquery

Example

filters:
  - type: to_csv
    column_name: payload
    delimiter: "\t"
    newline: CRLF
    newline_in_field: LF
    charset: UTF-8
    quote_policy: MINIMAL
    quote: '"'
    escape: "\\"
    null_string: "\\N"
    default_timezone: 'UTC'
    column_options:
      time: {format: '%Y-%m-%d %H:%M:%S', timezone: 'America/Los_Angeles'}

Run the example

$ ./gradlew classpath
$ embulk run example/config.yml -Ilib

Build

$ ./gradlew gem  # -t to watch change of files and rebuild continuously

embulk-filter-to_csv