Xml parser plugin for Embulk
Embulk parser plugin for parsing xml data by XPath perfectly!
Features
- namespace awareness
- nullable columns
- complex json array columns (with restrictions)
Overview
- Plugin type: parser
- Guess supported: no
Configuration
-
type: specify this plugin as
"xpath2"(string, required) - root: root element to start fetching each entries (string, required)
- schema: specify the attribute of table and data type (required)
- namespaces: specify namespaces (required)
-
stop_on_invalid_record: stop bulk load transaction if a invalid record is found (boolean, default is
false)
Example
parser:
type: xpath2
root: '/ns1:root/ns2:entry'
schema:
- { path: 'ns2:id', name: id, type: long }
- { path: 'ns2:title', name: title, type: string }
- { path: 'ns2:meta/ns2:author', name: author, type: string }
- { path: 'ns2:date', name: date, type: timestamp, format: '%Y%m%d' }
- { path: 'ns2:ratings/ns2:rating[@by="subscribers"]', name: ratings, type: json }
namespaces: {ns1: 'http://example.com/ns1/', ns2: 'http://example.com/ns2/'}Then you can fetch entries from the following xml:
<?xml version="1.0"?>
<ns1:root
xmlns:ns1="http://example.com/ns1/"
xmlns:ns2="http://example.com/ns2/">
<ns2:entry>
<ns2:id>1</ns2:id>
<ns2:title>Hello!</ns2:title>
<ns2:meta>
<ns2:author>maji-KY</ns2:author>
</ns2:meta>
<ns2:date>20010101</ns2:date>
<ns2:ratings>
<ns2:rating by="subscribers">1</ns2:rating>
<ns2:rating by="subscribers">2</ns2:rating>
<ns2:rating>3</ns2:rating>
</ns2:ratings>
</ns2:entry>
</ns1:root>complex json array column
Usage
parser:
type: xpath2
root: '/ns1:root/ns2:entry'
schema:
- { path: 'ns2:id', name: id, type: long }
- path: 'ns2:list'
name: list
type: json
structure: # adding structure key to enabling complex json array column
- path: 'ns2:list'
name: list
type: array
- path: 'ns2:list/ns2:elements'
name: elements
type: array
- path: 'ns2:list/ns2:elements/ns2:name'
name: elementName
type: string
- path: 'ns2:list/ns2:elements/ns2:value'
name: elementValue
type: long
- path: 'ns2:list/ns2:elements/ns2:active'
name: elementActive
type: boolean
namespaces: {ns1: 'http://example.com/ns1/', ns2: 'http://example.com/ns2/'}Structure configuration
- path: specify path from the XPath of the column (string, required)
- name: json key name (string)
- type: json data type (One of array, string, long, boolean., required)
Then you can fetch entries from the following xml:
<?xml version="1.0"?>
<ns1:root
xmlns:ns1="http://example.com/ns1/"
xmlns:ns2="http://example.com/ns2/">
<ns2:entry>
<ns2:id>1</ns2:id>
<ns2:list>
<ns2:elements>
<ns2:name>foo1</ns2:name>
<ns2:value>1</ns2:value>
<ns2:active>true</ns2:active>
</ns2:elements>
<ns2:elements>
<ns2:name>foo2</ns2:name>
<ns2:value>2</ns2:value>
<ns2:active>false</ns2:active>
</ns2:elements>
</ns2:list>
<ns2:list>
<ns2:elements>
<ns2:name>bar1</ns2:name>
<ns2:value>3</ns2:value>
<ns2:active>true</ns2:active>
</ns2:elements>
</ns2:list>
</ns2:entry>
</ns1:root>result of list column:
{
"list": [
{
"elements": [
{
"elementActive": true,
"elementName": "foo1",
"elementValue": 1
},
{
"elementActive": false,
"elementName": "foo2",
"elementValue": 2
}
]
},
{
"elements": [
{
"elementActive": true,
"elementName": "bar1",
"elementValue": 3
}
]
}
]
}Build
$ ./gradlew gem
Benchmark
$ sbt benchmark/jmh:run