No commit activity in last 3 years
No release in over 3 years
Distinct
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
2026
 Dependencies

Development

~> 1.0
>= 10.0
 Project Readme

Distinct filter plugin for Embulk

filter returns distinct records by columns you configured.

Overview

  • Plugin type: filter

Configuration

  • columns: column name list to distinguish records (array of string, required)

Example

filters:
  - type: distinct
    columns: [c0, c1]

Run Example

$ ./gradlew classpath
$ embulk run -I lib example/config.yml

Note

this plugin uses a lot of memory because of having distinct column values.

TODO

  • lessen further the amount of memory by filter. i.e. use crc32 of values as distinct key?
    • want ideas!
  • test

Build

$ ./gradlew gem  # -t to watch change of files and rebuild continuously