Project

magellan-gcs-proxy

0.0
No commit activity in last 3 years
No release in over 3 years
There's a lot of open issues
magellan-gcs-proxygroovenauts/magellan-gcs-proxyHomepageDocumentationSource CodeBug TrackerWiki
Adaptor for MAGELLAN BLOCKS batch type IoT board
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
 Popularity
Downloads
19,927
Stars
3
Forks
0
Watchers
18
 Releases
Current version
0.3.2
Total releases
9
First release
2016-11-21
Latest release
2017-02-09
 Issues
Open Issues
5
Closed Issues
5
Total Issues
10
Issue Closure Rate
50%
 Pull Requests
Open Pull Requests
0
Closed Pull Requests
0
Merged Pull Requests
81
Pull Request Acceptance Rate
100%
 Development
Primary Language
Go
Licenses
MIT
Average date of last 50 commits
2018-07-01
Reverse Dependencies
0

 Dependencies

Development

bundler
~> 1.13
rake
~> 10.0
rspec
~> 3.0

Runtime

dotenv
>= 0
google-cloud-logging
>= 0
google-cloud-storage
>= 0
logger_pipe
>= 0
 Project Readme

blocks-gcs-proxy

Build Status

blocks-gcs-proxy is a proxy for MAGELLAN BLOCKS concurrent batch board.

Features

Installation

Download the file from https://github.com/groovenauts/blocks-gcs-proxy/releases and put it somewhere on PATH.

Usage

Create config.json like this:

{
  "job": {
    "subscription": "projects/proj-dummy-999/subscriptions/pipeline01-job-subscription"
  },
  "progress": {
    "topic": "projects/proj-dummy-999/topics/pipeline01-progress-topic"
  }
}

Run blocks-gcs-proxy with command

blocks-gcs-proxy COMMAND ARGS...

blocks-gcs-proxy calls COMMAND with ARGS for each message from Pubsub subscription specified by job.subscription in config.json.

config.json

Key Type Required Default Description
job map False
job.error_response string False ack Response type on error. It must be one of {ack, nack, none}
job.interval_on_error int False 0 The interval time in second to return response on error
job.pull_interval int False 10 The interval time in second to pull when it gets no job message.
job.subscription string False projects/{{ .GCP_PROJECT }}/subscriptions/{{ .PIPELINE }}-job-subscription The subscription name to pull job messages
job.sustainer map False
job.sustainer.delay int False See Sustainer The new deadline in second to extend deadline to ack
job.sustainer.disabled bool False See Sustainer Disable sustainer if it's true
job.sustainer.interval int False See Sustainer The interval in second to send the message which extends deadline to ack
job_check map False
job_check.method string True "none" Method to check job before running. You can set one of none, buntdb or gcslock
job_check.database string False The database name to store job execution data. The usage depends on method
job_check.bucket string False The bucket name to store job execution data. The usage depends on method
job_check.timeout string False The timeout expression like '1h10m10s'. The usage depends on method
progress map False
progress.attributes map[string]string False {} Static attributes of progress notification message
progress.level string False info Log level to publish job progress. You can set one of debug, info, warn, error, fatal and panic.
progress.topic string False projects/{{ .GCP_PROJECT }}/topics/{{ .PIPELINE }}-progress-topic The topic name to publish job progress messages
log map False
log.command_severity string False info The Log severity of command outputs. You can set one of debug, info, warn, error, fatal and panic.
log.level string False info Log level of processing of blocks-gcs-proxy. You can set one of debug, info, warn, error, fatal and panic.
log.stackdriver map False
log.stackdriver.error_reporting_service string False The service name of ServiceContext
log.stackdriver.labels map[string]string True The labels of Monitored resource
log.stackdriver.log_name string True The resource name of the log that will receive the log entries
log.stackdriver.project_id string True GCP Project ID
log.stackdriver.type string True The type of Monitored resource
command map False
command.dryrun bool False false Don't run the command if this is true.
command.options map[key][]string False Define if you have to run one of multiple command. See Multiple command options for more detail.
download map False
download.allow_irregular_url bool False False Allow not strict URL to download
download.worker map False
download.worker.max_tries int False 0 The number of tries to download.
download.worker.workers int False 1 The number of thread to download.
upload map False
upload.content_type_by_ext bool False Set content type by file extension when uploading to GCS
upload.worker map False
upload.worker.max_tries int False 0 The number of tries to upload.
upload.worker.workers int False 1 The number of thread to upload.

Multiple command options

If you have commands data in your config.json like the following:

{
  "commands": {
    "options": {
      "key1": ["cmd1", "%{download_files}"],
      "key2": ["cmd2", "%{download_files.bar}", "%{uploads_dir}", "%{download_files.baz}"]
    }
  }
}

And when you run the command by

$ blocks-gcs-proxy %{attrs.foo}

you can choose which command is executed by message attribute named foo at runtime.

message attributes command magellan-gcs-proxy calls
{"foo": "key1"} cmd1 %{download_files}
{"foo": "key2"} cmd2 %{download_files.bar} %{uploads_dir} %{download_files.baz}

If the attribute value is not defined in commands keys, the message is ignored with error message.

Sustainer

When your command takes longer time than AckDeadline of the pipeline job subscription, Sustainer sends requests to the subscription to expand the deadline. If you don't set job.sustainer in your config.json, blocks-gcs-proxy sets them from the subscription's AckDeadline.

Key Default
job.sustainer.delay Subscription's AckDeadline
job.sustainer.interval Subscription's AckDeadline * 0.8

blocks-gcs-proxy check

Check the config.json is valid. You can give other file with --config or -c option.

$ ./blocks-gcs-proxy check -c config2.json
Error to load config.json.bak cause of invalid character '}' looking for beginning of object key string

blocks-gcs-proxy download

NAME:
   blocks-gcs-proxy download - Download the files from GCS to downloads directory

USAGE:
   blocks-gcs-proxy download [command options] [arguments...]

OPTIONS:
   --downloads_dir value, -d value  Path to the directory which has bucket_name/path/to/file
   --downloaders value, -n value    Number of downloaders (default: 6)
$ ./blocks-gcs-proxy download --help

Example

$ ./blocks-gcs-proxy download -d tmp/downloads -n 5 gs://bucket1/path/to/file1  gs://bucket1/path/to/file2  gs://bucket1/path/to/file3

blocks-gcs-proxy upload

$ ./blocks-gcs-proxy upload --help
NAME:
   blocks-gcs-proxy upload - Upload the files under uploads directory

USAGE:
   blocks-gcs-proxy upload [command options] [arguments...]

OPTIONS:
   --uploads_dir value, -d value  Path to the directory which has bucket_name/path/to/file
   --uploaders value, -n value    Number of uploaders (default: 6)

Example

$ ./blocks-gcs-proxy upload -d tmp/uploads -n 5