Argus

Extremely simple ruby daemon to build docker images and push them to AWS Elastic Container Registry.

Features:

• does one thing well (kinda)
• queue builds with SQS messages
• built-in ECR authentication
• can notify Slack of build details

The primary goal of this project is simplicity. It should be easy to adapt to your exact image-building needs.

Argus is implemented as a shoryuken worker. Shoryuken is a super efficient AWS SQS thread based message processor.

You will need:

• AWS credentials set, either with the usual environment variables or with a credentials file
• a github repo to build, and access to it, with ssh keys or GITHUB_TOKEN
• a repository created on ECR to push the docker image once built

If you have local docker, you can run a single build with:

argus-build rlister/argus:master

To run the builder daemon, create an SQS queue called argus, then:

gem install argus-builder
argus-send rlister/argus:master  # replace with your github repo
argus-worker

Install via ruby gems:

gem install argus-builder

or clone from git:

git clone rlister/argus

argus is implemented as a Shoryuken worker. It will poll SQS for build messages, and run:

• git pull
• docker build
• docker push to ECR

Set the following environment variables to configure argus:

export GITHUB_TOKEN=xxx
export AWS_REGION=us-east-1
export AWS_ACCESS_KEY_ID=xxx
export AWS_SECRET_ACCESS_KEY=xxx
export ARGUS_QUEUE=my_queue
export ARGUS_HOME=/data
export SLACK_WEBHOOK=https://hooks.slack.com/services/xxx

Git repositories will be kept in the ARGUS_HOME directory (default /tmp).

Send a message to your queue:

argus-send org/repo:branch

Start the worker daemon to start consuming the queue:

argus-worker

Argus builds images as follows. Examine and modify lib/argus/git.rb and lib/argus/docker.rb to change exact behaviour.

1. git clone a new github repo, or git checkout an existing one
2. authenticate to elastic container registry using provided credentials
3. attempt to docker pull existing image from repo to take advantage of cache
4. docker build the new image
5. docker push to the registry
6. notify Slack

Implementation of your own message sender is simple, using any library that can push a JSON object to SQS.

Example:

require 'aws-sdk'

msg = {
  org:    org,
  repo:   repo,
  branch: branch,     # optional, defaults to master
  tag:    repo:branch # optional. defaults to repo:branch
}

sqs = Aws::SQS::Client.new
sqs.send_message(
  queue_url:    sqs.get_queue_url(queue_name: myqueue).queue_url,
  message_body: msg.to_json
)

If no tag is given it will be constructed as registry/repo:branch. The git branch will have / changed to - to make a legal docker image name.

For example, git repo:

rlister/argus:feature/foo

will produce docker image:

xxx.dkr.ecr.us-east-1.amazonaws.com/argus:feature-foo

argus is a simple shoryuken worker. You can run it directly from the argus repository with any shoryuken options you like:

bundle exec shoyruken -r ./lib/argus/worker.rb -q myqueue -c 1

It is advisable to run a single thread (concurrency of 1) per host, as argus shares build directory between builds, to take advantage of git and docker caching.

Writing your own shoryuken worker class is straightforward, see lib/argus/worker.rb for details.

Argus runs happily inside a docker container, but needs access to a docker daemon to trigger builds. For example, using local socket:

docker run \
  --name argus \
  -e GITHUB_TOKEN \
  -e AWS_REGION \
  -e AWS_ACCESS_KEY_ID \
  -e AWS_SECRET_ACCESS_KEY \
  -e ARGUS_QUEUE \
  -e ARGUS_HOME=/data \
  -e SLACK_WEBHOOK \
  -v /data:/data \
  -v /var/run/docker.sock:/var/run/docker.sock \
  rlister/argus:latest

You will probably want to bind-mount the ARGUS_HOME data directory, to preserve git repos between container restarts. This avoids cloning the entire repos on the next build.

After checking out the repo, run bin/setup to install dependencies. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Bug reports and pull requests are welcome on GitHub at https://github.com/rlister/argus. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.

The gem is available as open source under the terms of the MIT License.