Orchestration

Orchestration aims to provide a convenient and consistent process for working with Rails and Docker without obscuring underlying components.

At its core Orchestration is simply a Makefile and a set of docker-compose.yml files with sensible, general-purpose default settings. Users are encouraged to tailor the generated build-out to suit their application; once the build-out has been generated it belongs to the application.

A typical Rails application can be tested, built, pushed to Docker Hub, and deployed to Docker Swarm with the following commands:

make setup test build push
make deploy manager=user@swarm.example.com env_file=/var/configs/myapp.env

Orchestration has been successfully used to build continuous delivery pipelines for numerous production applications with a wide range or requirements.

See upgrade guide if you are upgrading from 0.5.x to 0.6.x.

The below screenshot demonstrates Orchestration being installed in a brand new Rails application with support for PostgreSQL (via the PG gem) and RabbitMQ (via the Bunny gem):

Docker and Docker Compose must be installed on your system.

Add Orchestration to your Gemfile:

Ruby 3.x:

gem 'orchestration', '~> 0.7.9'

Ruby 2.x:

gem 'orchestration', '~> 0.6.12'

Install:

bundle install

Generate configuration files and select your deployment server:

rake orchestration:install server=unicorn # (or 'puma' [default], etc.)

To rebuild all build-out at any time, pass force=yes to the above install command.

You will be prompted to enter values for your Docker organisation and repository name. For example, the organisation and repository for https://hub.docker.com/r/rubyorchestration/sampleapp are rubyorchestration and sampleapp respectively. If you are unsure of these values, they can be modified later by editing .orchestration.yml in the root of your project directory.

Override these values from the command line by passing project and organization variables:

rake orchestration:install project=myapp organization=myorg

Orchestration generates the following files where appropriate. Backups are created if a file is replaced.

• config/database.yml
• config/mongoid.yml
• config/rabbitmq.yml (see RabbitMQ Configuration for more details)
• config/redis.yml (see Redis Configuration for more details)
• config/unicorn.rb
• config/puma.rb

You may need to merge your previous configurations with the generated files. You will only need to do this once.

Test and development dependency containers bind to a randomly-generated [at install time] local port to avoid collisions. You may compare e.g. orchestration/docker-compose.test.yml with the test section of the generated config/database.yml to see how things fit together.

When setup is complete, add the generated build-out to Git:

git add .
git commit -m "Add Orchestration gem"

All make commands provided by Orchestration (with the exception of test and deploy) recognise the env parameter. This is equivalent to setting the RAILS_ENV environment variable.

e.g.:

# Stop all test containers
make stop RAILS_ENV=test

The default value for env is development.

As with any Makefile targets can be chained together, e.g.:

# Run tests, build, and push image
make setup test build push

All auto-detected services will be added to the relevant docker-compose.<environment>.yml files at installation time.

make start

make stop

$(make compose RAILS_ENV=test) logs -f database

Image tags are generated using the following convention:

# See .orchestration.yml for `organization` and `repository` values.
<organization>/<repository>:<git-commit-hash>

# e.g.
acme/anvil:abcd1234

Note that git archive is used to generate the build context. Any uncommitted changes will not be included in the image.

make build

The include option can also be passed to provide a manifest file. Any files listed in this file will also be built into the Docker image. Files must be located within the project directory.

make build include=manifest.txt

# manifest.txt
doc/api/swagger.json
doc/api/soap.xml
doc/api/doc.html

See also build environment if you use gems hosted on private GitHub/Bitbucket repositories.

You must be logged in to a Docker registry. Use the docker login command (see Docker documentation for further reference).

make push

A .env file is created automatically in your project root. This file is not stored in version control. Set all application environment variables in this file.

Use vanilla Rails commands to load a server or console. If present, a .env file will be read and loaded (via the dotenv gem) to populate the environment.

bundle exec rails s # Launch a server
bundle exec rails c # Launch a console

A default test target is provided in your application's main Makefile. You are encouraged to modify this target to suit your application's requirements.

To launch all dependency containers, run database migrations, and run tests:

make setup test

The default test command can (and should) be extended. This command is defined in the root Makefile in the project and, by defaults, runs rspec and rubocop.

To skip the setup step and just run tests (i.e. once test containers are up and running and ready for use) simply run:

make test

Note that Orchestration will wait for all services to become fully available (i.e. running and providing valid responses) before attempting to run tests. This is specifically intended to facilitate testing in continuous integration environments.

(See sidecar containers if you are running your test/development server inside Docker).

Dependencies will be launched and then tested for readiness. The retry limit and interval time for readiness tests can be controlled by the following environment variables:

ORCHESTRATION_RETRY_LIMIT # default: 15
ORCHESTRATION_RETRY_INTERVAL # default: 10 [seconds]

Run a deployment environment locally to simulate your deployment platform:

make deploy manager=localhost

Ensure you have passwordless SSH access to your own workstation and that you have approved the host authenticity/fingerprint.

To connect via SSH to a remote swarm and deploy, pass the manager parameter:

make deploy manager=user@manager.swarm.example.com

The file orchestration/docker-compose.deployment.yml is created automatically. This file will be used for all deployments, regardless of Rails environment. Other environments should be configured using a separate .env file for each environment. i.e. to deploy a staging environment, create a staging.env (for example), set RAILS_ENV=staging and run:

make deploy manager=user@manager.swarm.example.com env_file=staging.env

This way you can set different publish ports and other application configuration variables for each stage you want to deploy to.

Roll back the app service of your stack:

make rollback manager=user@manager.swarm.example.com

Roll back a specific service:

make rollback manager=user@manager.swarm.example.com service=database

The project_name parameter is also supported.

The Docker stack name defaults to the name of your repository (as defined in .orchesration.yml) and the Rails environment, e.g. anvil_staging.

To override this default, pass the project_name parameter:

make deploy project_name=custom_stack_name

This variable will also be available as COMPOSE_PROJECT_NAME for use within your docker-compose.yml. e.g. to explicitly name a network after the project name:

networks:
  myapp:
    name: "${COMPOSE_PROJECT_NAME}"

Specify a path to a local .env file (see Docker Compose documentation):

make deploy env_file=/path/to/.env

Note that the following two variables must be set in the relevant .env file (will look in the current working directory if no path provided):

# Published port for your application service:
PUBLISH_PORT=3000

# Number of replicas of your application service:
REPLICAS=5

It is also recommended to set SECRET_KEY_BASE, DATABASE_URL etc. in this file.

The output from most underlying components is hidden in an effort to make continuous integration pipelines clear and concise. All output is retained in log/orchestration.stdout.log and log/orchestration.stderr.log. i.e. to view all output during a build:

tail -f log/orchestration*.log

A convenience Makefile target dump is provided. The following command will output all consumed stdout, stderr, and Docker Compose container logs for the test environment:

make dump RAILS_ENV=test

All commands also support the verbose flag which will output all logs immediately to the console:

make build verbose=1

The following environment variables will be passed as ARG variables when building images:

BUNDLE_BITBUCKET__ORG
BUNDLE_GITHUB__COM

Set these variables in your shell if your Gemfile references privately-hosted gems on either Bitbucket or GitHub.

See related documentation:

• https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/
• https://confluence.atlassian.com/bitbucket/app-passwords-828781300.html

Healthchecks are automatically configured for your application. A healthcheck utility is provided in orchestration/healthcheck.rb. The following environment variables can be configured (in the app service of orchestration/docker-compose.deployment.yml):

If your application does not have a suitable always-available route to use as a healthcheck, the following one-liner may be useful:

# config/routes.rb
get '/healthcheck', to: proc { [200, { 'Content-Type' => 'text/html' }, ['']] }

In this case, WEB_HEALTHCHECK_PATH must be set to /healthcheck.

A Dockerfile is automatically generated based on detected dependencies, required build steps, Ruby version, etc.

Real-world applications will inevitably need to make changes to this file. As with all Orchestration build-out, the provided Dockerfile should be treated as a starting point and customisations should be applied as needed.

An entrypoint script for your application is provided at orchestration/entrypoint.sh which does the following:

• Runs the CMD process as the same system user that launched the container (rather than the default root user);
• Creates various required temporary directories and removes stale pid files;
• Adds a route host.docker.internal to the host machine running the container (mimicking the same route provided by Docker itself on Windows and OS X).

If you need to start dependency services (databases, etc.) and connect to them from a Docker container (an example use case of this would be running tests in Jenkins using its Docker agent) then the container that runs your tests must join the same Docker network as your dependency services.

To do this automatically, pass the sidecar parameter to the start or test targets:

make setup test sidecar=1

When running in sidecar mode container-to-container networking is used so there is no benefit to binding dependency containers to a specific port on the host machine (only the target port will be used). For this reason a random, ephemeral port (chosen by Docker) will be used to allow multiple instances of each dependency to run alongside one another.

The Docker Compose project name (and derived network name) is also suffixed with a random token to avoid container/network name conflicts.

Note that a temporary file orchestration/.sidecar containing the random project name suffix will be created when sidecar mode is used. If this file exists then sidecar mode is always assumed to be on. This is to allow (e.g.) stopping services that have been started separately with another command, for example:

# Start dependencies and run tests in sidecar mode
make setup test sidecar=1

# Stop test dependencies in sidecar mode
make stop RAILS_ENV=test

Orchestration generates the following config/rabbitmq.yml:

development:
  url: amqp://127.0.0.1:51070
  management_url: http://127.0.0.1:5069

test:
  url: amqp://127.0.0.1:51068
  management_url: http://127.0.0.1:5067

production:
  url: <%= ENV['RABBITMQ_URL'] %>
  management_url: <%= ENV['RABBITMQ_MANAGEMENT_URL'] %>

The Bunny RabbitMQ gem does not recognise config/rabbitmq.yml or RABBITMQ_URL. If your application uses RabbitMQ then you must manually update your code to reference this file, e.g.:

connection = Bunny.new(Rails.application.config_for(:rabbit_mq)['url'])
connection.start

Using this approach, the environment variable RABBITMQ_URL can be used to configure Bunny in production (similar to DATABASE_URL and MONGO_URL).

This is a convention of the Orchestration gem intended to make RabbitMQ configuration consistent with other services.

Orchestration generates the following config/redis.yml:

development:
  url: redis://127.0.0.1:51071

test:
  url: redis://127.0.0.1:51069

The Redis gem does not recognise config/redis.yml. If your application uses Redis then you must manually update your code to reference this file, e.g.:

# config/initializers/redis.rb

ENV['REDIS_URL'] ||= Rails.application.config_for(:redis)['url']

Redis will then use REDIS_URL for all connections.

This allows development and test environments to auto-load the correct config for the relevant containers while also allowing production to use either the auto-generated Redis service or an external Redis instance.

If you have multiple databases configured in various config/database.*.yml files then the make wait command will automatically detect database configurations.

If a service database-example is included in the relevant Docker Compose configuration then config/database.example.yml will be used to load the connection configuration. Note that the service name must begin with database-.

MIT License

Feel free to make a pull request. Use make test to ensure that all tests, Rubocop checks, and dependency validations pass correctly.