Gladys - Benchmarking Databases, For Science

Gladys, named after a fictional scientist who is obsessed with testing, is a tool for benchmarking PostgreSQL databases and is inspired by pgbench and sysbench but was designed explicitly for benchmarking PostgreSQL, for science.

Gladys is currently being used by Heroku to benchmark their PostgreSQL databases across various plan types, compute sizes, regions, and configurations. Reproducibility of benchmarks are key so we wanted to provide a tool that allows us to run benchmarks in a reproducible way with minimal end-user configuration.

This tool relies heavily on Sequel to provide a simple and powerful way to define and run benchmarks. Documentation for Sequel is available at sequel.jeremyevans.net/documentation.html.

Interested in developing a new script? Check out the scripts documentation.

Show help. This will show all available scripts and their options.

$ gladys help

Prepare a benchmark.

$ gladys prepare tpcc --database-url=$DATABASE_URL

You can also prepare a benchmark with a specific option set.

$ gladys prepare tpcc --database-url=$DATABASE_URL --option-set tpcc-4g

Run a benchmark.

$ gladys benchmark tpcc --database-url=$DATABASE_URL --time 60

You can specify --json to get the output in JSON format, which is useful for integrating into a CI/CD or analytics pipeline.

$ gladys benchmark tpcc --database-url=$DATABASE_URL --time 60 --json

Clean up after a benchmark.

$ gladys cleanup tpcc --database-url=$DATABASE_URL

You can run the script with the --debug flag to get more information about what is happening with a particular script. Keep in mind that this will print a lot of information to the console and isn't recommended for production use.

$ gladys benchmark tpcc --database-url=$DATABASE_URL --debug --time 60

• Do not run benchmarks on production databases or any critical infrastructure.
• Run an option_set that is representative of your production workload in terms of size.
• Run an option_set that is ideally larger than your database's shared buffers otherwise you are basically just benchmarking your server's memory.
• Unless you have a good reason, run the benchmark on a compute instance that is geographically close to your database. Ideally in the same region or availability zone.
• Run the benchmark for a sufficient amount of time to get a representative result. At Heroku, we run for at least 10 minutes.
• Run the benchmark multiple times to ensure no anomalies are present. If you detect problems with the benchmark scripts themselves, please open an issue or a pull request.

Gladys can be run in a Docker container. This is useful for running benchmarks on a remote server or in a CI/CD pipeline.

$ docker build -t gladys .

$ docker run -it gladys

$ docker run -it gladys cleanup tpcc --database-url=$DATABASE_URL
$ docker run -it gladys prepare tpcc --database-url=$DATABASE_URL --threads 8 --option-set tpcc-500g
$ docker run -it gladys benchmark tpcc --database-url=$DATABASE_URL --threads 8 --time 600

TPC-C is a benchmark that measures the performance of a database system. It is a synthetic workload that simulates a typical retail store's inventory management system.

You can read more about TPC-C on their website.

Gladys is designed to be a generic benchmarking tool that can be used to benchmark any PostgreSQL database. If you have a specific benchmark in mind that you would like to see supported, please open an issue and we will consider adding support for it, or even better, a pull request!

Gladys is written in Ruby because it is a language that is easy to use and has a large community. Ruby is used heavily by the Heroku Data team and we've developed a lot of expertise in writing highly performant Ruby code. Additionally, using Ruby gives us the ability to use the excellent Sequel library to define and run benchmarks.

Most database tasks tend to be IO bound and not CPU bound so Ruby's GIL is not as much of a problem as we initially expected. If you are concerned about the GIL skewing the results then you are welcome to use one of the many alternative implementations of Ruby that do not have a GIL, like JRuby.

No, Gladys is a standalone tool that can be used to benchmark any PostgreSQL database. No part of Heroku's infrastructure is required to use Gladys.

There is a bug in the postgres driver that causes the application to segfault when running the benchmark. To fix this, you need to set the PGGSSENCMODE environment variable to disable.

This bug appears to not be present in pg = 1.6.0.

export PGGSSENCMODE="disable"