The project is in a healthy, maintained state
Pipelines and State Machines for composable, trackable business logic
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
 Dependencies

Runtime

>= 7.1.3
 Project Readme

Operations

Build your business logic operations in an easy to understand format.

Most times when I'm adding a feature to a complex application, I tend to end up drawing a flowchart.

"We start here, then we check that option and if it's true then we do this, if it's false then we do that"

In effect, that flowchart is a state machine - with "decision states" and "action states". And Operations is intended to be a way of designing your ruby class so that flowchart becomes easy to follow.

Usage

Here's a simplified example from Collabor8Online - in C8O when you download a document, we need to check your access rights, as well as ensuring that the current user has not breached their monthly download limit. In addition, some accounts have a "filename scrambler" switched on - where the original filename is replaced (which is a feature used by some of our clients on their customers' trial accounts).

Defining an operation

The flowchart, for this simplified example, is something like this:

START -> CHECK AUTHORISATION
         Is this user authorised?
         NO -> FAIL
         YES -> CHECK DOWNLOAD LIMITS

         CHECK DOWNLOAD LIMITS
         Is this user within their monthly download limit?
         NO -> FAIL
         YES -> CHECK FILENAME SCRAMBLER

         CHECK FILENAME SCRAMBLER
         Is the filename scrambler switched on for this account?
         NO -> PREPARE DOWNLOAD
         YES -> SCRAMBLE FILENAME

         SCRAMBLE FILENAME
         Replace the filename with a scrambled one
         THEN -> PREPARE DOWNLOAD

         PREPARE DOWNLOAD
         Return the document's filename so it can be used when sending the document to the end user
         DONE

We have five states - three of which are decisions, one is an action and one is a result.

Here's how this would be represented using Operations.

class PrepareDocumentForDownload < Operations::Task
  inputs :user, :document, :use_filename_scrambler
  starts_with :authorised?

  decision :authorised? do
    inputs :user
    condition { user.can?(:read, data.document) }

    if_true :within_download_limits?
    if_false { fail_with "unauthorised" }
  end

  decision :within_download_limits? do
    inputs :user
    condition { user.within_download_limits? }

    if_true :use_filename_scrambler?
    if_false { fail_with "download_limit_reached" }
  end

  decision :use_filename_scrambler? do
    inputs :use_filename_scrambler
    condition { use_filename_scrambler }

    if_true :scramble_filename
    if_false :return_filename
  end

  action :scramble_filename do
    inputs :document

    self.filename = "#{Faker::Lorem.word}#{File.extname(document.filename.to_s)}"
    go_to :return_filename
  end

  result :return_filename do |results|
    inputs :document
    optional :filename

    results.filename = filename || document.filename.to_s
  end
end

task = PrepareDocumentForDownload.call user: @user, document: @document, use_filename_scrambler: @account.feature_flags[:use_filename_scramber]
puts task.results[:filename]

The task declares that it requires user, document and use_filename_scrambler parameters and that it starts in the authorised? state.

The five states are represented as three decision handlers, one action handler and a result handler.

Decisions

A decision handler evaluates a condition, then changes state depending upon if the result is true or false.

decision :is_it_the_weekend? do 
  condition { Date.today.wday.in? [0, 6] }

  if_true :have_a_party 
  if_false :go_to_work
end

A decision can also mark a failure, which will terminate the task and raise an Operations::Failure.

decision :authorised? do 
  condition { user.administrator? }

  if_true :do_some_work 
  if_false { fail_with "Unauthorised" }
end

(In theory the block used in the fail_with case can do anything within the DataCarrier context - so you could set internal state or call methods on the containing task - but I've not tried this yet).

You can specify the data that is required for a decision handler to run by specifying inputs and optionals:

decision :authorised? do 
  inputs :user 
  optional :override
  condition { override || user.administrator? }

  if_true :do_some_work 
  if_false { fail_with "Unauthorised" }
end

In this case, the task will fail (with an ArgumentError) if there is no user specified. However, override is optional (in fact the optional method does nothing and is just there for documentation purposes).

Actions

An action handler does some work, then moves to another state.

action :have_a_party do
  self.food = task.buy_some_food_for(number_of_guests)
  self.beer = task.buy_some_beer_for(number_of_guests)
  self.music = task.plan_a_party_playlist

  go_to :send_invitations
end

You can specify the required and optional data for your action handler within the block. optional is decorative and to help with your documentation. Ensure you call inputs at the start of the block so that the task fails before you do any meaningful work.

action :have_a_party do
  inputs :number_of_guests 
  optional :music 

  self.food = task.buy_some_food_for(number_of_guests)
  self.beer = task.buy_some_beer_for(number_of_guests)
  self.music ||= task.plan_a_party_playlist

  go_to :send_invitations
end

Do not forget to call go_to from your action handler, otherwise the operation will just stop whilst still being marked as in progress. (TODO: don't let this happen).

Waiting

Wait handlers only work within background tasks. They define a condition that is evaluated; if it is false, the task is paused and the condition re-evaluated later. If it is true, the task moves to the next state.

Results

A result handler marks the end of an operation, optionally returning some results. You need to copy your desired results from your data to your results object. This is so only the information that matters to you is stored as the results.

action :send_invitations do 
  self.invited_friends = (0..number_of_guests).collect do |i|
    friend = friends.pop
    FriendsMailer.with(recipient: friend).party_invitation.deliver_later unless friend.nil?
    friend 
  end.compact

  go_to :ready_to_party
end

result :ready_to_party do |results|
  results.invited_friends = invited_friends
end

After this result handler has executed, the task will then be marked as completed?, the task's state will be ready_to_party and results[:invited_friends] will contain an array of the people you sent invitations to.

If you don't have any meaningful results, you can omit the block on your result handler.

result :go_to_work

In this case, the task will be marked as completed?, the task's state will be go_to_work and results will be empty.

You can also specify the required and optional data for your result handler within the block. optional is decorative and to help with your documentation. Ensure you call inputs at the start of the block.

action :send_invitations do 
  inputs :number_of_guests, :friends

  self.invited_friends = (0..number_of_guests).collect do |i|
    friend = friends.pop
    FriendsMailer.with(recipient: friend).party_invitation.deliver_later unless friend.nil?
    friend 
  end.compact

  go_to :ready_to_party
end

result :ready_to_party do |results|
  inputs :invited_friends 

  results.invited_friends = invited_friends
end

Calling an operation

You would use the earlier PrepareDocumentForDownload operation in a controller like this:

class DownloadsController < ApplicationController 
  def show 
    @document = Document.find(params[:id])
    @task = PrepareDocumentForDownload.call(user: Current.user, document: @document, use_filename_scrambler: Current.account.use_filename_scrambler?)

    send_data @document.contents, filename: @task.results[:filename], disposition: "attachment"

  rescue => failure
    render action: "error", locals: {error: failure.message}, status: 422
  end
end

OK - so that's a pretty longwinded way of performing a simple task.

But many operations end up as complex flows of conditionals and actions, often spread across multiple classes and objects. This means that someone trying to understand the rules for an operation can spend a lot of time tracing through code, understanding that flow.

In Collabor8Online, the actual operation for handling downloads has over twenty states, with half of them being decisions (there are a number of feature flags and per-account configuration options which need to be considered). Now they are defined as four ruby classes (using composition to split the workload) and the logic is extremely easy to follow.

Data and results

Each operation carries its own, mutable, data for the duration of the operation.

This is provided when you call the operation to start it and is passed through to each decision, action and result. If you modify the data then that modification is passed on to the next handler.

Within handlers you can read the data directly (the implementation uses instance_eval/instance_exec). Here the build_name action knows the first_name and last_name provided and adds in a new property of name.

class CombineNames < Operations::Task 
  inputs :first_name, :last_name 
  starts_with :build_name 

  action :build_name do 
    self.name = "#{first_name} #{last_name}"
    go_to :done 
  end

  result :done do |results|
    results.name = name 
  end
end

task = CombineNames.call first_name: "Alice", last_name: "Aardvark"
task.results[:name] # => Alice Aardvark

Because handlers are run in the context of the data carrier, you do not have direct access to methods or properties on your task object. However, the data carrier holds a reference to your task; use task.do_something or task.some_attribute to access it. The exceptions are the go_to, fail_with, call and start methods which the data carrier understands (and are intercepted when you are testing).

Both your task's data and its final results are stored in the database, so they can be examined later. The results because that's what you're interested in, the data as it can be useful for debugging or auditing purposes.

They are both stored as hashes that are encoded into JSON.

Instead of using the standard JSON coder, we use a GlobalIDSerialiser. This uses ActiveJob::Arguments to transform any models into GlobalIDs before storage and convert them back to models upon retrieval.

If the original database record was deleted between the time the hash was serialised and when it was retrieved, the GlobalID::Locator will fail. With ActiveJob, this means that the job cannot run and is discarded. For Operations, we attempt to deserialise a second time, returning the GlobalID string instead of the model. So be aware that when you access data or results you may receive a string (similar to "gid://test-app/User/1") instead of the models you were expecting. And the error handling deserialiser is very simple so you may get format changes in some of the data as well. If serialisation fails you can access the original JSON string as data.raw_data or results[:raw_data].

TODO: Replace the ActiveJob::Arguments deserialiser with the transporter from plumbing

Failures and exceptions

If any handlers raise an exception, the task will be terminated. It will be marked as failed? and the results hash will contain results[:failure_message], results[:exception_class] and results[:exception_backtrace] for the exception's message, class name and backtrace respectively.

You can also stop a task at any point by calling fail_with message. This will raise an Operations::Failure exception, marking the task as failed? and the results has will contain results[:failure_message].

Task life-cycle and the database

There is an ActiveRecord migration that creates the operations_tasks table. Use bin/rails operations:install:migrations to copy it to your application, then run bin/rails db:migrate to add the table to your application's database.

When you call a task, it is written to the database. Then whenever a state transition occurs, the task record is updated.

This gives you a number of possibilities:

  • you can access the results (or error state) of a task after it has completed
  • you can use TurboStream broadcasts to update your user-interface as the state changes - see "status messages" below
  • tasks can run in the background (using ActiveJob) and other parts of your code can interact with them whilst they are in progress - see "background operations" below
  • the tasks table acts as an audit trail or activity log for your application

However, it also means that your database table could fill up with junk that you're no longer interested in. Therefore you can specify the maximum age of a task and, periodically, clean old tasks away. Every task has a delete_at field that, by default, is set to 90.days.from_now. This can be changed by calling Operations::Task.delete_after 7.days (or whatever value you prefer) in an initializer. Then, run a cron job, or other scheduled task, once per day that calls Operations::Task.delete_expired. This will delete any tasks whose delete_at time has passed.

Status messages

Documentation coming soon.

Sub tasks

Any operation can be composed out of other operations and can therefore call other subtasks.

class PrepareDownload < Operations::Task 
  inputs :user, :document 
  starts_with :get_authorisation
  
  action :get_authorisation do 
    inputs :user, :document 

    results = call GetAuthorisation, user: user, document: document 
    self.authorised = results[:authorised]

    go_to :whatever_happens_next
  end
end

If the sub-task succeeds, call returns the results from the sub-task. If it fails, then any exceptions are re-raised.

You can also access the results in a block:

class PrepareDownload < Operations::Task 
  inputs :user, :document 
  starts_with :get_authorisation
  
  action :get_authorisation do 
    inputs :user, :document 

    call GetAuthorisation, user: user, document: document do |results|
      self.authorised = results[:authorised]
    end

    go_to :whatever_happens_next
  end
end

If you want the sub-task to be a background task, use start instead of call. This will return the newly created sub-task immediately. As the sub-task will not have completed, you cannot access its results unless you wait (which is only possible if the parent task is a background task as well).

Background operations and pauses

If you have ActiveJob configured, you can run your operations in the background.

Instead of using call, use start to initiate the operation. This takes the same data parameters and returns a task object that you can refer back to. But it will be waiting? instead of in_progress? or completed?. An Operations::TaskRunnerJob will be queued and it will mark the task as in_progress?, then call a single state handler. Instead of handling the next state immediately another Operations::TaskRunnerJob is queued. And so on, until the task either fails or is completed.

By itself, this is not particularly useful - it just makes your operation take even longer to complete.

But if your operation may need to wait for something else to happen, background tasks are perfect.

Waiting for user input

For example, maybe you're handling a user registration process and you need to wait until the verification link has been clicked. The verification link goes to another controller and updates the user record in question. Once clicked, you can then notify the administrator.

class UserRegistration < Operations::Task 
  inputs :email
  starts_with :create_user 
  
  action :create_user do 
    inputs :email 

    self.user = User.create! email: email
    go_to :send_verification_email 
  end 

  action :send_verification_email do 
    inputs :user 

    UserMailer.with(user: user).verification_email.deliver_later 
    go_to :verified? 
  end 

  wait_until :verified? do 
    condition { user.verified? }
    go_to :notify_administrator 
  end 

  action :notify_administrator do 
    inputs :user 

    AdminMailer.with(user: user).verification_completed.deliver_later
  end
end

@task = UserRegistration.start email: "someone@example.com"

Because background tasks use ActiveJobs, every time the verified? condition is evaluated, the task object (and hence its data) will be reloaded from the database. So the user.verified? property will be refreshed on each evaluation.

Waiting for sub-tasks to complete

Alternatively, you may have a number of sub-tasks that you want to run in parallel then continue once they have all completed. This allows you to spread their execution across multiple processes or even servers (depending upon how your job queue processes are configured).

class ParallelTasks < Operations::Task 
  inputs :number_of_sub_tasks
  starts_with :start_sub_tasks 
  
  action :start_sub_tasks do 
    inputs :number_of_sub_tasks
    self.sub_tasks = (1..number_of_sub_tasks).collect { |i| start LongRunningTask, number: i }
    go_to :do_something_else 
  end 

  action :do_something_else do 
    # do something else while the sub-tasks do their thing
    go_to :sub_tasks_completed? 
  end 

  wait_until :sub_tasks_completed? do 
    condition { sub_tasks.all? { |t| t.completed? } }
    go_to :done
  end 

  result :done 
end

@task = ParallelTasks.start number_of_sub_tasks: 5

The principle is the same as above; we store the newly created sub-tasks in our own data property. As they are ActiveRecord models, they get reloaded each time sub_tasks_completed? is evaluated - and we check to see that they have all completed before moving on.

Delays and Timeouts

When you run an operation in the background, it schedules an ActiveJob which performs the individual state handler, scheduling a follow-up job for subsequent states. By default, these jobs are scheduled to run after 1 second - so a five state operation will take a minimum of five seconds to complete (depending upon the backlog in your job queues). This is to prevent a single operation from starving the job process of resources and allow other ActiveJobs the time to execute.

However, if you know that your wait_until condition may take a while you can change the default delay to something longer. In your operations definition, declare the required delay:

class ParallelTasks < Operations::Tasks 
  delay 1.minute
  ...
end

Likewise, it's possible for a background task to get stuck. In the sub-tasks example above, if one of the sub-tasks fails, waiting for them all to complete will never happen. Every operation has a default timeout of 5 minutes - if the operation has not completed or failed 5 minutes after it was initially started, it will fail with an Operations::Timeout exception.

If you need to change this (such as the user verification example above), you can declare the timeout when defining the operation. Long timeouts fit well with longer delays, so you're not filling the job queues with jobs that are meaninglessly evaluating your conditions.

class UserRegistration < Operations::Task 
  timeout 24.hours 
  delay 15.minutes
  ...
end

Testing

Because operations are intended to model long, complex, flowcharts of decisions and actions, it can be a pain coming up with the combinations of inputs to test every path through the sequence.

Instead, you can test each state handler in isolation.

As the handlers are stateless, we can call one without hitting the database; instead creating a dummy task object and then triggering the handler with the correct parameters.

This is done by calling handling, which yields a test object that we can inspect.

Testing state transitions

To test if we have moved on to another state (for actions or decisions):

MyOperation.handling(:an_action_or_decision, some: "data") do |test|
  assert_equal test.next_state, "new_state"
  # or
  expect(test).to have_moved_to "new_state"
end

Testing data modifications

To test if some data has been set or modified (for actions):

MyOperation.handling(:an_action, existing_data: "some_value") do |test|
  # has an existing data value been modified?
  assert_equal test.existing_data, "some_other_value"
  # or
  expect(test.existing_data).to eq "some_other_value"
  # has a new data value been added?
  assert_equal test.new_data, "new_value"
  # or
  expect(test.new_data).to eq "new_value"
end

Testing results

To test the results from a result handler:

MyOperation.handling(:a_result, some: "data") do |test|
  assert_equal test.outcome, "everything is as expected"
  # or
  assert_equal test[:outcome], "everything is as expected"
  # or
  expect(test.outcome).to eq "everything is as expected"
  # or
  expect(test[:outcome]).to eq "everything is as expected"
end

(Note - although results are stored in the database as a Hash, within your test, the results object is still carried as an OpenStruct, so you can access it using either notation).

Testing sub-tasks

MyOperation.handling(:a_sub_task, some: "data") do |test|
  # Test which sub-tasks were called
  assert_includes test.sub_tasks.keys, MySubTask
  # or 
  expect(test.sub_tasks).to include MySubTask
end

TODO: I'm still figuring out how to test the data passed to sub-tasks. And calling a sub-task will actually execute that sub-task, so you need to stub MySubTask.call if it's an expensive operation.

# Sorry, don't know the Minitest syntax for this
@sub_task = double "Operations::Task", results: { some: "answers" }
allow(MySubTask).to receive(:call).and_return(@sub_task)

MyOperation.handling(:a_sub_task, some: "data") do |test|
  expect(test.sub_tasks).to include MySubTask
end

Testing failures

To test if a handler has failed:

expect { MyOperation.handling(:a_failure, some: "data") }.to raise_error(SomeException)

If you are using RSpec, you must require "operations/matchers" to make the matchers available to your specs.

Installation

Step 1: Add the gem to your Rails application's Gemfile:

gem "standard_procedure_operations"

Step 2: Run bundle install, then copy and run the migrations to add the tasks table to your database:

bin/rails operations:install:migrations 
bin/rails db:migrate

Step 3: Create your own operations by inheriting from Operations::Task and revel in the stateful flowcharts!

class DailyLife < Operations::Task
  starts_with :am_i_awake?

  decision :am_i_awake? do 
    condition { (7..23).include?(Time.now.hour) }

    if_true :live_like_theres_no_tomorrow 
    if_false :rest_and_recuperate
  end 

  result :live_like_theres_no_tomorrow 
  result :rest_and_recuperate
end

Step 4: If you're using RSpec for testing, add `require "operations/matchers" to your "spec/rails_helper.rb" file.

License

The gem is available as open source under the terms of the LGPL License. This may or may not make it suitable for your needs.

Roadmap

  • Specify inputs (required and optional) per-state, not just at the start
  • Always raise errors instead of just recording a failure (will be useful when dealing with sub-tasks)
  • Deal with actions that have forgotten to call go_to (probably related to future pause functionality)
  • Simplify calling sub-tasks (and testing them)
  • Figure out how to stub calling sub-tasks with known results data
  • Figure out how to test the parameters passed to sub-tasks when they are called
  • Split out the state-management definition stuff from the task class (so you can use it without subclassing Operations::Task)
  • Make Operations::Task work in the background using ActiveJob
  • Add pause/resume capabilities (for example, when a task needs to wait for user input)
  • Add wait for sub-tasks capabilities
  • Add ActiveModel validations support for task parameters
  • Option to change background job queue and priority settings
  • Replace the ActiveJob::Arguments deserialiser with the transporter from plumbing
  • Maybe? Split this out into two gems - one defining an Operation (pure ruby) and another defining the Task (using ActiveJob as part of a Rails Engine)