Org Converge

A framework to create documented reproducible runs using Org Mode, borrowing several ideas of what is possible to do with tools like chef-solo, rake, foreman and capistrano.

gem install org-converge

Org Mode has proven to be flexible enough to write reproducible research papers, then I believe that configuring and setting up a server for example, is something that could also be done using the same approach, given that converging the configuration is a kind of run that one ought to be able to reproduce.

Taking the original Emacs implementation as reference, Org Converge uses the Ruby implementation of the Org mode parser to implement and enhance the functionality from Org Babel by adding helpers to define the properties of a run while taking advantage of what is already there in the Ruby ecosystem.

Org Converge supports the following kind of runs below:

Each one of the code blocks is run on an independent process. This is akin to having a Procfile based application, where one of the runner command would be to start a web application and the other one to start the a worker processes.

In the following example, we are defining 2 code blocks, one that would be run using Ruby and one more that would be run in Python. Notice that the Python block has :procs set to 2, meaning that it would spawn 2 processes for this.

#+TITLE: Sample parallel run
　
#+name: infinite-worker-in-ruby
#+begin_src ruby
$stdout.sync = true
loop { puts "working!"; sleep 1; }
#+end_src
　
#+name: infinite-worker-in-python
#+begin_src python :procs 2
import sys
import time

while True:
  print "working too"
  sys.stdout.flush()
  time.sleep(1)
#+end_src

The above example can be run with the following:

org-run procfile-example.org

Sample output of the run:

[2014-06-07T18:05:48 +0900] infinite-worker-in-ruby     -- started with pid 19648
[2014-06-07T18:05:48 +0900] infinite-worker-in-python:1 -- started with pid 19649
[2014-06-07T18:05:48 +0900] infinite-worker-in-python:2 -- started with pid 19650
[2014-06-07T18:05:48 +0900] infinite-worker-in-python:1 -- working too
[2014-06-07T18:05:48 +0900] infinite-worker-in-python:2 -- working too
[2014-06-07T18:05:48 +0900] infinite-worker-in-ruby     -- working!
[2014-06-07T18:05:49 +0900] infinite-worker-in-python:1 -- working too
[2014-06-07T18:05:49 +0900] infinite-worker-in-python:2 -- working too

In case the code blocks form part of a runlist that should be ran sequentially, it is possible to do this by specifying the runmode=sequential option.

#+TITLE:   Sample sequential run
#+runmode: sequential
　
#+name: first
#+begin_src sh
echo "first"
#+end_src
　
#+name: second
#+begin_src sh
echo "second"
#+end_src
　
#+name: third
#+begin_src sh
echo "third"
#+end_src
　
#+name: fourth
#+begin_src sh
echo "fourth"
#+end_src

In order to specify that this is to be run sequentially, we set the runmode option in the command line:

org-run runlist-example.org --runmode=sequential

Another way of specifying this is via the Org mode file itself:

#+TITLE:   Defining the runmode as an in buffer setting 
#+runmode: sequential

Sample output:

[2014-06-07T18:10:33 +0900] first     -- started with pid 19845
[2014-06-07T18:10:33 +0900] first     -- first
[2014-06-07T18:10:33 +0900] first     -- exited with code 0
[2014-06-07T18:10:33 +0900] second    -- started with pid 19846
[2014-06-07T18:10:33 +0900] second    -- second
[2014-06-07T18:10:33 +0900] second    -- exited with code 0
[2014-06-07T18:10:33 +0900] third     -- started with pid 19847
[2014-06-07T18:10:33 +0900] third     -- third
[2014-06-07T18:10:33 +0900] third     -- exited with code 0
[2014-06-07T18:10:33 +0900] fourth    -- started with pid 19848
[2014-06-07T18:10:33 +0900] fourth    -- fourth
[2014-06-07T18:10:33 +0900] fourth    -- exited with code 0

For example, using Org Babel tangling functionality we can spread config files on a server by writing the following on a server.org file…

　
Configuration for a component that shoul be run in multitenant mode:
　
#+begin_src yaml :tangle /etc/component.yml
multitenant: false
status_port: 10004
#+end_src

Then run:

sudo org-tangle server.org

Note: The above tangle command is a reimplementation of the tangling functionality from Org mode using Ruby. If you are interested in tangling from the command line without losing functionality and have available a local Emacs Org mode install, the following project can be useful: https://github.com/ngirard/org-noweb

A run can have idempotency checks (similar to how the execute resource from Chef works).

An example of this, would be when installing packages. In this example, we want to install the build-essential package once, and skip it in following runs:

  ** Installing the dependencies
  　
  Need the following so that ~bundle install~ can compile 
  the native extensions correctly.
  　　
  #+name: build-essential-installed
  #+begin_src sh
  dpkg -l | grep build-essential
  #+end_src
  　 　
  #+name: build_essentials
  #+begin_src sh :unless build-essential-installed
  apt-get install build-essential -y
  #+end_src
　　
  #+name: bundle_install
  #+begin_src sh
  cd project_path
  bundle install
  #+end_src

Furthermore,since we are using Org mode syntax, it is possible to reuse this setup file by including it into another Org file:

 #+TITLE: Another setup
 　
 Include the code blocks from the server into this:
　　
 #+include: "server.org"
 　
 #+name: install_org_mode
 #+begin_src sh
 apt-get install org-mode -y
 #+end_src

Since this a run that involves converging into a state, it would be run sequentially with idempotency checks applied:

sudo org-converge setup.org

In this type of runs we use the :after and :before header arguments to specify the prerequisites for a code block to run, similar to some of the functioality provided by tools like rake (Behind the scenes, these arguments create Rake tasks)

In order for this kind of run to work, it has to be specified what is the task that we are converging to by using the #+final_task: in buffer setting:

#+TITLE:           Linked tasks example
#+runmode:         tasks
#+final_task:      final
　
#+name: second
#+begin_src sh :after first
for i in `seq 5 10`; do 
  echo $i >> out.log
done
#+end_src
　
#+name: first
#+begin_src ruby
5.times { |n| File.open("out.log", "a") {|f| f.puts n } }
#+end_src
　
#+name: final
#+begin_src python :after second :results output
print "Wrapping up with Python in the end"
f = open('out.log', 'a')
f.write('11')
f.close()
#+end_src
　
#+name: prologue
#+begin_src sh :before first :results output
echo "init" > out.log
#+end_src

org-run chained-example.org --runmode=chained

Instead of using --runmode options, it is also possible to just declare in buffer that the Org file should be run chained mode.

#+TITLE:   Defining the runmode as an in buffer setting 
#+runmode: chained

Sample output:

[2014-06-07T18:14:25 +0900] Running final task: final
[2014-06-07T18:14:25 +0900] prologue  -- started with pid 20035
[2014-06-07T18:14:25 +0900] prologue  -- exited with code 0
[2014-06-07T18:14:25 +0900] first     -- started with pid 20036
[2014-06-07T18:14:26 +0900] first     -- exited with code 0
[2014-06-07T18:14:26 +0900] second    -- started with pid 20038
[2014-06-07T18:14:26 +0900] second    -- exited with code 0
[2014-06-07T18:14:26 +0900] final     -- started with pid 20040
[2014-06-07T18:14:26 +0900] final     -- Wrapping up with Python in the end
[2014-06-07T18:14:26 +0900] final     -- exited with code 0

For any of the cases above, it is also possible to specify whether the code blocks should be run remotely on another node. This is done by using :dir in the code block header argument.

#+sshidentityfile:	vagrant/keys/vagrant
#+name: remote-bash-code-block
　
#+begin_src sh :results output :dir /vagrant@127.0.0.1#2222:/tmp
random_number=$RANDOM
for i in `seq 1 10`; do echo "[$random_number] Running script is $0 being run from `pwd`"; done
#+end_src

Note that in order for the above to work, it is also needed to set identity to be used by ssh.

In case the Org mode file has a results block which represents the expected result, there is an org-spec command which can be useful to check whether there was a change that no longer makes the results from the Org file valid. Example:

#+TITLE:   Expected results example
　　
#+name: hello
#+begin_src ruby :results output
10.times do 
  puts "hola"
end
#+end_src
　　
#+RESULTS: hello
#+begin_example
hola
hola
hola
hola
hola
hola
hola
hola
hola
hola
#+end_example

We can be able to verify whether this is still correct by running:

org-spec test.org

Checking results from 'hello' code block:	OK

As an example, let’s say that the behavior of the original code block changed, and now says hello 5 times instead. In that case the output would be as follows:

Checking results from 'hello' code block:	DIFF
@@ -1,11 +1,6 @@
-hola
-hola
-hola
-hola
-hola
-hola
-hola
-hola
-hola
-hola
+hello
+hello
+hello
+hello
+hello

The project is still in very early development and a proof of concept at this moment. But if you feel that it is interesting enough, please create a ticket to start the discussion.