[WiP] Faymora

Clustering and distributed computing made easy.

• Distributed architecture.
  • Agents placed on nodes (machines) to provide spawning capabilities of worker Processes.
• Secure communications utilising the mutual-TLS approach.
  • Certificate Authority (RAKTR_TLS_CA)
  • Server
    • Certificate (RAKTR_TLS_SERVER_CERTIFICATE)
    • Private Key (RAKTR_TLS_SERVER_PRIVATE_KEY)
    • Public Key (RAKTR_TLS_SERVER_PUBLIC_KEY)
  • Client
    • Certificate (RAKTR_TLS_CLIENT_CERTIFICATE)
    • Private Key (RAKTR_TLS_CLIENT_PRIVATE_KEY)
    • Public Key (RAKTR_TLS_CLIENT_PUBLIC_KEY)
• Load-balancing of Processes via a network (Grid) of Agents.
  • No need to setup a topology manually, Nodes will reach convergence on their own, just point them to an existing Grid member.
  • Scaling up and down can be easily achieved by plugging or unplugging nodes.
  • Horizontal (default) and vertical workload distribution strategies available.
    • weight floating point attribute supported (as an option to Agent) in order to tip the scales of specific _Agent_s, as well as the current strategy, by changing signs (+, -).
    • Can be adjusted on boot, on demand, runtime, on the fly.
  • Fault tolerant -- one application per process (Instance).
  • Self-healing -- keeps an eye out for disappearing and also re-appearing members.
• Elasticity at the worker level.
  • Faymora will automatically spawn and kill workers as needed, to match the throughput of the pipeline from start to finish.
• Effortless provisioning of resources per worker.
  • Just state your recommended resource requirements and the right place for your Instance will be found.
• Grid-wide support for a communications Channel across all Grid Nodes.
  • Can create as many Channels as needed on demand.
• Easy to implement Add-on functionality to Nodes without inheriting/monkey-patching or messing with the internals.

For example, lets assume we have a pipeline of 3 Instances to overall deal with Items.

entry -> processing -> output

      [High Throughput `Item` Queue]
                |
                ▼
      [Faymora Entry `Instance` for input]
                │
--------------- ▼ ------------------------
|        ┌────────────────────┐          |
|        │      Equalizer     │          |
|        └──--─────┬───────┬──┘          |
|          │       |       │             |
|          ▼ spawn ▼ spawn ▼ spawn       |
|        ┌─────────┐ ┌─────────┐         |
|        │ Worker 1│ │ Worker 2│ … → N   |
|        └─────┬───┘ └─────┬───┘         |
|              │           │             |
|----------------------------------------- 
               |           |
               |           |
               ▼           ▼
              [Exhaust Output]
                     |
                     ▼
                  [ ... ]

The Equalizer component in the heart of Faymora only appears when needed based on the throughput differences of the pipeline; it's not always-on.

Seen as pressure, setting the pace:

• High: Our processing Instance is a bottleneck.
  • Spawn more of it to match the throughput of the input line.
• Even: Throughput is even, nothing to do.
• Low: Throughput is higher than input (just slipped, make it impossible), shutdown a Worker, we can spare it.

The worker processes are spawned from the least burdained Node in the Grid.

Workers are also shutdown on demand when the throughput of the pipeline matches the input throughput of its workload Items.

Basically, this design will constantly strive to match the throughput of any input for every Instance.

A work item, an OpenStruct with abstract attributes.

The data to process and be informed of framework status.

An instantiated worker class to process Items.

The Equilizer does its best to automatically match each worker's processing throughput, to the throughput of its' Item's input stream.

Meaning, if a Process is judged to be a bottleneck due to its processing latency, more of its kind will be spawned to match the throughput of the input.

A Node armoured with Instance spawning capabilities.

When the Agent is part of a Grid, the spawn will take place from the least burdained Agent member.

A representation of a node/machine which is part of the Faymora network.

The basis of an Agent.

When an existing Node (already running) is set as a peer of a new Node, a Grid/Cluster is formed automatically - same with the Agents

The Node can also be part of an Agent cluster, to have access to its Channels and Add-ons; but without Instance spawning capabilities.

Nodes can setup channels across the Cluster in order to communicate keys => values.

Channels also support callbacks for channel operations.

Abstract functionality, attached to Nodes (and Agents), without having to subclass or monkey-patch them.

See the examples/ directory.

See LICENSE.md file.