AllMyCircuits
AllMyCircuits is intended to be a threadsafe-ish circuit breaker implementation for Ruby. See Martin Fowler's article on the Circuit Breaker pattern for more info.
Usage
class MyService
include Singleton
def initialize
@circuit_breaker = AllMyCircuits::Breaker.new(
name: "my_service",
watch_errors: [Timeout::Error], # defaults to AllMyCircuits::Breaker.net_errors,
# which includes typical net/http and socket errors
sleep_seconds: 10, # leave circuit open for 10 seconds, than try to call the service again
strategy: AllMyCircuits::Strategies::PercentageOverWindowStrategy.new(
requests_window: 100, # number of requests to calculate the average failure rate for
failure_rate_percent_threshold: 25 # open circuit if 25% or more requests within 100-request window fail
# must trip open again if the first request fails
}
)
end
def run
begin
@breaker.run do
Timeout.timeout(1.0) { my_risky_call }
end
rescue AllMyCircuits::BreakerOpen => e
# log me somewhere
rescue
# uh-oh, risky call failed once
end
end
end
Testing
So, what have we got:
- Time-sensitive code: ...check
- Concurrent code: ...check
Dude, that's a real headache for someone who's not confortable enough with concurrent code. I haven't figured any awesome way of automated testing in this case. So, in the script folder, there are:
fake_service.rb
the Fake Service has 3 modes of operation: up (default), die and slowdown.
-
up(default) - http://localhost:8081/up - normal mode of operation, latency up to 50ms -
die- http://localhost:8081/die - exceptions are raised left and right, slight delay in response -
slowdown- http://localhost:8081/slowdown - successful responses with a significant delay.
Graphing Stress Test
runs WORKERS number of workers which continuously hit http://localhost:8081. Graphs are served at http://localhost:8080.
This app allows to catch incorrect circuit breaker behavior visually.
Logging
Logger can be configured by setting AllMyCircuits.logger.
Default log device is STDERR, and default level is ERROR (can be overridden with ALL_MY_CIRCUITS_LOG_LEVEL environment variable).
TODO
- global controls through redis?
