Tuesday, November 4, 2008

Circuit Breaker pattern AOP style

image

 

The concept

Today I'll be talking about the circuit breaker pattern, which you can find in "Release It!" book I wrote about recently. Name of the pattern comes from regular electrical circuit breakers that prevent the wiring form being overloaded (which may cause a fire). When the situation is back to normal, a circuit breaker can be reset to restore normal current flow. The purpose of this pattern is to decrease influence of integration points on your application's stability. By integration point I mean a piece of code that you use to call an external system. When the external system is behaving unstable, hangs or crashes, the problem can propagate to your system - for example your application will be busy retrying the external call instead of doing some useful work. Not to mention that those hammering calls could prevent the external system from recovering. So the idea behind the Circuit Breaker pattern is to continue work with reduced functionality instead of retrying external calls.  Of course this depends on your scenario, because further operation without the external system might be impossible.

I am aware of two existing implementations of the Circuit Breaker pattern, but I wanted to try this out myself as an exercise. I didn't peek at any of those two before doing my own implementation.

The design

The circuit breaker pattern employs state automaton as shown below.

image

When application starts, circuit breaker is in closed state, which means it allows calls to external system. When certain failure threshold is reached, circuit breaker opens and does not allow external calls. I assumed that failure threshold is reached when there was a certain number of failures over a defined time period in the past. Circuit breaker stays in open state for a defined period of time. After that it transitions to half-open state. The external calls are allowed in half open state, but when failure happens, circuit breaker returns to open state. After a defined period of time in half open state, circuit breaker transitions to closed state. This is a modification of original circuit breaker patter. In original pattern a successful call cause circuit breaker to transition to closed state. I think that this modification may be a good move, but it'd have to try it in production.

You'll find a class diagram representing my circuit breaker implementation below.

image

This is a variation of state machine pattern. The main point of interest is the ICircuitBreaker interface. The method ShouldInvokeTarget return true if an external entity should call external system. AddFailure method adds a failure date to failure list. This list is examined to find whether a failure frequency over a defined period of time was reached. ICircuitBreaker is implemented by CircuitBreakerContext and also classes that represent states. The context forwards ICircuitBreaker calls to the current state (a object of class representing current state). The state classes are responsible for transitions. All transitions happen either when a failure is added or based on time event.

The repository is used to create and manage circuit breaker. Each circuit breaker in application is named and therefor can be referred to by name.

So far so good. But this circuit breaker is useless without something using it to determine whether external call can be performed. To provide this mechanism I decided to use the interception mechanism built into Unity 1.2 (which now is a part of Enterprise Library 4.1). The interception is built on Policy Injection Application Block. Basically what I want to achieve is a way of declaratively turning on a circuit breaker on calls to a class that is a proxy to external system. Suppose that class implements interface ITarget. This is what I want to have:

[CircuitBreaker("TEST")]
public interface ITarget
{
    void CallExternalSystem();
}

This means that every call to CallExternalSystem method should first pass through a circuit breaker of name TEST. If ITarget had more methods, instead of placing an attribute on interface, I could place it on individual methods selecting which of them should use which circuit breakers.

Unity together with PIAB provide a nice way to achieve this. Take a look at the diagram below:

image

The CircuitBreakerAttribute creates a CircuitBreakerHandler which will intercept any calls to CallExternalSystem method. Then depending of circuit breaker state it will either allow a call or throw CircuitBreakerOpenException. The diagram below shows how this functions in a case, when the call is permitted and it fails:

image

After registering ITarget and its implementation in Unity container, it will automatically wrap instance returned by Resolve with interceptor.

The code

Full source together with unit tests is available here. You'll have to get yourself NUnit and Enterprise Library 4.1 in order to run this. There are some highlights shown further.

First is an example of implementation of state class. The ClosedState is defined as follows:

public class ClosedState : StateBase
{
    public ClosedState() : base() { }
    
    public ClosedState(IContext context)
        : base(context)
    {
    }
    
    public override bool ShouldInvokeTarget()
    {
        return true;
    }

    public override void AddFailure()
    {
        RemoveExpiredFailures();
        
        Context.Failures.Enqueue(DateTime.Now);

        if (Context.Failures.Count >= Context.FrequencyThreshold)
            Context.SwitchToState(Context.OpenState);
    }

    public override void OnEnter()
    {
        base.OnEnter();
        Context.Failures.Clear();
    }

    private void RemoveExpiredFailures()
    {
        if (Context.Failures.Count == 0)
            return;
        
        while (Context.Failures.Peek() < DateTime.Now - Context.ProbingPeriod)
        {
            Context.Failures.Dequeue();
        }
    }
}

The ShouldInvokeTarget method always returns true since it's the circuit breaker's behavior in closed state. When adding new failure using AddFailure method, a first step is to remove any expired failure notifications (the ones that are older than ProbingPeriod). Then new failure is added. If the failure count exceeds FrequencyThreshold, circuit breaker transitions to open state. Here's the test for that transition (using RhinoMocks):

[Test]
public void ClosedState_Should_Transition_To_OpenState_On_Threshold()
{
    var queue = new Queue<DateTime>();
    var mocks = new MockRepository();
    var context = mocks.StrictMock<IContext>();
    var openState = mocks.DynamicMock<StateBase>();
    context.Expect(c => c.FrequencyThreshold).Return(2).Repeat.Any();
    context.Expect(c => c.ProbingPeriod).Return(TimeSpan.FromDays(1)).Repeat.Any();
    context.Expect(c => c.OpenState).Return(openState).Repeat.Any();
    context.Expect(c => c.Failures).Return(queue).Repeat.Any();
    // called once
    context.Expect(c => c.SwitchToState(openState)).Repeat.Once();

    mocks.ReplayAll();

    var closedState = new ClosedState(context);
    closedState.AddFailure();
    closedState.AddFailure();

    mocks.VerifyAll();
}

This test expects SwitchToState method to be called once as threshold (2) is reached.

Implementations of other states are similar. Download the solution if you're interested. The Unity bits are more interesting. This is the implementation of CircuitBreakerAttribute:

public class CircuitBreakerAttribute : HandlerAttribute
{
    private string name;

    public CircuitBreakerAttribute(string name)
    {
        this.name = name;
    }
    
    public override ICallHandler CreateHandler(IUnityContainer container)
    {
        return new CircuitBreakerHandler(name);
    }
}

And CircuitBreakerHandler:

public class CircuitBreakerHandler : ICallHandler
{
    private string name;

    public CircuitBreakerHandler(string name)
    {
        this.name = name;
        if (CircuitBreakerRepository.Get(name) == null)
        {
            throw new ArgumentException(
                "The circuit breaker of specified name does not exist", "name");
        }
    }

    #region ICallHandler Members

    public IMethodReturn Invoke(IMethodInvocation input, GetNextHandlerDelegate getNext)
    {
        IMethodReturn methodReturn;
        
        var circuitBreaker = CircuitBreakerRepository.Get(name);
        if (circuitBreaker.ShouldInvokeTarget())
        {
            // call next member of chain
            methodReturn = getNext()(input, getNext);
            
            if ((methodReturn.Exception != null) 
                && !(methodReturn.Exception is CircuitBreakerOpenException))
            {
                circuitBreaker.AddFailure();
            }
            return methodReturn;
        }
        else
        {
            return input.CreateExceptionMethodReturn(
                new CircuitBreakerOpenException("The circuit breaker is open"));
        }
    }

    public int Order { get; set; }

    #endregion
}

The handler intercepts calls to a target method. It's associated with circuit breaker of certain name. When call begins, the handler first asks the circuit breaker if the call is allowed. If it's not, the CircuitBreakerOpenException is thrown. Otherwise the target method is called (or more specifically the next handler in chain). If the method throws an exception, the handler notifies the circuit breaker about it. There's also a check if exception is of type CircuitBreakerOpenException. This can happen when there are more than one circuit breaker handlers in the chain.

Now the tests:

[TestFixture]
public class HandlerTests
{
    IUnityContainer container;

    public const string CircuitBreakerName = "TEST";

    [SetUp]
    public void Setup()
    {
        container = new UnityContainer();

        container.AddNewExtension<Interception>();
        container.Configure<Interception>()
            .SetInterceptorFor<ITarget>(new InterfaceInterceptor());

        if (CircuitBreakerRepository.Get(CircuitBreakerName) != null)
            CircuitBreakerRepository.Remove(CircuitBreakerName);
    }

    [Test]
    [ExpectedException(typeof(CircuitBreakerOpenException))]
    public void Target_Should_Throw_CircuitBreakerOpenException_When_Threshold_Reached()
    {
        CircuitBreakerRepository.Create(CircuitBreakerName, TimeSpan.FromSeconds(10),
            TimeSpan.FromSeconds(10), TimeSpan.FromSeconds(30), 3);

        var mocks = new MockRepository();
        var target = mocks.DynamicMock<ITarget>();
        target.Expect(t => t.CallExternalSystem())
            .Throw(new InvalidOperationException()).Repeat.Any();

        mocks.ReplayAll();

        container.RegisterInstance<ITarget>(target);
        var resolvedTarget = container.Resolve<ITarget>();
        for (int i = 0; i < 5; i++)
        {
            try
            {
                resolvedTarget.CallExternalSystem();

            }
            catch (InvalidOperationException)
            {
                // do nothing
            }
        }
    }

    [Test]
    public void Target_Should_Not_Throw_When_Threshold_Not_Reached()
    {
        CircuitBreakerRepository.Create(CircuitBreakerName, TimeSpan.FromSeconds(10),
            TimeSpan.FromSeconds(10), TimeSpan.FromMilliseconds(400), 3);

        var mocks = new MockRepository();
        var target = mocks.DynamicMock<ITarget>();
        target.Expect(t => t.CallExternalSystem())
            .Throw(new InvalidOperationException()).Repeat.Any();

        mocks.ReplayAll();

        container.RegisterInstance<ITarget>(target);
        var resolvedTarget = container.Resolve<ITarget>();
        for (int i = 0; i < 6; i++)
        {
            try
            {
                Thread.Sleep(250);
                resolvedTarget.CallExternalSystem();
            }
            catch (InvalidOperationException)
            {
                // do nothing
            }
        }
    }
}

 

What's left to do

This implementation of circuit breaker surely isn't production ready. What it lacks the most is a way to report state changes and to check it's current status. This information is critical in production environments - it helps you monitor application health and debug problems. These aspects are not covered in this post.

Hope this helps!

DotNetKicks Image

No comments:

Share