Marcin Budny 's blog

Process integrity

2011-11-17T22:28:00.001+01:00

Recently I finished the “Enterprise SOA: Service-Oriented Architecture Best Practices” book by Dirk Krafzig, Karl Banke and Dirk Slama. A great book I must say, because it covers not only purely technical and architectural aspects, but also the business and policy environment with which you have to deal in large organizations actually using the SOA approach. One particular thing in the architecture part of the book got me really interested, namely process integrity.

Some theory…

Imagine this: you are creating a web service method (whoa!). The method is responsible for conducting some business process. The challenge is that this process involves interaction with two external systems over which you have no control. For example you might be booking a flight ticket along with hotel reservation (first external system) and car rental reservation (second external system). External systems tend to be, well… external, which means you have to assume than sooner or later communication with them is going to fail. What if you are able to make hotel reservation but then car rental reservation fails? You have to be able to a) detect when something goes wrong, b) be able to handle this situation for example by cancelling the hotel reservation and the ticket (compensation process).

The most natural way to handle this would be to have some kind of ACID transaction spanning all the systems involved. While it is possible to use distributed transactions in some cases, there are a lot of problems involved. First, there are technology constraints. The external system probably won’t support a compatible transaction coordination system / standard. Then there is lack of support for long-lived processes (for example reservation confirmation from hotel system takes up to an hour). Also, distributed transactions over internet are probably bad idea.

The idea to deal with that problem in SOA environment is to have so called persistent queues and transactional steps. You split the whole process into steps where each step is transactional. Then after taking each of those steps you note the fact that it either passed or failed in a persistent way. A passed step is a sign that you can perform next step in the process or finish the whole process if you have taken all the steps.

(click to enlarge)

What happens here? First we save a ticket information and enqueue the hotel reservation task. Then hotel reservation processor, which is periodically checking its queue, loads the task and processes it. After successful hotel reservation, car rental reservation task is enqueued. If hotel reservation task failed, the transaction would be rolled back, and the hotel reservation task would return into queue. Similar process happens for car rental reservation task.

The important thing to note here is that the user who initiated the process doesn’t get the information whether it was successful or not straight away. The process is asynchronous, so the user has to check its status periodically. Also all tasks and data saved to database should be marked with a common process (or transaction) id. This way we’ll be able to identify what is the state of the process.

So what can be done if one of the tasks fails? Maybe the external system is just overloaded or under maintenance. Then we can just try again later. The problem might also be with data passed to the system (for example it doesn’t pass validation). In this case trying over and over again won’t do much good. Tasks with no good chances of completing successfully should be marked as failed. Then, either automatically or manually (by a technician), the processes of which they are part should be rolled back, which usually means executing some kind of compensating logic.

…and some practice

So what is the best way to implement all that? You may don’t know it, but you’ve got all the infrastructure ready in… the Workflow Foundation. Well maybe it’s not 1:1 with what I was explaining above, but serves the purpose. Yet WF may be too heavyweight solution for you, since it requires a certain shift of thinking about building your service.

Sticking to transactional steps and queues you may wonder how to implement such a queue mechanism. I think two most popular approaches would be to have either database queue or MSMQ queue. Both have their pros and cons. MSMQ is nicely suited for processing large amounts of messages, but causes some overhead in maintenance and also requires you to promote your transactions to distributed if you want them to span the queue and the database. With database the transactions are local and there’s no maintenance overhead, but it is quite challenging to create an efficient queuing mechanism due to locking problems. Then again – you might need the integrity, not the performance.

Here I have a simple example how to create database queue and task processors working in background. I’m not working with a hotel or car rental system, but rather with Twitter and Facebook. Let’s pretend for a minute that it’s absolutely crucial for you to have your status updates hit both Twitter and Facebook in an integral manner :) In this example I’m using a nice scheduling library called Quartz.NET, and also Twitterizer and Facebook C# SDK. (It is amazing how NuGet makes putting together such mashup apps quick and easy). Grab the source here.

It’s a simple console app:

static void Main(string[] args)
{
    // required to use sql ce with entity framework
    Database.DefaultConnectionFactory = new SqlCeConnectionFactory("System.Data.SqlServerCe.4.0");

    // start scheduler
    var factory = new StdSchedulerFactory();
    var scheduler = factory.GetScheduler();
    scheduler.Start();

    // those are our task processors
    var twitterJobDetail = new JobDetail("twitterJob", null, typeof(PostOnTwitterJob));
    var facebookJobDetail = new JobDetail("facebookJob", null, typeof(PostOnFacebookJob));

    // they check the queue every 5 seconds
    var twitterTrigger = TriggerUtils.MakeSecondlyTrigger(5);
    twitterTrigger.Name = "twitter";
    var facebookTrigger = TriggerUtils.MakeSecondlyTrigger(5);
    facebookTrigger.Name = "facebook";

    // lets start them
    scheduler.ScheduleJob(twitterJobDetail, twitterTrigger);
    scheduler.ScheduleJob(facebookJobDetail, facebookTrigger);

    while (true)
    {
        // get post text from user
        var post = Console.ReadLine();
        // and initiate posting
        TaskQueue.Enqueue(new TasksDbContext(), Guid.NewGuid(), new PostOnTwitterTaskData { Text = post });
        // also you could force the twitter job to run straight away so that you don't wait <= 5s
    }
}

Basically we’re taking user input and initiate posting process by storing the post on Twitter task in the database. The database representation of the task is very simple:

public class PersistentTask
{
    public Guid Id { get; set; }
    
    /// <summary>
    /// Helps to identify the process across different tasks
    /// </summary>
    public Guid TransactionId { get; set; }

    public bool IsFailed { get; set; }

    public string Type { get; set; }

    public string SerializedData { get; set; }
}

The task queue:

public class TaskPair<TTaskData>
 {
     /// <summary>
     /// DB representation of the task
     /// </summary>
     public PersistentTask PersistentTask { get; set; }

     /// <summary>
     /// Deserialized task data
     /// </summary>
     public TTaskData TaskData { get; set; }

     public TaskPair(PersistentTask persistentTask, TTaskData taskData)
     {
         PersistentTask = persistentTask;
         TaskData = taskData;
     }
 }
 
 public static class TaskQueue
 {
     /// <summary>
     /// Puts a task into database queue
     /// </summary>
     public static void Enqueue(TasksDbContext context, Guid trasactionId, object taskData)
     {
         Console.WriteLine("Enqueueing {0} with transactionId = {1}", taskData.GetType().Name, trasactionId);
         
         var serializer = new XmlSerializer(taskData.GetType());
         var writer = new StringWriter();
         serializer.Serialize(writer, taskData);

         var persistentTask = new PersistentTask
         {
             Id = Guid.NewGuid(),
             TransactionId = trasactionId,
             IsFailed = false,
             Type = taskData.GetType().Name,
             SerializedData = writer.ToString()
         };

         context.PersistentTasks.Add(persistentTask);
         context.SaveChanges();
     }

     /// <summary>
     /// Dequeues a task from database queue and deserializes its data
     /// </summary>
     public static TaskPair<TTaskData> Dequeue<TTaskData>(TasksDbContext context)
     {
         var task = context.PersistentTasks
             .Where(pt => pt.Type == typeof(TTaskData).Name && !pt.IsFailed).FirstOrDefault();
         if (task == null)
         {
             return null;
         }

         Console.WriteLine("Dequeued {0} with transactionId = {1}", typeof(TTaskData).Name, task.TransactionId);

         context.PersistentTasks.Remove(task);
         context.SaveChanges();

         var serializer = new XmlSerializer(typeof(TTaskData));
         var reader = new StringReader(task.SerializedData);

         return new TaskPair<TTaskData>(task, (TTaskData)serializer.Deserialize(reader));
     }

     /// <summary>
     /// Marks a certain task as failed
     /// </summary>
     public static void MarkFailed(TasksDbContext context, Guid taskId)
     {
         var task = context.PersistentTasks.SingleOrDefault(pt => pt.Id == taskId);
         if (task != null)
         {
             task.IsFailed = true;
             context.SaveChanges();
         }
     }
 }

And the jobs (or task processors):

public abstract class TaskJobBase<TTaskDescription> : IStatefulJob
{
    public void Execute(JobExecutionContext jobContext)
    {
        TaskPair<TTaskDescription> pair = null;
        ObjectContext objctx = null;

        using (var context = new TasksDbContext())
        {
            // unfortunately with sql ce you have to manually open/close connection or it'll not enlist in transaction
            objctx = ((IObjectContextAdapter)context).ObjectContext;
            objctx.Connection.Open();

            // foreach task waiting in the database
            do
            {
                try
                {
                    // open transaction
                    using (var scope = new TransactionScope())
                    {
                        // dequeue task
                        pair = TaskQueue.Dequeue<TTaskDescription>(context);
                        if (pair != null)
                        {
                            // execute it
                            PerformTask(context, pair);
                            // if ok - commit transaction
                            scope.Complete();
                        }
                    }
                }
                catch
                {
                    // transaction was rolled back
                    Console.WriteLine("Task {0} with transaction id = {1} failed!", pair.TaskData.GetType().Name,
                        pair.PersistentTask.TransactionId);

                    // in this example we also mark the task as failed                        
                    if (pair != null)
                    {
                        using (var scope = new TransactionScope())
                        {
                            TaskQueue.MarkFailed(context, pair.PersistentTask.Id);
                            scope.Complete();
                        }
                    }
                }
            } while (pair != null);

            objctx.Connection.Close();
        }
    }

    public abstract void PerformTask(TasksDbContext context, TaskPair<TTaskDescription> pair);
}

public class PostOnTwitterJob : TaskJobBase<PostOnTwitterTaskData>
{
    public override void PerformTask(TasksDbContext context, TaskPair<PostOnTwitterTaskData> pair)
    {
        Console.WriteLine("Posting \"{0}\" on Twitter", pair.TaskData.Text);

        OAuthTokens tokens = new OAuthTokens();
        tokens.AccessToken = "[your access token]";
        tokens.AccessTokenSecret = "[your access token secret]";
        tokens.ConsumerKey = "[your consumer key]";
        tokens.ConsumerSecret = "[your consumer secret]";

        TwitterResponse<TwitterStatus> tweetResponse = TwitterStatus.Update(tokens, pair.TaskData.Text);
        if (tweetResponse.Result != RequestResult.Success)
        {
            throw new Exception("Unable to post to twitter!");
        }

        // ok, posted! now the next step - facebook
        // it's still in the transaction!
        TaskQueue.Enqueue(context, pair.PersistentTask.TransactionId, new PostOnFacebookTaskData { Text = pair.TaskData.Text });
    }
}

public class PostOnFacebookJob : TaskJobBase<PostOnFacebookTaskData>
{
    public override void PerformTask(TasksDbContext context, TaskPair<PostOnFacebookTaskData> pair)
    {
        Console.WriteLine("Posting \"{0}\" on Facebook", pair.TaskData.Text);

        // in real world app you need to refresh this token since it expires after ~2h
        var fb = new FacebookClient("[your access token]");
 
        dynamic parameters = new ExpandoObject();
        parameters.message = pair.TaskData.Text;

        dynamic result = fb.Post("me/feed", parameters);
        var id = result.id;

        // nothing left to do - process finished
    }
}

Of course the app is not something I would put into production, but it should give you an idea how things can be organized. Grab the source here.

WebSockets in .NET 4.5: a simple game

2011-10-13T22:18:00.001+02:00

Websockets are one of the new great web standards, which applied in a right way, will enable development of completely new web applications. Microsoft introduces native support for them in .NET 4.5 and IE 10. Other browsers (Firefox 7, Chrome 14) already support the standard. To play with the new technology I created a simple game of pong (download code here):

Basically when two players open the website, the game begins. If further players connect, new games are created for them. If one of the players in a game disconnects, the game ends.

There are some interesting aspects of this sample:

connections from players are grouped in games and the messages are broadcasted among game participants
there is a thread on the server responsible for moving the ball and broadcasting its position
different kinds of messages are sent

In order to run the sample you’ll need:

Windows 8 developer preview
IIS 8 with websockets module installed
Visual Studio 11 developer preview + NuGet
ASP.NET MVC 4 for Visual Studio 11

In this post I’m only going to show most interesting parts related to websockets usage, if you want to see more, grab the source.

The client

The client is the browser so we’ll be coding javascript. I prefer to use OO javascript so this is the application’s structure:

function Pong(..some initialization parameters..) {
    ...initialization code and fields here...
}

Pong.prototype = {
    ...methods here...
};

Following code shows how to open a WebSocket on client side:

    // start connecting websockets
    // Firefox 6 and 7 requires Moz prefix...
    if(typeof(MozWebSocket) == "function") {
        this.socket = new MozWebSocket(applicationUrl);
        this.openStateConst = MozWebSocket.OPEN;
    } else {
        this.socket = new WebSocket(applicationUrl);
        this.openStateConst = WebSocket.OPEN;
    }

    // register to socket events
    this.socket.onopen = $.proxy(function () {
        this.info("You are connected...");
    }, this);
    this.socket.onclose = $.proxy(function () {
        this.info("Other player disconnected! Refresh page to restart game");
    }, this);
    this.socket.onerror = $.proxy(function () {
        this.info("You have been disconnected! Sorry for that, refresh page to restart game");
    }, this);
    this.socket.onmessage = $.proxy(function (msg) {
        this.processMessage(msg);
    }, this);

Unfortunately as of Firefox 7, websocket API is still treated as a vendor extension, so you have to use Moz prefix. Otherwise it works great. Next code fragment shows how the incoming messages from server are handled:

    // processes message from server basing on its type
    processMessage: function (msg) {
        var data = JSON.parse(msg.data);

        switch (data.Type) {
            case "PlayerNumberMessage":
                this.myPlayer = data.PlayerNumber;
                if (this.myPlayer == 0) {
                    this.info("<span style='color:red'>You are red.</span>");
                } else {
                    this.info("<span style='color:blue'>You are blue.</span>");
                }
                break;

            case "PlayerPositionMessage":
                this.otherPlayer().yPos = data.YPos;
                this.draw();
                break;

            case "BallPositionMessage":
                this.ball.xPos = data.XPos;
                this.ball.yPos = data.YPos;
                this.draw();
                break;

            case "ScoreMessage":
                this.score = data.Score;
                this.displayScore();
                break;
        }
    }

Data is sent in JSON, so first thing to do is to deserialize it. Messages carry a property named Type that allows to differentiate between different kinds of data being sent. Message types include other player’s position, ball’s position, score change and a message sent when player is assigned his side (red or blue). I think this is quite self-explanatory.

One more thing on client’s side to mention is sending the player’s position to server:

    // register on mousemove over canvas
    $(canvasSelector).mousemove($.proxy(function (e) {
        // has player been initialized?
        if (this.myPlayer == -1)
            return;

        // calulate mouse vertical position relative to canvas
        var offset = $(this.canvas).offset();
        var mouseY = e.pageY - offset.top;

        // constraint movement
        if (mouseY < this.playerHeight / 2)
            mouseY = this.playerHeight / 2;
        if (mouseY > this.canvas.height - this.playerHeight / 2)
            mouseY = this.canvas.height - this.playerHeight / 2;

        this.players[this.myPlayer].yPos = mouseY;

        // send message to server with new position
        this.sendMessage({ YPos: mouseY });

        this.draw();
    }, this));

In reaction to mousemove over the canvas, a message with new player position is sent to the server. Sending the message:

    // serializes and sends message to server
    sendMessage: function (msg) {
        if (this.socket != "undefined" && this.socket.readyState == this.openStateConst) {
            var msgText = JSON.stringify(msg);
            this.socket.send(msgText);
        }
    },

Quite easy and strightforward.

The server

Entry point for server code is an implementation of IHttpHandler that accepts websocket connections:

    /// <summary>
    /// Http handler accepting web socket requests
    /// </summary>
    public class PongHttpHandler : IHttpHandler
    {
        public bool IsReusable
        {
            get { return false; }
        }

        public void ProcessRequest(HttpContext context)
        {
            if (context.IsWebSocketRequest)
            {
                // create a player
                var player = new PongPlayer();
                PongApp.JoinPlayer(player);

                // start receiving from socket
                context.AcceptWebSocketRequest(player.Receiver);
            }
        }
    }

HttpContext.IsWebSocketRequest is a new property returning true if incoming connection is a websocket request. Note, that as of now, you have to be using IIS 8 with websockets module installed. Otherwise this property always returns false.

In this case we accept the connection using AcceptWebSocketRequest by providing it with a delegate, that will asynchronously take care of receiving messages. This a little confusing, because there are two different ways of doing this that you can find by browsing different blogs. The other way is to provide an instance of WebSocketHandler class. The thing is that MSDN knows nothing about such class, and neither does it know about WebSocketCollection. Strange.

Anyway, the PongPlayer class representing a connection to a player looks like this:

    /// <summary>
    /// Represents a player and wraps his websocket operations
    /// </summary>
    public class PongPlayer
    {
        private PongGame _game;
        private object _syncRoot = new object();
        private AspNetWebSocketContext _context;

        public event Action<PongPlayer, PlayerPositionMessage> PlayerMoved;
        public event Action<PongPlayer> PlayerDisconnected;

        // player vertical positon
        public int YPos { get; private set; }

        public void SetGame(PongGame game)
        {
            if (_game != null)
                throw new InvalidOperationException();
            _game = game;
        }

        /// <summary>
        /// This method is used as delegate to accept WebSocket connection
        /// </summary>
        public async Task Receiver(AspNetWebSocketContext context)
        {
            _context = context;
            var socket = _context.WebSocket as AspNetWebSocket;
            // prepare buffer for reading messages
            var inputBuffer = new ArraySegment<byte>(new byte[1024]);

            // send player number to player
            SendMessage(new PlayerNumberMessage { PlayerNumber = _game.GetPlayerIndex(this) });

            try
            {
                while (true)
                {
                    // read from socket
                    var result = await socket.ReceiveAsync(inputBuffer, CancellationToken.None);
                    if (socket.State != WebSocketState.Open)
                    {
                        if (PlayerDisconnected != null)
                            PlayerDisconnected(this);
                        break;
                    }

                    // convert bytes to text
                    var messageString = Encoding.UTF8.GetString(inputBuffer.Array, 0, result.Count);
                    // only PlayerPositionMessage is expected, deserialize
                    var positionMessage = JsonConvert.DeserializeObject<PlayerPositionMessage>(messageString);

                    // save new position and notify game
                    YPos = positionMessage.YPos;
                    if (PlayerMoved != null)
                        PlayerMoved(this, positionMessage);

                }
            }
            catch (Exception ex)
            {
                if (PlayerDisconnected != null)
                    PlayerDisconnected(this);
            }
        }

        /// <summary>
        /// Sends message through web socket to player's rowser
        /// </summary>
        public async Task SendMessage(object message)
        {
            // serialize and send
            var messageString = JsonConvert.SerializeObject(message);
            if (_context != null && _context.WebSocket.State == WebSocketState.Open)
            {
                var outputBuffer = new ArraySegment<byte>(Encoding.UTF8.GetBytes(messageString));
                await _context.WebSocket.SendAsync(outputBuffer, WebSocketMessageType.Text, 
                    true, CancellationToken.None);
            }
        }

        /// <summary>
        /// Closes player's web socket
        /// </summary>
        public void Close()
        {
            if (_context != null && _context.WebSocket.State == WebSocketState.Open)
            {
                _context.WebSocket.CloseOutputAsync(WebSocketCloseStatus.NormalClosure, 
                    "Closing...", CancellationToken.None).Wait();
            }
        }
    }

Basically we’re waiting here in asynchronous manner for messages from a player in a infinite loop. When message arrives, it is deserialized and processed on the same thread. Sending messages is initiated externally by PongGame class and is performed on different thread.

The PongGame represents a game between two players and groups connections:

    public enum GameState
    {
        NotInitiated,
        WaitingForPlayer,
        InProgress
    }

    /// <summary>
    /// Represents pong game between two players
    /// </summary>
    public class PongGame
    {
        public const int LeftPlayer = 0;
        public const int RightPlayer = 1;

        ... more boring constants here ...

        private object _syncRoot = new object();
        private PongPlayer[] _players = new PongPlayer[2];
        private Vector _ballPosition = new Vector(FieldWidth / 2, FieldHeight / 2);
        private Vector _ballDirection = Vector.SW;
        private double _ballSpeed = BallStartingSpeedPixPerSecond;
        private int[] _score = new int[2];

        /// <summary>
        /// Token used to cances ball moving task
        /// </summary>
        private CancellationTokenSource _ballCancellationTokenSource;

        public GameState State { get; private set; }
        
        ... code here...
    }

When new player joins the game, JoinPlayer is called:

        /// <summary>
        /// Joins new player to this game
        /// </summary>
        /// <param name="player"></param>
        public void JoinPlayer(PongPlayer player)
        {
            lock (_syncRoot)
            {
                if (_players[LeftPlayer] != null && _players[RightPlayer] != null)
                    throw new InvalidOperationException();

                player.SetGame(this);
                player.PlayerDisconnected += OnPlayerDisconnected;
                player.PlayerMoved += OnPlayerMoved;

                if (_players[LeftPlayer] == null)
                {
                    _players[LeftPlayer] = player;
                    State = GameState.WaitingForPlayer;
                }
                else
                {
                    _players[RightPlayer] = player;
                    // we have two players, so start the game
                    State = GameState.InProgress; 
                    StartBall();
                }
            }
        }

We just store players in private field. When two players join, the game starts. There is a thread that is responsible for moving the ball and broadcasting ball’s position to both players:

        private void StartBall()
        {
            // this delegete is responsible for ball moving, bouncing and counting score
            Action<CancellationToken> ballMover = (cancellationToken) =>
            {
                var lastTime = DateTime.Now;

                while (true)
                {
                    var thisTime = DateTime.Now;
                    // how many seconds elapsed since last pass
                    var secondsElapsed = (thisTime - lastTime).TotalMilliseconds / 1000.0;

                    MoveBall(secondsElapsed);
                    lastTime = thisTime;

                    Thread.Sleep(50);

                    // finish task if cancel requested
                    if (cancellationToken.IsCancellationRequested)
                        break;
                }
            };

            // prepare cancellation token and run the task
            _ballCancellationTokenSource = new CancellationTokenSource();
            Task.Factory.StartNew(() => ballMover(_ballCancellationTokenSource.Token), 
                _ballCancellationTokenSource.Token, TaskCreationOptions.LongRunning, 
                TaskScheduler.Default);
        }

        private void MoveBall(double secondsElapsed)
        {
            lock (_syncRoot)
            {
                // calculate new position
                _ballPosition += _ballDirection * (_ballSpeed * secondsElapsed);

                ...checking for collision and bouncing the ball..

                // check for scores
                if (_ballPosition.X < 0 || _ballPosition.X > FieldWidth)
                {
                    _score[_ballPosition.X < 0 ? RightPlayer : LeftPlayer]++;
                    // broadcast score message
                    BroadcastMessage(new ScoreMessage { Score = _score });

                    ...reseting ball position...
                }

                // broadcast ball position message
                BroadcastMessage(new BallPositionMessage { XPos = (int)_ballPosition.X, 
                    YPos = (int)_ballPosition.Y });
            }
        }

        /// <summary>
        /// Sends a message to both players
        /// </summary>
        /// <param name="message"></param>
        private void BroadcastMessage(object message)
        {
            foreach (var player in _players)
            {
                player.SendMessage(message);
            }
        }

One more interesing thing about PongGame is the way it reacts to player events:

        private void OnPlayerMoved(PongPlayer player, PlayerPositionMessage position)
        {
            var otherPlayer = OtherPlayer(player);

            if (otherPlayer != null)
            {
                // send new player position to the other player
                otherPlayer.SendMessage(new PlayerPositionMessage { YPos = player.YPos });
            }
        }

        private void OnPlayerDisconnected(PongPlayer player)
        {
            lock (_syncRoot)
            {
                // stop the ball moving task
                _ballCancellationTokenSource.Cancel();
                var otherPlayer = OtherPlayer(player);

                if (otherPlayer != null)
                {
                    // close connection to other player, which means game is over
                    otherPlayer.Close();
                }
                if (GameOver != null)
                    GameOver(this);
            }
        }

The last bit here is the PongApp class which basically is a collection of ongoing games.

    /// <summary>
    /// Aggregates all games
    /// </summary>
    public static class PongApp
    {
        private static object _syncRoot = new object();

        private static List<PongGame> _games;

        static PongApp()
        {
            _games = new List<PongGame>();
        }
        
        /// <summary>
        /// Joins new player to existing game or creates new one 
        /// </summary>
        /// <param name="player"></param>
        public static void JoinPlayer(PongPlayer player)
        {
            lock (_syncRoot)
            {
                var game = _games.Where(g => g.State == GameState.WaitingForPlayer)
                    .FirstOrDefault();
                if (game == null)
                {
                    game = new PongGame();
                    _games.Add(game);
                    game.GameOver += OnGameOver;
                }
                game.JoinPlayer(player);
            }
        }

        private static void OnGameOver(PongGame game)
        {
            lock (_syncRoot)
            {
                _games.Remove(game);
            }
        }
    }

And that would be pretty much it. Again, the code is here.

Your first app with CouchDB and relax-net

2011-10-02T19:28:00.001+02:00

This is a tutorial that will show you how to create a simple tagged gallery ASP.NET MVC web app using CouchDB as a data store and relax-net as data access layer. It requires that you understand basic concepts behind CouchDB and have knowledge of ASP.NET MVC. I’ll be focusing mainly on CouchDB related code in the post, but you can download full source here.

Preparation

In order to start, you first have to have a CouchDB up and running. I suggest you grab the community edition of Couchbase Single Server for Windows. After installing and making sure that CouchbaseServer service is started, you can visit http://localhost:5984/_utils/index.html address to run the Futon – CouchDB management interface. Create new database named tagged_gallery.

Create new ASP.NET MVC 3 project and call it TaggedGallery. Download and reference relax-net. Unfortunately it’s not available as NuGet package, so either grab the zipped release or just build it from source.

What do we build?

A simple gallery that allows registered users to upload pictures that all visitors can browse. Each picture can have multiple tags associated with it. All tags related to currently visible pictures display on the side along with count of pictures with given tag. If you click a tag, you can browse all pictures with that tag.

Infrastructure

relax-net takes some design patterns after NHibernate and one of them is a concept of session. Session is something that caches your entities during its lifetime and takes care that when you get a certain entity twice, you’ll get the same object reference. As with NHibernate, the best usage in web application is session per request pattern, which means you open session on BeginRequest and close it (or “return” in relax-net) on EndRequest. We have to have some means of managing the session and accessing it. The SessionManager component ‘s purpose is just that.

public interface ISessionManager
{
    void SetConnection(Connection connection);
    
    void OpenSession(string database);
    void ReturnSession();
    Session GetCurrentSession();

    IRepository<TEntity> GetRepositoryFor<TEntity>() where TEntity : class;
    Query<TEntity> GetQueryFor<TEntity>(string design, string view, bool reduce) where TEntity : class;
}

SetConnection initializes the SessionManager by providing Connection which basically points to CouchDB server. OpenSession and ReturnSession are self explanatory. In terms of relax-net you don’t “close” the session, but rather return it to the pool. GetCurrentSession allows you to get an open session for current request from any point in your app having access to SessionManager. There are also helper methods GetRepositoryFor<TEntity> (which creates and initializes a relax-net repository for given entity type) and GetQueryFor<TEntity> (which creates and initializes a query – query is a wrapper over CouchDB view). Now for the implementation:

public class SessionManager : ISessionManager
{
    private IUnityContainer _container;
    
    public SessionManager(IUnityContainer container)
    {
        _container = container;
    }
    
    public void SetConnection(Connection connection)
    {
        Contract.Requires(connection != null, "Connection cannot be null");

        connection.Observers.Add(c => _container.Resolve<AuditObserver>());
        connection.Observers.Add(c => _container.Resolve<TypeAssigningObserver>());
        HttpContext.Current.Application["DbConnection"] = connection;
    }

    public Session GetCurrentSession()
    {
        return DoGetSession();
    }

    public void OpenSession(string database)
    {
        var connection = HttpContext.Current.Application["DbConnection"] as Connection;
        Contract.Assert(connection != null, "Connection was not set");
        var session = connection.CreateSession(database);
        HttpContext.Current.Items["DbSession"] = session;
    }

    public void ReturnSession()
    {
        var session = DoGetSession();
        session.Connection.ReturnSession(session);
    }

    private static Session DoGetSession()
    {
        var session = HttpContext.Current.Items["DbSession"] as Session;
        Contract.Assert(session != null, "Session was not opened");
        return session;
    }


    public IRepository<TEntity> GetRepositoryFor<TEntity>() where TEntity : class
    {
        var resolverOverride = new ParameterOverride("sx", DoGetSession());
        return _container.Resolve<IRepository<TEntity>>(resolverOverride);
    }


    public Query<TEntity> GetQueryFor<TEntity>(string design, string view, bool reduce) where TEntity : class
    {
        return new Query<TEntity>(DoGetSession(), design, view, reduce);
    }
}

As you can see the connection is stored in HttpApplication (as it is shared for all requests) and session is stored in HttpContext (as it is private to a request). As might also noticed I’m using Unity as dependency injection framework. You can grab it by NuGet.

All relax-net entities have to inherit from Document base class, which has some basic properties like Id and Revision. By default Id for new document is created in form of [lowercase type name]-[guid]. We will introduce one more layer over the Document class in order to additionally store easily accessible type name in all entities. It will also allow us to add some common behavior for all our entities if we need that later.

public class BaseEntity : Document
{
    public string Type { get; set; }
}

There’s also the IAuditable interface which will allow us to track who and when created or modified the entity.

public interface IAuditable
{
    DateTime? CreatedDate { get; set; }
    DateTime? LastModifiedDate { get; set; }

    Reference<User> CreatedBy { get; set; }
    Reference<User> LastModifiedBy { get; set; }
}

Reference<TEntity> is relax-net way to associate different entities. Using reference in this case you can traverse from one entity to another dotting your way through. The references are lazily loaded, which means first time when you access the reference, relax-net will request appropriate entity it from CouchDB.

We don’t want to worry about having to fill fields from IAuditable or BaseEntity by hand every time an entity is created. Fortunately we can create observers that will react to different entity lifetime events. In SessionManager the connection is initialized with two observers: AuditObserver and TypeAssigningObserver.

public class TypeAssigningObserver : IObserver
{
    public void AfterDelete(object entity, Document document)
    {
    }

    public void AfterLoad(object entity, Document document)
    {
    }

    public void AfterSave(object entity, Document document)
    {
    }

    public Disposition BeforeDelete(object entity, Document document)
    {
        return Disposition.Continue;
    }

    public Disposition BeforeSave(object entity, Document document)
    {
        var baseEntity = entity as BaseEntity;
        if (baseEntity != null)
        {
            baseEntity.Type = baseEntity.GetType().Name.ToLowerInvariant();
        }
        return Disposition.Continue;
    }
}

public class AuditObserver : IObserver
{
    private ISessionManager _sessionManager;
    
    public AuditObserver(ISessionManager sessionManager)
    {
        _sessionManager = sessionManager;
    }

    public void AfterDelete(object entity, Document document)
    {
    }

    public void AfterLoad(object entity, Document document)
    {
    }

    public void AfterSave(object entity, Document document)
    {
    }

    public Disposition BeforeDelete(object entity, Document document)
    {
        return Disposition.Continue;
    }

    public Disposition BeforeSave(object entity, Document document)
    {
        var auditable = entity as IAuditable;
        if (auditable != null)
        {
            // create lazy references instead of weak references to overcome problems while running under debbuger
            // no perf penalty
            
            // if revision is empty, this document is being created
            if (string.IsNullOrEmpty(document.Revision))
            {
                auditable.CreatedDate = DateTime.Now;
                auditable.CreatedBy = Reference.To<User>(_sessionManager.GetCurrentSession(), 
                    "user-" + HttpContext.Current.User.Identity.Name);
            }
            auditable.LastModifiedDate = DateTime.Now;
            auditable.LastModifiedBy = Reference.To<User>(_sessionManager.GetCurrentSession(),
                "user-" + HttpContext.Current.User.Identity.Name);
        }
        return Disposition.Continue;
    }
}

As you might already guessed we won’t be using a standard Id format for users, but rather something like user-[login].

Model and views

Our model will consist of two entities: Picture and User.

public class Picture : BaseEntity, IAuditable, IHasAttachments
{        
    [Required(ErrorMessage = "Title is required")]
    [StringLength(50, ErrorMessage = "Cannot be longer than 50 chars")]
    public string Title { get; set; }

    public string ContentType { get; set; }

    public List<string> Tags { get; set; }

    public DateTime? CreatedDate { get; set; }
    public DateTime? LastModifiedDate { get; set; }
    public Reference<User> CreatedBy { get; set; }
    public Reference<User> LastModifiedBy { get; set; }

    public Attachments Attachments { get; set; }
}

public class User : BaseEntity
{
    public string Login { get; set; }
    public string Email { get; set; }
    public string PasswordHash { get; set; }
}

As you can see, Picture has a collection of tags, but tag is not a separate entity as it would be in normalized relational model. Tag collection will be stored within picture document. The IHasAttachements is mandatory for relax-net entities, that have binary attachments associated with them (this scenario is supported by CouchDB).

There are also additional classes used to provide a strongly typed model for the views.

public class TagWithCount
{
    public string Tag { get; set; }
    public int Count { get; set; }
}

public class SinglePictureModel
{
    public Picture Picture { get; set; }
    public TagWithCount[] Tags { get; set; }
}

public class PictureListModel
{
    public Picture[] Pictures { get; set; }
    public TagWithCount[] Tags { get; set; }
}

To query our database efficiently and get required data we have to implement some views. Views will be stored in _design/picture design document, which will help relax-net to load them basing on entity type name. You can edit the views using Futon. First of them named by-date will allow us to list all pictures in database ordered by creation date:

// by-date

// map
function (doc) {
    if (doc.Type == 'picture')
        emit(doc.CreatedDate, doc._id);
}

This view returns CreatedDate picture property as a key, which means that returned values (ids) will be sorted by this property. Second view named by-tag-date will allow us to list pictures with given tag ordered by creation date.

// by-tag-date

//map
function (doc) {
    if (doc.Type == 'picture')
        for (var i in doc.Tags)
            emit([doc.Tags[i], doc.CreatedDate], doc._id);
}

This view emits composite key (an array of values) where first element is the tag, and second is the creation date. This means CouchDB will sort returned values first by tag, then by creation date. By putting constraints on returned key range we’ll be able to select only pictures with given tag. The last view named tags-with-count will allow us to get information of number of pictures assigned a given tag.

// tags-with-count

// map
function (doc) {
    if (doc.Type == 'picture')
        for (var i in doc.Tags)
            emit(doc.Tags[i], 1);
}

// reduce
function (keys, values, rereduce) {
    return sum(values);
}

The map part emits tag as a key and 1 as value. The reduce part takes all the “1s” returned for given tag and sums them. As a result we get picture count for that tag. You can try out those views in Futon.

The PictureController

Last really interesting bit is the PictureController which will cover all of the picture related actions. Its general structure is as follows:

public class PictureController : Controller
{
    public const int PageSize = 6;

    private SessionManager _sessionManager;

    public PictureController(SessionManager sessionManager)
    {
        _sessionManager = sessionManager;
    }

    ...
}

First, actions for adding a new picture.

/// <summary>
/// Displays picture adding form
/// </summary>
public ActionResult Add()
{
    return View();
} 

/// <summary>
/// Saves new picture to db
/// </summary>
/// <param name="picture">The picture desciption</param>
/// <param name="fileUpload">The file upload</param>
[HttpPost]
[ValidateAntiForgeryToken]
public ActionResult Add(Picture picture, string commaTagList, HttpPostedFileBase fileUpload)
{
    try
    {
        if (fileUpload == null)
        {
            ModelState.AddModelError("fileUpload", "Choose a file to upload");
            return View();
        }
        
        if (!fileUpload.ContentType.StartsWith("image/"))
        {
            ModelState.AddModelError("fileUpload", "File has to be an image");
            return View();
        }

        if (!string.IsNullOrWhiteSpace(commaTagList))
        {
            picture.Tags = commaTagList.Split(',')
                .Select(tag => tag.Trim())
                .Where(tag => !string.IsNullOrWhiteSpace(tag))
                .ToList();
        }

        var repo = _sessionManager.GetRepositoryFor<Picture>();
        repo.Save(picture);
        repo.Attach(picture, fileUpload);

        return RedirectToAction("Single", new { id = picture.Id });
    }
    catch(Exception e)
    {
        ModelState.AddModelError("", e);
        return View();
    }
}

User provides picture title, comma separated tag list and selects a file to upload. By using appropriate repository we save the document to the database and then attach a binary file with the picture. Note that saving a document and creating an attachment are two separate operations and they are not included in any transaction. Lack of transactions is a problem with CouchDB as I wrote earlier. In real world application you’d probably want some compensation code, that will remove picture document if creating attachment fails.

Now the action that displays a single picture with a tag list on its side.

public ActionResult Single(string id)
{
    var repo = _sessionManager.GetRepositoryFor<Picture>();
    var picture = repo.TryGet(id);
    if (picture == null || picture.Attachments.Count == 0)
    {
        return new HttpNotFoundResult();
    }

    var model = new SinglePictureModel
    {
        Picture = picture,
        Tags = picture.Tags != null ? LoadTagCounts(picture.Tags.ToArray()) : new TagWithCount[] { }
    };

    return View(model);
}

private TagWithCount[] LoadTagCounts(params string[] tags)
{
    var tagsWithCount = new List<TagWithCount>();
    foreach (var tag in tags)
    {
        var query = _sessionManager.GetQueryFor<TagWithCount>("picture", "tags-with-count", true)
            .All().Exactly(tag);
        var result = query.Execute();

        if(result != null && result.Rows.Length > 0)
            tagsWithCount.Add(new TagWithCount { Tag = tag, Count = int.Parse(result.Rows[0].Value.ToString()) });
    }

    return tagsWithCount.ToArray();
}

We load the picture document using TryGet(id) method od picture repository. Getting a document by its id doesn’t require creating any additional view. Then for all tags associated with this picture we want to load number of pictures under given tag. To do this we query previously created tags-with-count view and constrain returned key range to one tag. We query CouchDB as many times as there are tags, which is not very efficient. There is a feature in CouchDB that allows to load multiple keys with one go to the database. Unfortunately realx-net doesn’t currently support this feature. I sent the library’s author a patch for that, but as of now I didn’t receive any answer.

Next action sends the actual picture file to user’s browser.

/// <summary>
/// Sends picture file to browser
/// </summary>
/// <param name="id">Id of the picture</param>
public ActionResult Show(string id)
{
    var repo = _sessionManager.GetRepositoryFor<Picture>();
    var picture = repo.TryGet(id);
    if (picture == null || picture.Attachments.Count == 0)
    {
        return new HttpNotFoundResult();
    }

    var att = picture.Attachments.First().Value;
    return File(att.LoadBytes(), att.ContentType);
}

In this action we download the image file to the memory and then stream it to the user. There is another way to do this. You could have user’s browser download the file directly from CouchDB by providing it with appropriate URL. But a lot of security issues arise with this approach, so here I decided to download the file to web application’s memory first.

Next action lists all pictures in our gallery.

/// <summary>
/// Lists all pictures
/// </summary>
/// <param name="page">Page number</param>
public ActionResult Index(int? page)
{
    if (page == null)
        page = 1;
    ViewBag.Page = page.Value;

    var pictureQuery = _sessionManager.GetQueryFor<Picture>("picture", "by-date", false)
        .Skip((page.Value - 1) * PageSize)
        .Limit(PageSize)
        .Descending()
        .WithDocuments();
    var result = pictureQuery.Execute();

    ViewBag.HasMore = result.Total > (page.Value * PageSize);

    var pictures = result.Rows.Select(r => r.Entity).ToArray();
    var tags = pictures.Where(p => p.Tags != null).SelectMany(p => p.Tags)
        .Distinct().OrderBy(tag => tag).ToArray();

    var model = new PictureListModel
    {
        Pictures = pictures,
        Tags = LoadTagCounts(tags)

    };

    return View(model);
}

The by-date vie is used and there is paging using Skip() and Limit(). The Total property of query result stores number of all rows in queried view. The value of the view is merely a document id, so we also need to download rest of the document. We could query for each picture separately, which isn’t very efficient. Fortunately there is also option to download whole document related to view entry along with that entry by using WithDocuments().

Next action lists pictures under given tag.

/// <summary>
/// Lists pictures uder given tag
/// </summary>
/// <param name="tag">The tag</param>
/// <param name="page">Page number</param>
public ActionResult Tag(string tag, int? page)
{
    if (string.IsNullOrWhiteSpace(tag))
        return RedirectToAction("Index", new { page = 1 });
    
    if (page == null)
        page = 1;
    ViewBag.Page = page.Value;

    var pictureQuery = _sessionManager.GetQueryFor<Picture>("picture", "by-tag-date", false)
        .Skip((page.Value - 1) * PageSize)
        .Limit(PageSize)
        .Descending()
        .From(new string[] { tag, "Z" }).To(new string[] { tag })
        .WithDocuments();
    var result = pictureQuery.Execute();

    var total = LoadTagCounts(tag)[0].Count;

    ViewBag.HasMore = total > (page.Value * PageSize);

    var pictures = result.Rows.Select(r => r.Entity).ToArray();
    var tags = pictures.Where(p => p.Tags != null).SelectMany(p => p.Tags).Distinct().OrderBy(t => t).ToArray();

    var model = new PictureListModel
    {
        Pictures = pictures,
        Tags = LoadTagCounts(tags)

    };

    return View("Index", model);
}

Two interesting things here. First, we can’t use result.Total in order to employ paging since we don’t want number of all entries in the view, but rather the number of pictures with given tag. For this we use LoadTagCounts() helper method. Second thing is the way we constraint the key range loaded from by-tag-date view. Suppose we have tags like “kittens”, “puppies” and “zebras”. Suppose we load all keys from [“puppies”] to [“puppies”, “Z”]. [“puppies”] is after all keys like [“kittens”, date] and before all keys like [“puppies”, date]. [“puppies”, “Z”] is after all keys like [“puppies”, date] (because no date starts with letter “Z”) and just before entries like [“zebras”, “date”]. This may be hard to grasp at first…

Finally, the action to delete a picture. Only the uploader can delete the picture.

/// <summary>
/// Deletes the picture
/// </summary>
/// <param name="id">Picture id</param>
[HttpPost]
[ValidateAntiForgeryToken]
public ActionResult Delete(string id)
{
    try
    {
        var repo = _sessionManager.GetRepositoryFor<Picture>();
        var picture = repo.TryGet(id);
        if (picture == null)
        {
            return new HttpNotFoundResult();
        }

        if (!picture.CreatedBy.Value.Login.Equals(HttpContext.User.Identity.Name, StringComparison.InvariantCultureIgnoreCase))
        {
            return RedirectToAction("Single", new { id = id });
        }

        repo.Delete(picture);
 
        return RedirectToAction("Index");
    }
    catch
    {
        return RedirectToAction("Single", new { id = id });
    }
}

Wrapping it up

There things like views, authentication, routes and setup in Global.asax that I won’t be covering here in the post, but if you’re interested, grab the source.

Few thoughts on CouchDB

2011-10-01T15:50:00.001+02:00

// EDIT: See my post with tutorial of CouchDB + relax-net usage to build simple tagged gallery in ASP.NET MVC

Apache CouchDB is one of the most popular NoSQL document database engines out there. In this post I would like to make a quick comparison to RavenDb as seen with .NET developer eyes.

Architecture

Again, we’re talking about document database, which means, that structure of data stored is very elastic and there is no referential integrity. If you want to keep your data pieces in some particular relation to each other, it’s up to your code to enforce this relations. Yet the data you store is not a blob understandable only to your application, but rather has document semantics, which means you can access particular properties of a document at database level. As with RavenDb, database can’t tell your entities apart, it’s up to client to store entity metadata in some understandable form.

CouchDB exposes REST API and is heavily javascript based. Documents are transferred as JSON objects. All map-reduce queries, validation functions and show functions are written in javascript. Javascript is not my favourite language, but I must say, in case of CouchDB it serves its purpose well. Still, using javascript in your database is something a DBA has to get used to :)

CouchDB doesn’t support transactions. It uses optimistic concurrency model which allows to avoid some cases when your database might enter inconsistent state, but if your code has a lot of places where multiple db operations have to either pass or fail as a group, then working with CouchDB will require a lot of extra work (for example implementing compensation code). This is a sacrifice CouchDB makes to gain performance and scalability.

What’s quite interesting, CouchDB allows you to write so called validation functions (validate data before data operations) and show functions (format a return data in user readable form – ex. HTML). Thanks to that, you could implement whole application using nothing more than a CouchDB itself. I guess that would rarely be the case if you are implementing something more than a simple blog, because you’d want to use a solid framework like ASP.NET MVC or Rails to make things easier.

The database implements some basic authentication mechanisms and you can extend it by implementing more. As for authorization, you can grant a user read-only or full access to your database. If you want something more sophisticated, I guess you have to use validation functions.

Performance

A NoSQL solution like CouchDB is built with performance in mind. Builtin replication mechanism allows to scale the application. What’s more you can create hierarchies of database instances which will operate on subtrees of a indexed structure of your documents and by that you gain extra scalability possibilities. Unfortunately there is no builtin sharding support, but there are external frameworks to support just that (see Gizzard).

Ease of use

Usage of CouchDB is made easier by some tools and projects that make up the Couch's ecosystem. This is a big advantage over RavenDb, even for .NET developers. Couchbase is a free bundle of CouchDB and it’s management interface Futon, that is easy to install and use on a Windows machine. As for .NET clients you have a choice, but in my opinion, relax-net seems to have best design and is easies to use. You won’t get Linq support as with RaveDb, but it’s still bearable.

Documentation

There is documentation on project’s page available, but if you’re new to this, I suggest you start with this free online ebook.

Final thoughts

CouchDB seems to be a solid piece of software used by really big players and what’s more – it’s free. It’s well developed community guarantees that you will get the support and tools you need to build your great apps. Yet, lack of transaction support will make your life harder.

Few thoughts on RavenDb

2011-03-24T20:49:00.001+01:00

Now, that the NoSQL hype in the blogosphere has gone down from ecstatic adoration to constructive criticism, is a good time to get a little more interested in the topic. I decided to look on most popular solutions to know what do they offer. Actually I just peeked at one until now: RavenDb. I started there not because it’s the most popular NoSQL solution. Rather because it’s implemented in and designed for use in .NET and this technology is most familiar to me. And I have some thoughts I’d like to share.

Architecture

RavenDb is a document database. Its underlying data model is plain simple: you have a big bag into which you throw whatever Json documents you like. It’s up to you to differentiate various types of entities you might have. The client library supports different types of entities in a transparent way, but if you would like to directly use REST API, the problem is yours. Is it good or bad? Hard to say, probably depends on usage scenario. Each document is identified by id and Raven is good at quickly retrieving documents by this id. The database supports transactions. It also supports so called attachments which basically are binary pieces of data stored in the database.

RavenDb queries use indexes, which are calculated in the background and not during data modification. More on that later.

RavenDb can be extended by bundles. Some of the bundles shipped with Raven include support for document versioning, authorization, replication, etc. You can also extend it by writing triggers (in .NET) and custom responders.

I can’t find any authentication mechanisms in RaveDb by default. There is authorization bundle, however it relies on trust relationship with application server, that has to be placed between db and the client. So no data consumption from javascript code unless you want to expose everything you have to the internet. Perhaps IIS deployment of the db (which is an optional scenario) resolves this matter by standard IIS authorization mechanisms like basic authentication + SSL. Still, there would be no way to directly associate users with document level permissions.

Performance

There is a great promise of performance and scalability behind NoSQL databases. I didn’t perform any benchmark myself, but it really looks like this promise can be fulfilled by RavenDb. The queries are based on indexes, that do not slow down writing to the db, but rather are calculated in background. Queries allow for stale data so you can you can decide what is more important for you: up to date results or getting results fast. RESTful API allows for employing caching at HTTP protocol level (proxies?). Built-in sharding and replication support deliver scalability. Really impressive.

Ease of use

It’s easy to start using RavenDb. Just add references to client, point it to server instance and you’re good to go. Running the server itself is also as easy as it takes care of creating required data files and configuration automatically. As for the client itself, you’ll find some solutions similar to NHibernate. There’s a session (unit of work) and Linq provider. It seems there’s everything a jolly .NET developer needs…

Yet, the fact that you have to set up indexes upfront to cover all your document searching criteria is not very encouraging. You can define ad-hoc queries, but they are said to be not very effective. As for queries, if you want to construct them dynamically (for example set order by basing on user input), you have two options: either to use Linq and write a switch that will grow with each subsequent ordering criteria, or use Lucene syntax to express the query. Lucene doesn’t look too friendly, but I guess you can learn anything. You can also use something like dynamic Linq.

Reliability and disaster recovery

You can’t really tell how reliable the database is. All I know is that it’s a young project and surely there are bugs to be found in RaveDb. As for disaster recovery, Raven supports replication schemes so that you can perform failover to another node, when on of the nodes is down. There is standard backup / restore mechanism, but I can’t find any information about incremental backups. This aspect is important when you think about data volumes that NoSQL solutions are designed to support. Perhaps in VSS mode there is support for incremental backups.

Documentation

It’s definitely the worst aspect of RavenDb as documentation is chaotic and incomplete. It’s not what you’d expect from commercial solution. I wasn’t even able to run official samples for the db. This is normal for OSS, but in this case – come on!

Final thoughts

I guess RavenDb will be a slick NoSQL solution in the future, but its authors really need to put some work into developer friendliness and documentation.

Event driven javascript components

2011-03-05T21:01:00.001+01:00

Somehow javascript code usually tends to be very messy and mangled. This is especially true when web page is complicated and contains a lot of dependencies between components. I can see a lot of developers forgetting principles of fine software design when switching from C# to javascript. In order to keep js code clean and clear, I always enclose code in objects. A piece of code together with HTML markup make up a control that is maintainable and can be reused. The problem occurs when different components have to interact with each other. Direct method calling between components creates a lot of coupling and reduces ability to reuse code. Next natural thing you would think about is events. Javascript doesn’t support custom events out of the box, but you can simulate them. Still, you have to wire the events together with handlers somehow. What I’m suggesting instead is a centralized event broker which provides complete decoupling of event publishers and event subscribers. The idea is a lot like the event broker in Patterns and Practices’ Composite Application Block. Events are identified by a unique string (topic) and you can attach additional data to the event, which will be passed from publisher to subscriber. This way components are completely decoupled and you can even substitute any side of the event contract as long as event identifier and data format matches.

I’ll be using an example here in which an objects list (a component) is filtered by drop down list of object categories (another component). Let’s see the EventBroker code first:

EventBroker = function() { 
    this._handlers = [];
};

EventBroker.prototype = {
    subscribe : function(eventTopic, handler) {
        if(!this._handlers[eventTopic])
            this._handlers[eventTopic] = [];
        this._handlers[eventTopic].push(handler);
    },
    
    publish : function(eventTopic, eventArgs) {
        if(this._handlers[eventTopic]) {
            $.each(this._handlers[eventTopic], function(idx, handler) {
                if(eventArgs)
                    handler(eventArgs);
                else
                    handler();
            });
        }
    }
};

document.eventBroker = new EventBroker();

The EventBroker instance is registered globally by adding a property to document. Let’s take a look at main HTML file:

<html>
<body>
<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.5.1/jquery.min.js"></script>
<script type="text/javascript" src="eventBroker.js"></script>
<script type="text/javascript" src="foodList.js"></script>
<script type="text/javascript" src="foodFilter.js"></script>
<script type="text/javascript">
    $(document).ready(function() {

        var foodData = [
            ["Fruit", "Orange"],
            ["Fruit", "Apple"],
            ["Fruit", "Banana"],
            ["Vegetables", "Carrot"],
            ["Vegetables", "Potatoes"]
        ];

        var foodList = new FoodList(foodData, "#foodList");
        var foodList2 = new FoodList(foodData, "#foodList2");
        var foodFilter = new FoodFilter("#foodFilter");
    });
</script>

<select id="foodFilter">
    <option>Fruit</option>
    <option>Vegetables</option>
</select>
<ul id="foodList"></ul>
<ul id="foodList2"></ul>

</body>
</html>

I created two object lists in order to show how event publishing is independent from its consumption. Next let’s look at the FoodFilter class:

FoodFilter = function(filterSelector) {
    $(filterSelector).change(this.onFilterChange);
};

FoodFilter.prototype = {
    onFilterChange : function(e) {
        var filterValue = $(e.target).val();
        document.eventBroker.publish("event://filterFood", filterValue);
    }
};

When dropdown selection changes, the “event://filterFood” event is published. Selected category is passed along with the event. And last but not least – the consumer:

FoodList = function(foodData, listSelector) {
    this._foodData = foodData;
    this._listSelector = listSelector;
    document.eventBroker.subscribe("event://filterFood", $.proxy(this.onFilter, this));
};

FoodList.prototype = {
    onFilter : function(filter) {
        var list = $(this._listSelector);
        list.empty();
        jQuery.each(this._foodData, function(idx, entry) {
            if(filter == entry[0])
                $("<li />").text(entry[1]).appendTo(list);
        });
    }
};

During the subscription, a proxy is used to change call context to the instance of FoodList class. Filter value is passed as onFilter handler parameter.

This code works fine, but if you’re a challenger, you would be interested in a different approach using a feature of HTML 5. There’s a way to pass a message to javascript code originating from different domain and residing within an iframe (for example facebook widgets) using the window.postMessage() method. The main use of this mechanism is cross domain communication. But we can used use it as well to substitute the EventBroker class. Event publishing looks like this:

window.postMessage('{ "topic" : "event://filterFood", "args" : "' + filterValue + '" }', "*");

Unfortunately postMessage only accepts messages to pass in form of string. In order to pass a complex type, you have to create it’s a JSON representation and then parse it at subscriber. The message contains both event topic and event argument. And now the FoodList class adapted to the new approach:

FoodList = function(foodData, listSelector) {
    this._self = this;
    this._foodData = foodData;
    this._listSelector = listSelector;
    window.addEventListener("message", $.proxy(this.onFilter, this), false);
};

FoodList.prototype = {
    onFilter : function(e) {
        var parsedData = $.parseJSON(e.data)
        if(parsedData.topic != "event://filterFood")
            return;
        
        var list = $(this._listSelector);
        list.empty();
        jQuery.each(this._foodData, function(idx, entry) {
            if(parsedData.args == entry[0])
                $("<li />").text(entry[1]).appendTo(list);
        });
    }
};

Of course the approach choice is up to you. You can download the code here.

Open/closed principle

2010-04-04T18:05:00.002+02:00

It’s one of object-oriented design pillars. According to wikipedia definition, class following this principle is open to extension, but closed to modifications. I think it might be a little vague for beginning developers, so I present a little example. Let’s say we have a small application for processing wine orders. Here you can see a fragment of code that might be part of this application:

public class ShipmentInfo
{
    public DateTime EstimatedArrivalDate { get; set; }

    public decimal ShipmentCost { get; set; }
}

public class WineOrder
{
    public int NumberOfBottles { get; set; }

    public decimal BottlePrice { get; set; }

    public decimal OrderValue
    {
        get
        {
            return NumberOfBottles*BottlePrice;
        }
    }

    public ShipmentInfo GetShipmentInfo()
    {
        return new ShipmentInfo()
                   {
                       EstimatedArrivalDate = DateTime.Today.AddDays(3),
                       ShipmentCost = (NumberOfBottles/10 + 1)*16.0M
                   };
    }
}

I try to keep things really simple: this class can calculate order’s value and generate shipment info. The application is used on local market. In first version of the application we only use one way of generating shipment info: our transport service provider charges 16 ZŁ (Polish currency) per each 10 bottles of wine being transported.

Everything’s fine until there’s need to expand to international market. Transport to foreign countries is charged differently. We will now try to add a functionality to our application, that will handle this situation. Seemingly the easiest way to do this is to put both ways of generating shipment info to our WineOrder class:

public class WineOrder
{
    public int NumberOfBottles { get; set; }

    public decimal BottlePrice { get; set; }

    public string ShipmentCountry { get; set; }

    public decimal OrderValue
    {
        get
        {
            return NumberOfBottles * BottlePrice;
        }
    }

    public ShipmentInfo GetShipmentInfo()
    {
        if (ShipmentCountry == "Poland")
        {
            return new ShipmentInfo()
                       {
                           EstimatedArrivalDate = DateTime.Today.AddDays(3),
                           ShipmentCost = (NumberOfBottles/10 + 1)*16.0M
                       };
        }
        else
        {
            return new ShipmentInfo()
                       {
                           EstimatedArrivalDate = DateTime.Today.AddDays(7),
                           ShipmentCost = (NumberOfBottles/5 + 1)*12.0M
                       };
        }
    }
}

As you can see I added destination country to WineOrder class and then put a conditional logic inside GetShipmentInfo method. Fine, we’re done. What if the transport pricing will change so that it will depend on continent to which wine is shipped? No problem, we’ll just add more if-elses…

Is this really object-oriented programming we’re doing here? This obviously violates the open/closed principle. Few more changes in business rules and our GetShipmentInfo method has a few hundred lines of code. This level of complexity will surely lead to bugs and code maintenance will be a pain. While adding new code to WineOrder, there is a chance of breaking other things, that now work. Also shipment info generating logic testing is trickier, because you have to set WineOrder’s state so that right algorithm is picked. Last, but not least – we’re breaking the SRP. Let us do some refactoring and introduce strategy pattern in place of conditional logic:

public interface IShipmentCalculator
{
    ShipmentInfo GetShipmentInfo(int numberOfBottles);
}

public class LocalShipmentCalculator : IShipmentCalculator
{
    public ShipmentInfo GetShipmentInfo(int numberOfBottles)
    {
        return new ShipmentInfo()
        {
            EstimatedArrivalDate = DateTime.Today.AddDays(3),
            ShipmentCost = (numberOfBottles / 10 + 1) * 16.0M
        };
    }
}

public class InternationalShipmentCalculator : IShipmentCalculator
{
    public ShipmentInfo GetShipmentInfo(int numberOfBottles)
    {
        return new ShipmentInfo()
        {
            EstimatedArrivalDate = DateTime.Today.AddDays(7),
            ShipmentCost = (numberOfBottles / 5 + 1) * 12.0M
        };
    }
}

public class WineOrder
{
    public int NumberOfBottles { get; set; }

    public decimal BottlePrice { get; set; }

    public decimal OrderValue
    {
        get
        {
            return NumberOfBottles * BottlePrice;
        }
    }

    private readonly IShipmentCalculator _shipmentCalculator;

    public WineOrder(IShipmentCalculator shipmentCalculator)
    {
        _shipmentCalculator = shipmentCalculator;
    }

    public ShipmentInfo GetShipmentInfo()
    {
        return _shipmentCalculator.GetShipmentInfo(NumberOfBottles);
    }
}

Now this looks much better. What have we achieved?

No need to modify WineOrder when shipment info generating rules change. No proven code will be broken. Class is closed to modification.
Algorithm for generating shipment info can be chosen elsewhere, possibly in configuration. Class is open to extension.
Less dependencies, SRP compatible.

Handling stylesheet references in ASP.NET MVC

2010-01-31T19:40:00.001+01:00

You can download code discussed in this post here.

Linking CSS to a web page seems to be a trivial task. But when you’re just plainly using LINK tag to attach a stylesheet you have to be aware of at least three things:

LINK tag is only valid within the HEAD section. This breaks encapsulation of partial views, since they cannot include stylesheets they’re using on their own, but instead they depend on master site or aspx page to include those stylesheets.
If you have several stylesheets, each of them will be downloaded from server separately, which is considered to have significant impact on page’s loading time, because a web browser can request only a limited number of resources from a server at a time.
If a partial view is rendered multiple times, you would like to have only one reference to stylesheet.

My goal here is to provide a mechanism to overcome those problems. Note that while I’m talking about CSS here, similar rules apply to scripts used on the site. Scripts could be included anywhere in the page, but their number is usually greater than number of stylesheets and also they’re significantly larger. I’ll be showing how to manage CSS to keep things simple, but with a little work you can do the same fro scripts. For simplicity’s sake I also use things like HttpContext directly, but you would want them wrapped in something that’s actually mockable (like HttpContextWrapper from System.Web.Abstractions assembly).

The idea is to gather all required stylesheet references and then to place them in HEAD section. Also we want to download all stylesheets at once, so we’re going to create a HttpHandler, that will take a list of files as a parameter and render their concatenated contents into the response. In order to gather required stylesheet references from any views rendered in current response and then to include them in HEAD section, there must be some kind of response postprocessing taking place. This is because you can’t modify header from within view or partial view, since it has already been rendered. We can do this by placing a placeholder for stylesheet reference inside HEAD section, and then replacing it with actual reference after all views have been rendered.

Let’s start coding:

public class StylesheetManager
{
    private List<string> _stylesheets = new List<string>();

    public static StylesheetManager Current
    {
        get
        {
            var manager = HttpContext.Current.Items["STYLESHEET_MANAGER"] as StylesheetManager;
            if(manager == null)
            {
                manager = new StylesheetManager();
                HttpContext.Current.Items.Add("STYLESHEET_MANAGER", manager);
            }
            return manager;
        }
    }

    public void RegisterStylesheet(string file)
    {
        file = file.ToLower();
        if(!_stylesheets.Contains(file))
            _stylesheets.Add(file);
    }

    public string GetStylesheetsReference()
    {
        var filesParam = HttpUtility.UrlEncode(string.Join(",", _stylesheets.ToArray()));
        var dateParam = HttpUtility.UrlEncode(GetLatestModificationDate(_stylesheets).ToString("s"));

        return string.Format("<link href=\"Css.axd?f={0}&d={1}\" rel=\"stylesheet\" type=\"text/css\" />",
                             filesParam, dateParam);
    }

    private DateTime GetLatestModificationDate(List<string> files)
    {
        if (files.Count == 0)
            return DateTime.MinValue;

        var latest = DateTime.MinValue;

        foreach (var file in files)
        {
            var fileDate = File.GetLastWriteTime(HttpContext.Current.Request.MapPath(file));
            if (fileDate > latest)
                latest = fileDate;
        }
        return latest;
    }
}

This class manages references to stylesheet files and is used by view and partial views. Each reference is registered only once. As you may have noticed the URL to stylesheets references the Css.axd resource. This will be our HttpHandler, that receives file list as a parameter. There is also another parameter – date of latest modification among registered files. This parameter is included due to caching reasons, which I’ll explain later. An instance of StylesheetManager is bound to current HttpContext.

Sample usage of StylesheetManager class is:

    <% 
        StylesheetManager.Current.RegisterStylesheet("~/Content/Site.css"); 
        StylesheetManager.Current.RegisterStylesheet("~/Content/Skin.css");
    %>

Next step is to define the HttpHandler to serve requests to Css.axd resource:

public class StylesheetHttpHandler : IHttpHandler
{
    public void ProcessRequest(HttpContext context)
    {
        var fileParam = context.Request.QueryString["f"];
        if(string.IsNullOrEmpty(fileParam))
            return;
        
        string[] files = HttpUtility.UrlDecode(fileParam).Split(',');
        if(files.Any(f => !VirtualPathUtility.IsAppRelative(f) || Path.GetExtension(f) != ".css"))
        {
            // only CSS files within application are allowed
            context.Response.StatusCode = 403; // forbidden
            return;
        }

        var responseBuilder = new StringBuilder();
        foreach (var file in files)
        {
            var content = File.ReadAllText(context.Request.MapPath(file));
            responseBuilder.Append(content);
        }

        context.Response.ContentType = "text/css";
        context.Response.Cache.SetCacheability(HttpCacheability.Public);
        context.Response.Cache.SetExpires(DateTime.Now.AddYears(1));
        context.Response.Write(responseBuilder.ToString());
    }

    public bool IsReusable
    {
        get { return true; }
    }
}

One interesting bit here is restriction to files with .css extension within application directory. We wouldn’t like anybody to download our web.config with connection strings, right? The other is setting cache control HTTP headers. We set the expiration date to one year ahead, so under normal circumstances user’s browser won’t have to request the stylesheet again after downloading it at first request.

But what if we want to change the script? How can we force user’s browser to download new version? That’s exactly what the date parameter in stylesheet URL is for. When we change the file, the parameter changes. Browser treats this as a new resource and it has to download it form server.

Of course the handler has to be configured in web.config:

<httpHandlers>
      <add verb="GET,HEAD" path="Css.axd" type="CssHandlingSample.Utilities.StylesheetHttpHandler, CssHandlingSample"/>
</httpHandlers>

One missing piece here is the postprocessing of the response. This is pretty easy to do by defining our own stream class. The most interesting is the Write method:

public override void Write(byte[] buffer, int offset, int count)
{
    if (HttpContext.Current.Response.ContentType != "text/html")
    {
        // when content type other than text/html, just forward content to 
        // wrapped stream
        _wrappedStream.Write(buffer, offset, count);
        return;
    }

    var page = new StringBuilder(Encoding.UTF8.GetString(buffer, offset, count));
    page.Replace("#STYLESHEET_REFERENCE#", StylesheetManager.Current.GetStylesheetsReference());

    var transformedPageBytes = Encoding.UTF8.GetBytes(page.ToString());
    _wrappedStream.Write(transformedPageBytes, 0, transformedPageBytes.Length);
}

Now we have to hook it into response processing. To do this, we have to define application’s BeginRequest handler:

protected void Application_BeginRequest()
{
    HttpContext.Current.Response.Filter =
        new PostprocessingStream(HttpContext.Current.Response.Filter);
}

Almost done! Last thing to do is to put the placeholder inside the page’s header:

<head runat="server">
    <title><asp:ContentPlaceHolder ID="TitleContent" runat="server" /></title>
    #STYLESHEET_REFERENCE#
</head>

After running the application, this is what we get in page’s source:

<head>
<title>Home Page</title>
<link href="Css.axd?f=%7e%2fcontent%2fsite.css%2c%7e%2fcontent%2fskin.css&d=2010-01-30T19%3a15%3a57" rel="stylesheet" type="text/css" />
</head>

Great!

Now, before putting this into production, there are some things to consider. Firstly you would probably provide some kind of server side caching of files’ modification dates and content. This way server won’t have to hit the disk at each new request. Secondly, be careful with client side caching. URL to the downloaded resource contains list of files being downloaded. If that list changes, the URL changes. That means ALL files are downloaded again as a new resource. Suppose you’re using this mechanism for scripts. You attach jQuery, MS AJAX and a custom script for currently displayed view. Those three scripts are concatenated, downloaded and stored in cache. Then browser is redirected to another view. Again you’re using jQuery, MS AJAX and a different custom script. File list changed. jQuery and MS AJAX are downloaded again, when they could be served from browser’s cache. To avoid such situation, in my production environment I decided to introduce resource categories. There are “Common” scripts and “View specific” scripts. There is one URL created for each category. This way common scripts never have to be downloaded again from server when another view is displayed.

To make this mechanism better, you could employ some script and stylesheet compressing methods like removing whitespace before sending response to the browser.

There is also an issue with script debugging. When script is downloaded as one resource, it is then hard to navigate in tools like FireBug and also information about line numbers is lost. To avoid this, I introduced a configuration switch that makes StylesheetManager class produce one resource reference per each registered file. I can turn this setting on when I need to debug scripts.

You can download code discussed in this post here.

Proxying a WCF client channel

2009-12-19T21:36:00.001+01:00

Recently I wanted to apply AOP when using a client channel to WCF service. I wanted to do this in order to manipulate the message being sent to the service, which involved attaching some additional information to the message. Creating a wrapper for this channel by hand by implementing channel’s interface (contract) would mean that I have to update this wrapper each time the services contract changes. This happens all too often during the development, so this solution was unacceptable. Hence the AOP. Unfortunately PostSharp doesn’t quite seem to support a scenario where you have to intercept calls to methods of existing instance returned by third party library. Especially because client channel returned by ChannelFactory<T> class is a remoting proxy created on the fly.

Using a dynamic proxy seemed right way to go, so I decided to use well known Castle.DynamicProxy library. Implementation turned out to be quite easy:

var proxyGenerator = new ProxyGenerator();

service = ChannelFactory<IInvoicingService>.CreateChannel(
    new NetNamedPipeBinding(), new EndpointAddress("net.pipe://something"));

service = proxyGenerator.CreateInterfaceProxyWithTarget(typeof(IInvoicingService),
    service, new MyInterceptor()) as IInvoicingService;

The channel is being proxied and the interceptor definition is as follows:

public class MyInterceptor : IInterceptor
{
    #region IInterceptor Members

    public void Intercept(IInvocation invocation)
    {
        // do something with message, it's in invocation.Arguments
        
        invocation.Proceed();
        
    }

    #endregion
}

This works fine until you realize, that the client channel not only implements the service contract, but it also implements ICommunicationObject interface used to perform communication related operations like closing the channel. Normally you would cast the channel to ICommunicationObject and use appropriate methods. But now that channel is proxied, it no longer implements ICommunicationObject interface. Fortunately Castle.DynamicProxy provides a way to deal with such situations. When creating a proxy, you can specify what additional interfaces should be implemented along with the main proxied interface:

service = proxyGenerator.CreateInterfaceProxyWithTarget(typeof(IInvoicingService),
    new Type[] { typeof(ICommunicationObject) }, service, 
    new MyInterceptor()) as IInvoicingService;

But this is not all, because proxy won’t forward calls to methods defined on ICommunicationObject interface by default, since it considers those methods to have no target. It means that interceptor is solely responsible for acting as a method body for those calls and any attempt to call invocation.Proceed() will result in an exception. Fortunately I can forward those calls by hand:

public class MyInterceptor : IInterceptor
{
    #region IInterceptor Members

    public void Intercept(IInvocation invocation)
    {
        if (invocation.Method.DeclaringType == typeof(ICommunicationObject))
        {
            var target = (invocation.Proxy as IProxyTargetAccessor).DynProxyGetTarget();
            var returnValue = invocation.Method.Invoke(target, invocation.Arguments);
            invocation.ReturnValue = returnValue;
        }
        else
        {
            // do something with message, it's in invocation.Arguments

            invocation.Proceed();
        }

    }

    #endregion
}

So this is a nice and clean way to apply AOP to WCF client channel and I have to say I impressed with how easy to use and powerful the Castle.DynamicProxy library is.

Unity auto registration

2009-11-03T23:06:00.001+01:00

Auto registration is a great feature for any Dependency Injection container. Suppose you decided to implement the Domain Event pattern. So you have a bunch of events and some event handlers. These event handlers are classes implementing some common interface, say IHandlerFor<TEvent>. You also have a central registry of these handlers, so that you can easily find handlers for particular event – and run them passing the event data. But how does the registry know about these handlers? Well, you probably have to register all of them in the registry with some kind of configuration code. If your registry is built upon your DI container you can take advantage of auto registration feature. That is if your container supports it like Castle does.

Unfortunately Unity doesn’t do that out of the box, but there is a nice library by Artem Govorov on Codeplex, that will let you do just that. It has a nice syntax and offers a great deal of flexibility. Basically in order to achieve auto registration you:

Select which assemblies are scanned for types to register
Specify which types are to be registered by providing conventions
Specify how the types should be registered in the container (select an interface for type, select a name under which it is to be registered)

Of course there are some predefined predicates and other methods that make working with the library easier. For more details, please visit the project’s page.

What I found the library lacked is a predicate that would allow to select types, which implement a generic interface that is constructed from specified open-generic interface (generic type definition).To use an example of domain event: If I have an event called OrderAdded and a handler implementing IHandlerFor<OrderAdded>, the open-generic interface is IHandlerFor<>. By specifying this open-generic interface I can register all handlers at once and later access concrete generics when needed. Of course you can write the predicate yourself, but it would be nice to have it out of the box.

So I introduced this predicate and Artem was kind enough to apply this patch to the library’s codebase. Now having this:

public class OrderAdded { }

public class NotifySalesHandler : IHandlerFor<OrderAdded> { }

public class SendConfirmationHandler : IHandlerFor<OrderAdded> { }

I can do this:

_container
    .ConfigureAutoRegistration()
    .IncludeAssemblies(_testAssemblies)
    .Include(type => type.ImplementsOpenGeneric(typeof(IHandlerFor<>)), 
        Then.Register().AsFirstInterfaceOfType().WithTypeName())
    .ApplyAutoRegistration();

var orderAddedHandlers = _container.Resolve<IHandlerFor<OrderAdded>>();

Of course with no additional setup required I am able to resolve any other hadlers for whatever events I might have.

Typed Expand() for Linq to NHiberante

2009-10-13T18:28:00.001+02:00

// EDIT: as of NHibernate 3.0, there is new built-in LINQ provider with typed Fetch method (I don’t like it though). This post was intended for NHibernate 2.x with external LINQ provider.

( You can get code discussed in this post here )

One of things that keep me irritated in Linq to NHibernate is the way you specify eager fetching paths. Suppose you have domain model like this:

Normally you do things like this:

using (var session = sessionFactory.OpenSession())
{
    var invoices = session.Linq<Invoice>().Expand("Customer").ToList();
}

I don’t like using strings to identify type members since this way they’re not impacted by refactoring changes using automatic tools. What’s more, Expand returns IQueryable<T> instead of INHibernateQueryable<T>, which means you can’t chain multiple paths and have to use some temporary variable.

So to address those issues it would be nice to have ability to specify eager fetching path using labmda expressions and to chain them. For this purpose I created a little extension to INHibernateQueryable<T>, which allows you to do this:

[Test]
public void CanExpandSimpleProperty()
{
    var nhQueryable = 
        MockRepository.GenerateStub<INHibernateQueryable<Invoice>>();
    nhQueryable.Expand(i => i.Customer);

    nhQueryable.AssertWasCalled(nhq => nhq.Expand("Customer"));
}

[Test]
public void CanExpandNestedProperties()
{
    var nhQueryable =
        MockRepository.GenerateStub<INHibernateQueryable<Invoice>>();
    nhQueryable.Expand(i => i.Customer.CustomerCategory);

    nhQueryable.AssertWasCalled(nhq => nhq.Expand("Customer.CustomerCategory"));
}

This is pretty easy to do. But there is another scenario that complicates things a little: what if you would like to fetch invoices along with invoice items and with products. Since Items is a collection of InvoiceItem object, so there is no way of directly specifying a path like: Invoice –> Items –> Product. If Items is an IList or IList<T>, the extension allows you to do this:

[Test]
public void CanExpandCollectionTraversal()
{
    var nhQueryable =
        MockRepository.GenerateStub<INHibernateQueryable<Invoice>>();
    nhQueryable.Expand(i => i.Items[0].Product);

    nhQueryable.AssertWasCalled(nhq => nhq.Expand("Items.Product"));
}

So you can traverse collections using the indexer. But what if you have a collection like ISet, which doesn’t have an indexer? For this scenario, extension provides notion of subpaths:

[Test]
public void CanExpandWithSubpath()
{
    var nhQueryable =
        MockRepository.GenerateStub<INHibernateQueryable<Invoice>>();
    nhQueryable.Expand(i => i.Items,
        Subpath.For<InvoiceItem>().Add(item => item.Product));

    nhQueryable.AssertWasCalled(nhq => nhq.Expand("Items.Product"));
}

You can also chain multiple Add calls in order to specify more than one subpath rooting from type provided in For<T> method.

You can get this extension here. Hope this helps!

Linq to NHibernate First() bug

2009-10-10T21:53:00.001+02:00

As I suspected Linq to NHibernate has some flaws. I just discovered a bug showing up when eagerly loading collections and using IQueryable<T>.First(). Suppose you’ve got persistent class like this:

public class Invoice 
{
    public virtual int Id { get; set; }

    public virtual IList<InvoiceItem> Items { get; set; }

}

Normally, if you want to get and instance by id with eagerly loaded collection of items, you’d write something like this:

using (var session = sessionFactory.OpenSession())
{
    var nhQueryable = session.Linq<Invoice>();
    nhQueryable.Expand("Items");
    var invoice = nhQueryable.Where(i => i.Id == id).First();
}

This code executes without any errors, but unfortunately you always get only one item in Items collection. A quick investigation there and it turns out there is a problem with SQL generated by this query. NHibernate performs eager fetching by means of outer joins. This means that SQL query would return as many rows as there are invoice items associated with this invoice. But usage of the First() method results in the SQL query having TOP(1) clause, which causes database to return only one invoice item. It seems to be a bug in Linq to NH because according to information found on the web, such queries expressed with Criteria API work just fine. Well, you can either not use Linq to NH or just write this:

var invoice = nhQueryable.Where(i => i.Id == id).ToArray()[0];

When in doubt, write some code

2009-09-27T20:26:00.001+02:00

Well, that was some time since I last posted. I had some private matters going on, but hopefully I will be able to make up for that time.

Currently I’m starting a new project – internet facing web application. Carefully considering technologies that I would use I came up with this:

Web layer:

ASP.NET MVC
jQuery

Application layer:

WCF for exposing services
NHibernate with additional help of Fluent NHiberante and Linq for NHibernate

Common:

PostSharp for AOP
Unity for DI

However I still have some doubts related to those choices. ASP.NET MVC while being seemingly perfect for internet facing application, is still quite new on the market and not really proven. Because I don’t have any experience with this technology, I don’t know what can be expected. As for NHibernate I have some previous experience and I generally know where the traps are set for me. But Fluent NHibernate and Linq for NHibernate are also new for me. Especially Linq for NH, despite being version 1.0, seems immature. There are some bugs open for it, which seem to be quite serious. I’m not really sure if it will be Linq or if I will just stick with Criteria API, which supports a very interesting feature of futures since NH 2.1.

With all those doubts I would be crazy just jumping into full scale development. In my opinion the best way to verify architectural assumptions is to write some code. So I intend to create a prototype of the application, which will reflect structure of the final solution, but with some simplifications, that would allow to achieve some basic functionality in short time. After this is done, I hope to be able to assess my assumptions and – if something is wrong – to correct this, before changes in architecture start to cost some real money.

Building a fluent interface

2009-06-20T23:04:00.001+02:00

Who doesn’t like fluent interfaces? They allow to write things down in more compact manner and keep them readable in the future. Almost all contemporary libraries have fluent APIs. Some of them don’t (see LLBLGen Pro for example, argh!), but they definitely should get one.

In this post I’d like to show how to build a really simple fluent interface, and then – how to make it a little better. In the example, we’ll be writing a simple interface for creating and running SQL queries. What we want to achieve is ability to write code like this:

var query = new QueryFactory().Create();

var result = query
    .From("Employee").Alias("e")
    .Join("Order").Alias("o").On("e.EmployeeID = o.EmployeeID")
    .Where("o.TotalPrice > 1000")
    .Execute<Employee>();

As you can see, fluency is gained by chaining method calls on an object. The QueryFactory class is our entry point and returns IQueryBuilder interface, which then serves as our fluent API for building the query. The IQueryBuilder interface is defined as follows:

public interface IQueryBuilder
{
    IQueryBuilder From(string tableName);
    IQueryBuilder Join(string tableName);
    IQueryBuilder Alias(string alias);
    IQueryBuilder On(string joinCondition);
    IQueryBuilder Where(string filter);

    IList<T> Execute<T>();
}

Every method returns IQueryBuilder interface, which allows to chain calls. A sample implementation could look like this:

public class Query : IQueryBuilder
{
    private class TableOptions
    {
        public string TableName { get; set; }
        public string Alias { get; set; }
        public string JoinCondition { get; set; }
    }

    private List<TableOptions> tableOptionsList = new List<TableOptions>();
    private TableOptions currentTableOptions;
    private string filter;

    #region IQueryBuilder Members

    public IQueryBuilder From(string tableName)
    {
        currentTableOptions = new TableOptions() { TableName = tableName };
        tableOptionsList.Add(currentTableOptions);

        return this;
    }

    public IQueryBuilder Join(string tableName)
    {
        currentTableOptions = new TableOptions() { TableName = tableName };
        tableOptionsList.Add(currentTableOptions);

        return this;
    }

    public IQueryBuilder Alias(string alias)
    {
        if (currentTableOptions == null)
            throw new InvalidOperationException("First specify a table");

        currentTableOptions.Alias = alias;
        
        return this;
    }

    public IQueryBuilder On(string joinCondition)
    {
        if (currentTableOptions == null)
            throw new InvalidOperationException("First specify a table");

        currentTableOptions.JoinCondition = joinCondition;
        
        return this;
    }

    public IQueryBuilder Where(string filter)
    {
        this.filter = filter;

        return this;
    }

    public IList<T> Execute<T>()
    {
        tableOptionsList
            .ForEach(options => Console.WriteLine(
                "From table: {0}, alias: {1}, join on: {2}", 
                options.TableName, options.Alias, options.JoinCondition));
        Console.WriteLine("Where: {0}", filter);    
        
        // fake result
        return new List<T>();
    }

    #endregion
}

The Query class always returns itself in each method, so that subsequent calls always refer to the same instance. This way information about query is collected step by step.

There are however some problems with this code. Some parts of the query are mandatory, while other are optional. Some can appear only once. Some have to be provided in specific sequence in order to make sense. You can always validate conditions in code and throw exceptions if user of your library does something wrong, but it’s always better to guide user and detect errors at compile time. Also, we should make our interface more intuitive. Suppose you don’t know the structure of SQL query, but want to use the library. What do you do if Intellisense shows you this?

Well, you refer to the documentation. But who likes to do this? Wouldn’t it be better if the library took you by the hand? We are going to improve our interface a little by ensuring following things:

From() is the only mandatory call here, so we’re not letting user do anything until this method is called
When Join() is called, only available options should be Alias() (to assign table an alias) and On() (to specify the join condition). After calling On() we want to return to regular interface.

In order to do this, we should define more than one interface for the Query class.

public interface IQueryBuilderMandatory
{
    IQueryBuilder From(string tableName);
}

public interface IQueryBuilder
{
    IQueryJoinBuilder Join(string tableName);
    IQueryBuilder Alias(string alias);
    IQueryBuilder Where(string filter);

    IList<T> Execute<T>();
}

public interface IQueryJoinBuilder
{
    IQueryJoinBuilder Alias(string alias);
    IQueryBuilder On(string joinCondition);
}

Create() method of the QueryFactory class now returns IQueryBuilderMandatory interface instead of IQueryBuilder. The Query class implements all three of the interfaces:

public class Query : IQueryBuilderMandatory, IQueryBuilder, IQueryJoinBuilder
{
    ...
}

Implementation also changes slightly. The Join() method now returns IQueryJoinBuilder interface (body stays the same).

public IQueryJoinBuilder Join(string tableName)
{
    ...
}

There also have to be two explicit implementations of the Alias() method (since it is defined in two interfaces with the same parameters, but different return type). Their body is the same as body of the Alias() method in first version of the class – or it can be extracted to private method of course.

IQueryBuilder IQueryBuilder.Alias(string alias)
{
    ...
}

IQueryJoinBuilder IQueryJoinBuilder.Alias(string alias)
{
    ...
}

And that’s it. Now when user opens Intellisense at the begining of query definition, he sees only the From() method:

After calling From(), more options become available:

And when defining a joined table, only relevant methods are available:

This still is far from perfect. There are possible some combinations, which are not correct syntax or make no sense. But it serves as a good example. Where you can take it is up to you.

Code contracts in .NET 4.0

2009-05-26T20:29:00.001+02:00

Code contracts are a nice feature, that will be included in .NET 4.0’s BCL. They allow you to express assertions on method parameters and object state in explicit manner. Surely everyone has already written his Guard class to achieve Design by Contract approach, but .NET’s code contracts work a little different. Look at this:

class Account
{
    private decimal balance;

    public Account(decimal balance)
    {
        this.balance = balance;
    }

    public void Debit(decimal amount)
    {
        Contract.Requires(10 < amount, 
            "Amount cannot be less than 10");
        Contract.Requires(amount <= this.balance, 
            "The amount to debit exceeds account balance");
        Contract.Ensures(this.balance == Contract.OldValue(this.balance) - amount,
            "Operation has resulted in incorrect final balance");

        this.balance -= amount;
    }

}

class Program
{
    static void Main(string[] args)
    {
        Account ac1 = new Account(100);
        ac1.Debit(5);
    }
}

There are at least two interesting things about this piece of code. First, note that expectations of exit condition are expressed before the value even gets calculated. Secondly, you can use variable’s old value (that is a value it had just before method invocation) in exit condition in order to compare it with this variable’s value after method has been executed. This is possible thanks to the way code contracts work. After compilation, assembly is processed and rewritten, so that assertions can be expressed in IL. This works in similar way as PostSharp. There are also other “special” values, that can be used in conditions like method’s return value or value of output parameter. Execution of the above code snippet results with this error notification:

Note, that you can actually see the condition in the message. You can also state conditions, that should always be true for object’s state:

[ContractInvariantMethod]
protected void ObjectInvariant()
{
    Contract.Invariant(this.balance >= 0);
}

There are also means to express contracts over interfaces and abstract methods. Contracts are inherited, but cannot be modified in subtypes (Liskov Substitution Principle).

So that was runtime checking, but there’s also another mode – static checking, which occurs during compilation. Warnings about possible assertion violations are displayed in Error List window. Unfortunately I was unable to get any satisfying results with the code snippet above. All I got was information, that my assertions are “unproven”:

This is not really helpful ;) I checked tutorial videos and forums, but for other people this appears to work just fine. Strange I say. Documentation only says:

First, a word of caution: Static code checking or verification is a difficult endeavor. It requires a relatively large effort in terms of writing contracts, determining why a particular property cannot be proven, and finding a way to help the checker see the light.
Before you start using the static checker in earnest, we suggest you spend enough time using contracts for runtime checking to familiarize yourself with contracts and the benefits they bring in that domain.

I don’t think this feature is going to be widely used if it will require special considerations from it’s users. Anyway, this feature seems to be only available for users of VSTS, which is a big disadvantage.

Why does everything have to be so hard?

2009-05-21T23:45:00.001+02:00

All I wanted was to get a really simple functionality of uploading quite large files (~100 MB) from mobile device to a server. Seems easy to achieve, right? Well it’s not.

My original idea was to create a web service, that would orchestrate the upload process. A client would request upload of one or more files. File information would be recorded in database, then the web service would return URLs for the client to perform HTTP PUT requests. After finishing upload of each file, the client would notify the web service. It is important that the upload has to be streamed, because buffering whole file in the memory on either client or web server would cause extreme performance penalty. Also, a mechanism for upload resuming should be provided, but this seemed easy to do with Content-Range header. I assumed I can use standard IIS mechanisms to achieve this and that’s where I was wrong.

Firstly, in order to get HTT PUT on IIS, the WebDAV feature has to be installed. And it turns out that the standard implementation provided by Microsoft doesn’t support Content-Range header (attaching this header resulted in HTTP 50* status code). There are other implementations, but since this application was a small internal project, it had no budget to buy one. What’s more, PUT verb always requires authentication with this implementation of WebDAV and that prevents streaming. Why? .NET’s HttpWebRequest needs to buffer whole request in order to perform authentication and setting AllowWriteStreamBuffering to false causes exception to be thrown. Well if you only need to perform Basic HTTP authentication this problem can be worked around by manually attaching authentication header to the request, but still!

Ok, so if not that way, maybe another then - FTP. Oh yeah, I just forgot that .NET CF doesn’t have classes for FTP support. Darn. There are of course commercial libs, but as I said earlier, I could not buy them. Maybe I should put out a web page with upload control and POST files to the server? Oops – not resumable.

Well why not implement my own http handler to handle the PUT requests? Yes, idea seemed brilliant, but unfortunately it turns out, that there’s no easy way to stream request from client. The request gets buffered in the memory and the http handler is called when whole request is completed. This was unacceptable. After a lot of digging I found this post, which describes how to work around that problem, but hey – there’s a code that sets private fields of ASP.NET classes using reflection! This definitely isn’t a supported scenario. I decided, that I can’t accept this solution. I mean, shouldn’t request streaming be an easy to perform task? Isn’t that quite common thing to do?

I ended up with solution, that uploads files by sequential web service requests, chunk by chunk. This is quite ugly and requires client to transfer around 33% percent more data than other solutions – due to Base64 encoding of binary data in SOAP messages. But at least it works!

So what is the point of me ranting about this? There’s none really ;) Sometimes I just want to scream: “Why does everything have to be so hard?!”

The Geneva framework

2009-05-14T20:15:00.001+02:00

The “Geneva”, while being a very interesting framework developed by Microsoft, isn’t getting much buzz on the blogs. A beta 2 version was just recently released and the final version is planned somewhere in 2009.

The framework aims at providing an easy to use solution for federated authentication and claims based security. None of those concepts are new in IT world. There are standards from WS-* family, which encompass both. In fact, WCF provides both claims based security plumbing code and bindings compatible with WS-Federation standard. The problem was, that those concepts were hard to implement in applications and required a lot of effort from developers. There also were no easy means of implementing own STS (Security Token Service). This changes with “Geneva”, which tries to make usage of federation and claims based security as easy as possible. As for claims, a developer can access them using IClaimsPrincipal interface (which derives from IPrincipal) – this integrates very well with security mechanisms built into .NET. There are even ASP.NET controls provided making implementation of ASP.NET based Relaying Parties and Security Token Services a lot easier. But Geneva is not only a library to use in your code. It also consists of two more components – the Geneva Server, which is a STS (with identity database and GUI for administration) as well as extensions to Windows CardSpace. I’m not going to go into details with this – there are very good whitepapers on this topic and I’d hate to just copy them here. Instead I would like to write about two usage cases, that got me most interested.

Authentication Assurance

Not directly related to federation, “Authentication assurance” sample from Geneva SDK presents a case, where different parts of system require different levels of security. For example browsing the order list is not a security critical operation and user can be authenticated based on his Windows account (not to bother him with different form of authentication). When performing more sensitive operations like canceling an order, system might require a stronger proof of user’s identity like a certificate stored on a smart card. Geneva’s programming model makes it easy to secure operations according to their sensitivity level. Developer just needs to access authentication type stored as one of user claims in IClaimsPrincipal. Then, if authentication type is too weak, user is required to prove his identity in another way (and possibly in another STS).

“Act As” Scenario

In one of the whitepapers you can read:

When you build a multi-tier system, typically with a web front end and a collection of web services and other resources on the back end, you have a tough choice to make. Should the web front end use its own identity to access those back end resources, or should delegate the user’s credentials to make those requests? If you choose the first option, you end up with what is known as a trusted subsystem model. If you choose the second, you’ll need some way to delegate the client’s identity to the back end. There are performance and security tradeoffs for both of these choices, with the trusted subsystem model generally favoring performance over security and the delegation model generally favoring security over performance.

The example is:

Web application talks to Service1, which in turn talks to Service2. How should Service1 authenticate at Service2? First solution is to use same credentials for each user of web application. This creates the trusted subsystem and Service2 has to trust Servce1 to only perform those operations, which are permitted for the user. The other solution requires Service1 to act on behalf of the user, which allows Service2 to make it’s own security assertions. In order to achieve that, a special kind of security token has to be issued for Service1, which contains ActAs property set to the users identity.

What’s more, Geneva provides means of transforming security token into Windows identity, which for example allows for using Windows authentication while accessing SQL Server data store by back end services. I recommend you download the beta and check it out yourself.

Tips on refactoring towards testability

2009-04-09T20:58:00.001+02:00

Last time I was expressing my doubts about the value of unit testing. This time I decided to go back to the beginning and show how to start with testing. Newcomers to unit testing often ask themselves how to test things, that are seemingly untestable. Here I gathered some tips on restructuring application’s code in order to achieve testability.

Base principles

What you really want to achieve is two things: possibly low level of component coupling and separated responsibilities. Both things mean ease of testing of one component in separation from other components in the system. The simplest thing to do in order to decouple two classes is to remove direct dependency between them by means of interfaces. When the class we chose for a test subject depends only on interfaces – not concrete classes – these interfaces can be easily mocked, either by hand or using one of the numerous mocking libraries. This allows you to test your class in separation and assert both it’s state and behavior. The single responsibility principle states, that each class should have only one responsibility. In practice this means, that classes performing business logic should not contain code related to any other of the application concerns, for example persistence. This way your logic is testable without necessity to have a database in place.

Where interfaces are not applicable, you can usually use inheritance.This however requires you to mark methods and properties of mocked classes as virtual. Most mocking frameworks support mocking of classes. Remember, that high coupling between application components makes unit testing very hard or even impossible due to extensive setup required to perform even simplest test.

Dependency injection containers

You may wonder how to wire up components of your application when they are supposed to be dependent merely on interfaces. Somewhere the instances of concrete types have to be instantiated anyway, right? This is where DI container come to the rescue. The idea is to register application’s components in a container using the component’s interface as a key. Then, all component dependencies defined in it’s constructor or as public properties can be injected with container’s contents. Components are added to the container at application’s setup phase, for example in entry point method or even declared in XML configuration (although I’m not a fan of this solution). This enables you to easily change container’s registrations during a test and to use mocks instead of real components. Don’t think that DI containers serve only the purpose of making your application testable. As Ayende Rahien writes, it’s not their main purpose. They help a lot, but what they mainly do is to enable you to structure your application in a better way.

Laws and principles

All principles of good object oriented design positively influence testability. Like for example Single Responsibility Principle (SRP) mentioned earlier. Please see “Design Principles and Design Patterns” by Robert C. Martin for more detailed information. I’ll just mention the Law of Demeter here. It states that a component should only directly interact with it’s immediate neighbors in dependency graph. The component should not know nothing of it’s neighbor’s properties structure. In code instead of:

musician.Trumpet.Blow();

you'd write:

musician.PlayInstrument();

You don’t care about PlayInstrument’s implementation. This way test subject is not dependent on Trumpet class, just the Musician class. There is less coupling and testing and mocking is easier. This law also applies to application’s layers – each layer should only interact with a layer directly underneath it.

What about the UI?

There are applications, that have a lot of logic related to user interface: disable this, change color of that… In traditional approach testing of that logic is almost impossible – unless you’re using GUI automated test’s, which didn’t convinced me yet as I wrote in my last post. However there are some design patterns like MVP, MVC and MVVM which allow you to achieve testability in this area. The idea behind all of those patterns is to extract view logic from views and place it in separate classes. Those classes interact with the view itself through interface (MVP, MVC) or don’t interact at all (MVVM) and therefore are testable without need to really make use of any real controls. As for the view itself – you need to keep it as dumb as possible. If there is almost no complexity, it’s not likely to break.

The framework doesn’t allow me to test this…

Quite often you have to make use of third party APIs that are not really designed with testability in mind. This API may be for a component that is slow or requires extensive setup in order to test. It would be better not to have it in test at all, but how to do this when we have direct dependency and no control of API’s structure? The answer is: build a layer of abstraction over this API. An example here is very easy to find: the System.IO.File class. It’ static methods are very useful, but couple your code with file system directly. Instead I suggest you build an adapter and extract it’s interface.

Final thoughts

Although mocking frameworks give you a lot of power, try to assert component’s state after operation, not it’s behavior during. Asserting behavior tends to couple the test and components implementation, which is not really good, because increases cost of maintenance. Testing is hard to start, but it gets easier with each step.

Random thoughts on automated testing

2009-03-31T22:14:00.001+02:00

Recently I’ve been thinking what is required for unit tests to be valuable. First conclusions is that you need good code coverage of critical areas and the tests need to be comprehensive. What is the value of a good test suite? It allows you to easily detect bugs introduced by making software modifications. If you lack test coverage in some areas, a bug can easily slip into your application unnoticed. The other conclusion was, that tests need to be written by people who fully understand the purpose. What I noticed is that some developers just want to “write a test” without any thought on what they really want to test. In worst case they just copy from another test and there you have it – useless piece of code that introduces problems only. So first state what is the purpose of your test (I’m a fan of naming convention, that describes test’s role), then implement it. With those two things you can say that build in your CI environment was successful.

Can you really prove unit tests are worth the effort? What if your manager comes to you and asks you for an example, where well written test suite saved the day? Or at least made your work easier? I whish there was a study on what types of application benefit the most from automated testing. I think that apps concerned mainly with complex business logic or mathematical calculations are at the top of the list, while CRUD and rich-UI-hardly-any-content applications are at the very bottom. With business logic it’s all about whether you can or cannot test components in separation from rest of the system. With UI the only way to automate testing process is to use some UI test automation tools, but they are specific to technology and usually very clumsy and therefore not popular.The best way to test in those cases is just to let your testers take over.

In any case one may wonder if automated tests are really worth the amount of work, that is put into their creation and maintenance. In my actual project I made an observation, that usually when test fails, it is due to desired modifications in code and correction has to be made in the test itself, not the code, in order to make it pass. I tried to think of the reason of this situation, but all I came up with was a conclusion, that the tests are too complicated. We try to keep components of our application as loosely coupled as possible, nonetheless they have dependencies, that need to be set up in order to run the test case. Usually a change in program’s structure causes a few tests to fail. We use RhinoMocks extensively and I’m aware, that state-based testing introduces a lot less complexity than behavior-based testing, but most of the components are about behavior and interaction and this way of testing is unavoidable. What I’m trying to say is that it’s not clear to me if automated tests save money for my organization. Maybe manual-only testing would be cheaper? A nice experiment would be to run same project on two different teams with two approaches to testing and see which works more effectively. It’s all about the money in the end. But how do you measure savings? When releasing buggy software you have to count both money lost for maintenance and money you won’t earn in future because of poor quality image of organization’s software. This is very hard to measure.

Handling complex use case flows

2009-02-28T21:35:00.001+01:00

There are applications, that have fairly complex flows in user interface. In application I’m currently working on there are forms which can be visited many times during execution of one business process, each time behaving differently according to context (values collected earlier). The challenge here is how to write code, that will be testable and maintainable and also not too complex?

I will use a simplified example here. Let’s say the application is used to collect information about car parts. Parts can grouped and each group get’s it’s own ID. You can also associate parts with a car, in which they have been mounted. If you decide to enter car information, it is obligatory to also enter information about car body (treated as car part). The business analyst says, he wants application to work following way:

If I want to enter car part information, I choose “Enter car part” from menu. Car part form is shown.
I then enter car part information and I have to provide car part group identifier.
I click OK and return to menu. Now I have two options: “Enter car part” as previously – this will create new car part group, and “Enter additional car part” – this will add a car part to previously created group (it’s number is a context).
There also is a possibility to choose “Enter car” from the menu. This option also creates new group, but shows car form instead of car part form.
After filling car information, I’m automatically redirected to car part form with car part type set to “car body”. I have to provide group identifier.
If car body number meets certain criteria, I’m redirected to registration certificate form, where I have to provide registration certificate information.
After doing this, I return to the menu and have option to start a new group or add parts to recently created group.

Let’s not go into details why exactly is the UI so awkwardly designed. Sometimes your users think differently than you do. Instead, let’s look at navigation between forms:

Looks pretty simple, but it doesn’t reflect the use case flow at all. The car part form in this simple example can work in 3 modes: entering a car part, entering additional car part and entering car body. This introduces some complexity already, but the tricky part is this: where do you go after completing the car part form? It can be either application menu or registration certificate form depending on car part information and what happened earlier in the flow. Basically what we need here is a state diagram.

This state machine represents our business process of collection car part group information. As you can see the transitions of states are a little more complex than navigation between forms. Three of the states use the same form to perform different tasks: Car, Car part and Car body. Transitions marked with light gray are not directly part of business process, but rather initiate new car part group. It now becomes clear where should we go after leaving car part form – this depends on what state are we currently in.

This diagram makes it clear for us how the application should behave. But how to implement this process? First and most obvious thought is to create a bunch of flag variables and implement some if-else statements. While this is OK in simple scenarios, the real world complexity will make such code completely unreadable and hard to maintain. Next you may consider a workflow engine. This allows you to express transition logic in readable way, but in most cases this is too heavyweight for regular applications. And what if you have to implement this on mobile device? No workflow engine there.

Suppose this application uses MVC pattern. We focus on the “C” here. The controller most probably has methods like “OnCarPartEntered” taking in car part information object as parameter. This method saves the information and decides what view to show next and in what mode. This is where your “if-else” would go if you opted for the first solution. In this post I suggest using the state machine pattern to implement the controller (a similar was used in my post on circuit breaker pattern). You have to implement your controller interface for each state separately. Following diagram depicts the situation:

The context class is an access point to the state machine. It knows all the states and knows which state is the current one. It exposes business process context (group number here) and methods used to switch states through IContext interface. Since it also implements the controller interface, it forwards all method calls to current state. Each application state has it’s own implementation of IMyController. These classes derive from StateBase, which can contain behavior common to all states. Derived classes override this behavior when appropriate. For example the CarBodyState will check the car body number criteria match after leaving car part form. The CarPartState will just return to main menu in this situation. The state classes are also responsible for switching states in reaction to certain actions of the user. Following diagram shows sample state transition:

Before executing this sequence, application is in Car body state and car part form is displayed. After execution, application is in Additional part state and menu form is displayed. The code is quite straightforward, so I won’t be cluttering this post with it :)

Zero friction APIs

2009-02-11T21:05:00.001+01:00

This is what Ayende talked about some time ago now and got a few interesting comments. The essence of his post is that you (an architect) should be able to setup a project in a way that will make the easiest way of doing things also the best way in terms of the architecture.

There is a lot of base code to be developed before the actual business functionality can be implemented. I mean, even if you’re in position when you can take advantage of latest technologies, all the jEntityMVCInjectionFoundation libraries, you usually build your own abstractions and base class libraries in order to both remove dependency on particular technology and encapsulate some reusable behaviors. These base class libraries of yours are used by developers on your team. The question is do you put enough effort in designing these libraries to be zero friction? I’ve seen some APIs being pretty slick (StructureMap) and some really ugly (LLBLGen Pro), but these are widely used libraries and their authors have to design friendly APIs as part of their job. You should too. Especially because the developers will most probably make direct use those other libraries less often than will use your base class library.

If I were to create a list of things that should be considered when creating a library, this would be in order of importance:

Your library should get things done. And do it the best way possible in terms of complexity, performance and maintainability. (Yes, this is always most important and it’s the point of creating the library in the first place. The other are worthless without this one – keep this in mind).
Name classes, methods, etc. so that they are self-explanatory and obvious. Don’t make me refer to any documentation. (This, as everything, is not always possible. If that’s the case, provide a friendly and useful documentation for me).
Think about testing. Is your library designed so that testing is easy? Can I mock your classes?
Make sure you don’t have any hidden dependencies. Your classes will be used by other people that won’t have an insight on their inner workings. Suppose you have method X. In order to call it, you have to call Y before. Y doesn’t return any value that is used as X’s parameter. It’s a hidden dependency and most probably you’re doing something wrong. Don’t make me refer to sample code in order to do event the simplest things (although samples will never hurt).
Think about whether your library is easy to extend and maintain. Also think about API versioning policy if applicable.
Don’t overcomplicate things. Do not provide a hundred layers of abstraction just because you think in few years they may come in handy. If they will, you can add them at that time – and refactor other things too. Remember, YAGNI.
Provide a slick syntax for me. A fluent one if applicable. Make the code look elegant, compact and easy to read.

If your base and helper class library fulfills most of those things, people using it will be grateful.

HTTPS with client certificate authentication in .NET CF

2009-02-04T22:21:00.001+01:00

The struggle with .NET CF 1.0 continues.

The HTTPS is just ordinary HTTP sent over a secure TLS (extension of SSL) connection. The secure connection is initiated in following way:

Server and client negotiate strongest cipher algorithms supported by both parties
Server sends its certificate to the client
Client can verify correctness of the certificate by verifying digital signature of the Certification Authority that issued the certificate
Client generates a random number (master secret), encrypts it with server’s public key and sends it to server. Only server can decrypt this message with its private key.
Both parties derive a key for symmetric algorithm from the master secret. Following communication is encrypted using this key.

After completing those steps, client and server have a secure way to communicate. Additionally the client knows, that server is trusted. The server on the other hand doesn’t have any information about the client. It often is required to authenticate the client before allowing it to access confidential resources. This can be done in many different ways, one of which is a client certificate. This option is supported directly in TLS – the client sends its certificate right after it receives certificate from server. To confirm, that it owns this certificate, the client also has to prove, that it owns the private key associated with certificate. In order to do this, after establishing master secret the client sends digital signature over all messages exchanged with server up to that point. Server verifies this signature using client’s public key.

I’m developing a client that has to authenticate at HTTPS web service with client certificate. Unfortunately this scenario is not supported in .NET CF prior to version 3.5. While looking for a way to work around this problem, I came by library called SecureBlackbox. The package called SSLBlackbox supports my scenario and additionally provides very comprehensive support for X509 certificates, which .NET CF lacks. What is interesting, the library developers don’t use any external APIs to perform the cryptography. The implemented all algorithms from scratch. To evaluate this library I needed a test environment. I decided to use simple ASP.NET page hosted on IIS server that would display client certificate information associated with request. The code comes from MSDN:

protected void Page_Load(object sender, EventArgs e)
{
    HttpClientCertificate cs = Request.ClientCertificate;
    if (cs != null)
    {
        Response.Write("ClientCertificate Settings:<br>");
        Response.Write("Certificate = " + cs.Certificate + "<br>");
        Response.Write("Cookie = " + cs.Cookie + "<br>");
        Response.Write("Flags = " + cs.Flags + "<br>");
        Response.Write("IsPresent = " + cs.IsPresent + "<br>");
        Response.Write("Issuer = " + cs.Issuer + "<br>");
        Response.Write("IsValid = " + cs.IsValid + "<br>");
        Response.Write("KeySize = " + cs.KeySize + "<br>");
        Response.Write("SecretKeySize = " + cs.SecretKeySize + "<br>");
        Response.Write("SerialNumber = " + cs.SerialNumber + "<br>");
        Response.Write("ServerIssuer = " + cs.ServerIssuer + "<br>");
        Response.Write("ServerSubject = " + cs.ServerSubject + "<br>");
        Response.Write("Subject = " + cs.Subject + "<br>");
        Response.Write("ValidFrom = " + cs.ValidFrom + "<br>");
        Response.Write("ValidUntil = " + cs.ValidUntil + "<br>");
        Response.Write("What's this = " + cs.ToString() + "<br>");
    }
    else
    {
        Response.Write("No certificate");
    }
}

I also needed to generate certificate for server and client. I used makecert.exe according to hints from Manu Cohen-Yashar’s post. The only modification I made is to generate a personal client certificate rather a one for machine. To do this just specify CurrentUser instead of LocalMachine as a target certificate store.

Next step is to configure IIS, I use version 7. You need to configure HTTPS binding with created server certificate – configured at web site level. Then you have to add an application under this web site for previously created ASP.NET page. Then, for this application you need to set SSL setting so that SSL is enabled and client certificates are required:

The easiest way to verify if server’s setup is ok is to browse create page with Internet Explorer. Since IE uses certificates installed in personal store, no additional setup is required for this browser. You can also use Firefox for example, but you need to export the client certificate to PFX file (with private key) and import it to the browser. When a page requests client certificate, the browser will let you choose certificate to be sent. Following is the result from my page:

Ok, after setting up the server, it was time to write some client code.

class Program
{
    static TElX509Certificate cert = new TElX509Certificate();
    
    static void Main(string[] args)
    {
        SBUtils.Unit.SetLicenseKey("<license key here>");
        var stream = File.OpenRead(@"<client certificate file>.pfx");
        cert.LoadFromStreamPFX(stream, "<private key password here>", 0);
        stream.Close();

        TElHTTPSClient https = new TElHTTPSClient();
        https.OnData += new SBSSLCommon.TSBDataEvent(https_OnData);
        https.OnCertificateValidate += 
            new SBSSLCommon.TSBCertificateValidateEvent(https_OnCertificateValidate);
        https.OnCertificateNeededEx += 
            new SBClient.TSBCertificateNeededExEvent(https_OnCertificateNeededEx);
        
        int code = https.Get("https://myurl/");
        Console.WriteLine(code);

        Console.ReadLine();
        https.Close(true);
    }

    static bool wasCert = false;

    static void https_OnCertificateNeededEx(object Sender, ref TElX509Certificate Certificate)
    {
        if (wasCert)
            return;

        Certificate = cert;
        wasCert = true;
    }

    static void https_OnCertificateValidate(object Sender, TElX509Certificate X509Certificate,
        ref bool Validate)
    {
        Validate = true;
    }

    static void https_OnData(object Sender, byte[] Buffer)
    {
        Console.WriteLine(Encoding.UTF8.GetString(Buffer));
    }
}

Am I the only one that gets impression that syntax could be improved here? Anyway this works just fine. If requested URL requires client certificate, the OnCertificateNeededEx event is called and you can return the certificate. This method is supposed to be called over and over until it returns null in Certificate parameter. This way you can return whole certificate chain. A developer is also required to validate server’s certificate (OnCertificateValidate event). I just return true here, so all certificates are accepted – not a production ready solution.

All would be great if not for performance issues on mobile devices. It seems that HTTPS handshake requires a lot of computation power, which mobile devices lack. I got results up to 14s for first handshake! I’m still trying to get around this. Also the library’s size is considerable – over 2mb is significant on mobile device.

Java web service interoperability from .NET CF 1.0

2009-01-23T17:53:00.001+01:00

I don’t suppose anyone else is developing in .NET CF 1.0 now in 2009. Well those are the constraints for project I’m involved in and I prefer to think about this as a challenge :)

Anyway current topic is interoperability with Java web services. This causes a lot of problems in .NET CF 1.0, some of them weren’t addressed until .NET CF 3.5. In my case it started with inability to generate web service client proxy using standard wsdl.exe tool.

In this example we have a Java web service for querying cars by name. It returns car description with name and collection of car parts. Each car parts can have a set of string attributes. The model looks like this:

Let’s say we have WSDL like this one:

<?xml version="1.0" encoding="UTF-8"?>
<wsdl:definitions name="ws" targetNamespace="http://tempuri.org" 
                  xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/" 
                  xmlns:tns="http://tempuri.org" 
                  xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" 
                  xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <wsdl:types>
    <xsd:schema targetNamespace="http://tempuri.org" xmlns:pref="http://tempuri.org/to">
        <xsd:import namespace="http://tempuri.org/to" schemaLocation="myschema.xsd" />
        <xsd:element name="getCarDesc">
            <xsd:complexType>
          <xsd:sequence>
            <xsd:element name="carQuery" type="pref:CarQuery" maxOccurs="1" minOccurs="1" />
          </xsd:sequence>
            </xsd:complexType>
        </xsd:element>
        <xsd:element name="getCatDescResponse">
            <xsd:complexType>
                <xsd:sequence>
                    <xsd:element name="response" type="pref:Car" maxOccurs="1" minOccurs="0" />
                </xsd:sequence>
            </xsd:complexType>
        </xsd:element>
    </xsd:schema>
  </wsdl:types>
  <wsdl:message name="carRequest">
    <wsdl:part element="tns:getCarDesc" name="parameters"/>
  </wsdl:message>
  <wsdl:message name="carResponse">
    <wsdl:part element="tns:getCarResponse" name="parameters"/>
  </wsdl:message>
  <wsdl:portType name="ws">
    <wsdl:operation name="getCar">
      <wsdl:input message="tns:carRequest"/>
      <wsdl:output message="tns:carResponse"/>
    </wsdl:operation>
  </wsdl:portType>
  <wsdl:binding name="wsSOAP" type="tns:ws">
    <soap:binding style="document" transport="http://schemas.xmlsoap.org/soap/http"/>
    <wsdl:operation name="getCar">
      <wsdl:input>
        <soap:body use="literal"/>
      </wsdl:input>
      <wsdl:output>
        <soap:body use="literal"/>
      </wsdl:output>
    </wsdl:operation>
  </wsdl:binding>
  <wsdl:service name="ws">
    <wsdl:port binding="tns:wsSOAP" name="wsSOAP">
      <soap:address location="http://myhost/get-cat-web/ws" />
    </wsdl:port>
  </wsdl:service>
</wsdl:definitions>

There is one service and it has one method, getCar. This method takes CarQuery object as a parameter and returns a Car object as response. The WSDL refers to external schema in myschema.xsd file, which contains type definitions.

<?xml version="1.0" encoding="UTF-8"?>
<schema xmlns="http://www.w3.org/2001/XMLSchema" 
        targetNamespace="http://tempuri.org/to" 
        xmlns:tns="http://tempuri.org/to" 
        elementFormDefault="qualified"
        xmlns:jaxb="http://java.sun.com/xml/ns/jaxb" jaxb:version="1.0">

    <complexType name="CarPart">
        <sequence>
            <element name="attribute" type="string" maxOccurs="unbounded" minOccurs="1" />
        </sequence>
    </complexType>

    <complexType name="Car">
        <sequence>
          <element name="name" type="string" maxOccurs="1" minOccurs="1" />
          <element name="parts" type="tns:CarPart" maxOccurs="unbounded" minOccurs="0" />
        </sequence>
    </complexType>

    <complexType name="CarQuery">
        <sequence>
            <element name="name" type="string" maxOccurs="1" minOccurs="1" />
        </sequence>
    </complexType>
</schema>

This works perfectly in Java and is used by Java guys to generate a web service. When you try to feed this to wsdl.exe, the only thing it has to say is:

Error: Unable to import binding 'wsSOAP' from namespace 'http://tempuri.org'.
 - Unable to import operation 'getCar'.
 - The operation binding 'getCar' from namespace 'http://tempuri.org' had invalid syntax.
Missing soap:operation binding.

At least wsdl.exe is being quite specific about this one. I went to validate my WSDL at this site and it confirmed, that I lack soap:operation tag. So I added it and specified operation URI:

    <wsdl:operation name="getCar">
      <soap:operation soapAction="http://tempuri.org/ws/getCar" style="document"/>
      <wsdl:input>
        <soap:body use="literal"/>
      </wsdl:input>
      <wsdl:output>
        <soap:body use="literal"/>
      </wsdl:output>
    </wsdl:operation>

The wsdl.exe finished successfully and generated proxy class along with used types. This however was not the end of my struggle. It turned out, that the generated code won’t compile int .NET CF 1.0 as there is no support for Form property of XmlElementAttribute class. The web service method was decorated with this attribute in following way:

[return: System.Xml.Serialization.XmlElementAttribute("response", 
    Form=System.Xml.Schema.XmlSchemaForm.Unqualified)]
public Car getCar(
    [System.Xml.Serialization.XmlElementAttribute(Form=System.Xml.Schema.XmlSchemaForm.Unqualified)] 
    CarQuery carQuery) 
{
    ...
}

It’s really strange that this tool would generate code that won’t compile (and yes, the result was the same when created using Add Web Reference in Smart Device project). I went to look for solution on the web and came up with this modification to WSDL (added elementFormDefault attribute):

...
<wsdl:types>
    <xsd:schema elementFormDefault="qualified" 
                targetNamespace="http://tempuri.org" xmlns:pref="http://tempuri.org/to">
...

This removed all usages of the Form property of XmlElementAttribute class. The code would now compile, but looked a little different than I expected. Take a look at the Car class:

public class Car {
    
    private string nameField;
    
    private string[][] partsField;
    
    /// <remarks/>
    public string name {
        get {
            return this.nameField;
        }
        set {
            this.nameField = value;
        }
    }
    
    /// <remarks/>
    [System.Xml.Serialization.XmlArrayItemAttribute("attribute", typeof(string), IsNullable=false)]
    public string[][] parts {
        get {
            return this.partsField;
        }
        set {
            this.partsField = value;
        }
    }
}

There’s no CarPart object collection. There was no CarPart class in generated code at all. Instead, the tool generated string[][] array for me. Well it’s not what I would like to have, but I could live with it. So I went on to code that test the web service and to my surprise the constructor of the proxy class threw FileNotFoundException as the application was unable to find a dynamically created assembly used by XML serializer. I tried to do the same with Visual Studio 2008 and here’s what I got:

InvalidOperationException was unhandled
Unable to generate a temporary class (result=1).
error CS0029: Cannot implicitly convert type 'string' to 'string[]'

Not only the wsdl.exe changed the object model, but also the XML Serializer was unable to deal with it. At this point I decided to manually modify the generated code and added the CarPart class. I also changed parts collection to CarPart[] type. And it finally worked. Yet I feel disappointed that I have to manually create proxy classes to work with Java web services.

A whole different thing is performance of web services on the smart device. SOAP and XML imply large message sizes and extensive use of reflection, which has a great impact on the application’s performance.

Dependency injection in .NET CF 1.0

2009-01-17T00:10:00.001+01:00

Recently I got involved in building an application for mobile devices. The problem is that they are old Pocket PC 2002 devices, so the only option to write managed code is to use also old .NET CF 1.0. Don’t even get me started on how limited it is and how much pain it is to work with VS2003 – do you remember the times when you had to add namespace imports by hand? Without ReSharper it’s almost unbearable.

Anyway I’m responsible for the design of the solution. I thought it would be nice to have DI container, but there are some constraints on the mobile platform:

Performance – There’s no possibility to use reflection extensively without running into performance problems.
Limited BCL - .NET CF 1.0 base class library is only a small subset the .NET 1.1 BCL, not to mention .NET 2.0

Taking this into account I thought it would be best to implement a really simple solution myself. The features of my simple container are:

Mapping interfaces to implementing types
Registering types both as singleton and as transient instance
Registering existing instances
Resolving both mapped interfaces and unmapped types
Injecting dependencies into constructors (recursively)
Fluent registration syntax
Reflection usage reduced to a minimum

This set of functionalities should allow me to work comfortably with dependency injection and keep the container as simple as possible. Here is the code, first the container interface:

public interface IContainer
{
    IContainer Register(Type mapFrom, Type mapTo);
    IContainer RegisterSingleton(Type mapFrom, Type mapTo);
    IContainer RegisterSingleton(Type mapFrom, object toInstance);
    object Resolve(Type type);
}

Register method allows you to register a mapping of a transient objects (instantiated on every resolve). RegisterSingleton is overloaded and allows you to register either a type mapping or type to existing instance mapping as a singleton (instantiated only once). Resolve returns instance of requested type – either mapped or not.

Now TypeItem class used to store type mappings (in a Hashtable):

public class TypeItem
{
    private Type targetType;

    public Type TargetType { get { return targetType; } }

    private bool sigleton;

    public bool Singleton { get { return sigleton; } }

    public TypeItem(Type targetType, bool singleton)
    {
        this.targetType = targetType;
        this.sigleton = singleton;
    }
}

And the container itself:

public class Container : IContainer
{
    private Hashtable typeMap = new Hashtable();
    private Hashtable singletonCache = new Hashtable();
    private ArrayList targetTypeList = new ArrayList();

    #region IContainer Members

    public IContainer Register(Type mapFrom, Type mapTo)
    {
        CheckRegistrationConditions(mapFrom, mapTo, true);

        TypeItem item = new TypeItem(mapTo, false);
        AddItem(mapFrom, item);
        return this;
    }

    public IContainer RegisterSingleton(Type mapFrom, Type mapTo)
    {
        CheckRegistrationConditions(mapFrom, mapTo, true);

        TypeItem item = new TypeItem(mapTo, true);
        AddItem(mapFrom, item);
        return this;
    }

    public IContainer RegisterSingleton(Type mapFrom, object toInstance)
    {
        CheckRegistrationConditions(mapFrom, null, false);
        Guard.AgainstNullArgument(toInstance, "toInstance");

        TypeItem item = new TypeItem(toInstance.GetType(), true);
        if(targetTypeList.Contains(item.TargetType))
        {
            throw new ArgumentException(
                "Cannot register singleton because another singleton mapping " + 
                "uses the type of the instance",
                "toInstance");
        }
        AddItem(mapFrom, item);
        singletonCache.Add(item.TargetType, toInstance);
        return this;
    }

    public object Resolve(Type type)
    {
        Guard.AgainstNullArgument(type, "type");

        if (typeMap.ContainsKey(type))
        {
            // mapped type
            TypeItem item = typeMap[type] as TypeItem;

            if (item.Singleton && singletonCache.ContainsKey(item.TargetType))
                return singletonCache[item.TargetType];

            object instance = BuildInstance(item.TargetType);

            if (item.Singleton)
                singletonCache.Add(item.TargetType, instance);

            return instance;
        }
        else
        {
            // unmapped type
            if (!type.IsClass || !type.IsPublic)
                throw new TypeResolveException(
                    "The type " + type.Name + 
                    " is neither mapped nor can it be instantiated");

            return BuildInstance(type);
        }
    }

    #endregion

    private void CheckRegistrationConditions(Type mapFrom, Type mapTo, bool mapToValid)
    {
        Guard.AgainstNullArgument(mapFrom, "mapFrom");
        if (mapToValid)
        {
            Guard.AgainstNullArgument(mapTo, "mapTo");
            if (!mapTo.IsClass || !mapTo.IsPublic)
                throw new ArgumentException("The type " + mapTo.Name + 
                    " cannot be instantiated");
        }

        if (typeMap.ContainsKey(mapFrom))
            throw new ArgumentException("Provided type is already mapped", "mapFrom");
    }

    private void AddItem(Type mapFrom, TypeItem item)
    {
        typeMap.Add(mapFrom, item);
        targetTypeList.Add(item.TargetType);
    }

    private object BuildInstance(Type type)
    {
        ConstructorInfo[] constructors = type.GetConstructors(
            BindingFlags.Public | BindingFlags.Instance);

        if (constructors.Length == 0)
            throw new TypeResolveException("The type " + type.Name +
                " has no public constructors");

        ParameterInfo[] parameters = constructors[0].GetParameters();
        object[] paramValues = new object[parameters.Length];

        for (int i = 0; i < parameters.Length; i++)
        {
            paramValues[i] = Resolve(parameters[i].ParameterType);
        }

        return constructors[0].Invoke(paramValues);
    }

}

As you might noticed, the BuildInstance method just takes the first public constructor returned by GetConstructors and uses it to create instance. This is enough for my scenario, but you might want to mark injection constructor with a custom attribute or select constructor basing on some criteria (ex. most resolvable parameters).

There are also some tests for the container, but I won’t be posting them here.

Comparing OpenID and canonical Subject-IP-RP pattern

2009-01-12T23:15:00.001+01:00

A hot topic lately is identity management in distributed environments and there are some interesting articles on this in the Architecture Journal. The basic thing to do in distributed environment, presumably SOA, is to delegate the identity management to separate component called Identity Provider (IP). It is responsible for maintaining user database and issuing security tokens. This security tokens are used by the subject (a user who wants to gain access to a resource) to authenticate at the Relaying Party (a component that maintains the resource). This way user doesn’t have to have separate user account on each Relaying Party and presumably authenticate in different way using different credentials. The Relaying Party trusts the Identity Provider to perform user authentication. Following diagram presents this situation:

Now in order to authenticate at the Relaying Party, following steps are taken [1]:

Subject first discovers how to authenticate at Relaying Party by getting the security policy. It then authenticates at Identity Provider using proper credentials and receives a security token, which is used to authenticate resource requests at Replaying Party. Relaying Party verifies the token’s signature to ensure it comes from the trusted Identity Provider.

OpenID is a decentralized and open identification standard basing on a concept of trusted Identity Provider. The “distributed” in practice means, that there are many IPs on the web and it’s up to user to decide which one will suit him. For example there is OpenID provider for Google’s Federated Login API, which you can use your Google account to authenticate with OpenID. There are also some dedicated IPs and you even can easily setup your own IP if you do not trust anybody else :). OpenID was created with authentication at web portals and communities in mind (it’s being used by Stack Overflow for instance). It’s well suited for this task and differs from the canonical Subject-IP-RP pattern. It is assumed, that user uses web browser to authenticate and the protocol relies heavily on HTTP/HTML (but implementation in desktop apps is also possible). Authentication process orchestration is taken from hands of the subject and directed by Relaying Party instead. Username has a form of URL and plays significant role during the process of authentication as follows [2](a rough view):

Relaying Party queries username URL to discover information on Identity Provider used by given user. Then, user’s browser is redirected to IP’s login site. After logging in, IP redirects user to RP’s return URL including information about authentication result in the request. RP verifies whether the login status information is legitimate by querying IP.

There are also some modifications and extensions to this protocol like for example ability for RP and IP to establish a shared secret used to verify login status information without the need to query IP every time a user logs in.

Implementation of OpenID authentication is easy, because there are many open libraries for each platform. For .NET there is DotNetOpenId, which even provides ready to use controls for ASP.NET. I suggest you read OpenID specs [2] for more detailed information on this topic. Anyway it would be nice to see OpenID authentication more popular and used by all community portals.

Resources:

[1] Vittorio Bertocci, “Claims and Identity: On-Premise and Cloud Solutions”. The Architecture Journal, Journal 16.

[2] OpenID Authentication 2.0 – Final. http://openid.net/specs/openid-authentication-2_0.html, 2009.

Exploring Boo’s Abstract Syntax Tree (AST)

2008-12-26T14:11:00.001+01:00

Currently I’m going through AST, but the documentation is scarce. AST is in-memory tree representation of code that can be modified and is used while creating syntactic macros. I was wondering what is the best way to check how syntax is transformed to AST when I came across Luis Diego Fallas’ post in which he shows a little application that parses Boo source file and displays AST in tree control. Unfortunately there are some issues with this application. Firstly, it contains a bug and doesn’t show full syntax tree. Secondly, it only displays node names without information what code the node represents. So I decided to modify his code and fix this. Here is a sample screenshot:

There is AST tree structure of code on the left and information about selected subtree on the right. It includes the code itself and serialized form of the subtree. Name of the file to parse is passed to the application as a parameter.

Visitor pattern is employed to visit all nodes in AST tree. Unfortunately deriving from Boo’s DepthFirstVisitor is a tedious task, because there are as many methods to override as there are AST node types. Luis proposed a solution basing on dynamically created type, that derives from DepthFirstVisitor. Method overrides are constructed in-memory from AST nodes and then the dynamic type is compiled. While being compact, this solution is not very readable, especially when you’re not familiar with AST. Fortunately there seems to be alternative way to do this, namely the mysterious [| <code here> |] syntax. I think it’s called quasiquotes. I couldn’t find any documentation for this, only one example in Boo package. From what I was able to figure out, this code (where m is instance of MethodInfo class):

// Node for TreeNode('')
tnCreation = MethodInvocationExpression(ReferenceExpression("TreeNode"))                                 
tnCreation.Arguments.Add(
        StringLiteralExpression(m.GetParameters()[0].ParameterType.Name))

// Create node for 't = TreeNode('')' 
decl = DeclarationStatement(
      Declaration: Declaration(Name:"t",
                 Type: SimpleTypeReference("System.Windows.Forms.TreeNode")),
      Initializer: tnCreation)

is equivalent to this code:

decl = [| t = TreeNode($(m.GetParameters()[0].ParameterType.Name)) |]

So there is a huge improvement. Code between [| and |] is being transformed to AST and assigned to a variable. The $() construct is similar to string interpolation construct. It allows you to insert a string from variable as if it was part of the code (here it is a parameter of the TreeNode constructor, but it also can be a method name for example). When using this syntax, you have to be careful about the indentation. It seems that statements can be in the same line as [|, but when declaring members, they have to be in new line and indented. Otherwise you get compilation errors.

Luis has overridden “Enter*” and “Leave*” (for example EnterBlockExpression) methods of DepthFirstVisitor, which is a problem because “Enter*” and ”Leave*” methods exist only for nodes that can contain subnodes. So for example there is no Enter method for IntegerLiteralExpression node. In my code I override the “On*” methods.

Here is the full code:

import System
import System.Reflection
import System.IO
import Boo.Lang.Compiler
import Boo.Lang.Compiler.Ast
import Boo.Lang.Compiler.Ast.Visitors
import Boo.Lang.Parser
import System.Windows.Forms
import System.Xml.Serialization
import System.Drawing


// Get the type instance for DepthFirstVisitor
a = Assembly.Load("Boo.Lang.Compiler")
visitorType = a.GetType("Boo.Lang.Compiler.Ast.DepthFirstVisitor")

// Create a class definition for a custom visitor
myVisitor = ClassDefinition(Name: "DynamicVisitor")
myVisitor.BaseTypes.Add(SimpleTypeReference("Boo.Lang.Compiler.Ast.DepthFirstVisitor"))

// Add a new field for our node stack
stackField = [| 
    public stack as System.Collections.Stack 
|]
myVisitor.Members.Add(stackField)

// Override On* methods
for m as MethodInfo in [m for m in visitorType.GetMethods() if m.Name.StartsWith("On")]:
    method = [|
        def $(m.Name)(node as $(m.GetParameters()[0].ParameterType.FullName)):
            t = TreeNode($(m.GetParameters()[0].ParameterType.Name))
            t.Tag = node;
            (stack.Peek() as TreeNode).Nodes.Add(t)
            stack.Push(t)
            // call method in superclass to visit subtrees
            super.$(m.Name)(node)
            stack.Pop() 
    |]
    myVisitor.Members.Add(method)

// Compile unit and module creation
cu = CompileUnit()
mod = Boo.Lang.Compiler.Ast.Module(Name:"Module")
cu.Modules.Add(mod)

mod.Imports.Add(Import(Namespace:"System"))
mod.Imports.Add(Import(Namespace:"System.Windows.Forms"))
mod.Imports.Add(Import(Namespace:"Boo.Lang.Compiler.Ast"))
mod.Members.Add(myVisitor)

// Initialize compiler preferences
pipeline = Pipelines.CompileToMemory()
ctxt = CompilerContext(cu)

// Run the compiler
pipeline.Run(ctxt)

// Check the results
if(ctxt.GeneratedAssembly == null):
    for e in ctxt.Errors:
        print e
    Console.ReadLine()
    return    

// Create an instance of the new visitor and initialize it
visitorInstance as object = ctxt.GeneratedAssembly.CreateInstance("DynamicVisitor");
visitorType = ctxt.GeneratedAssembly.GetType("DynamicVisitor")
tStack = System.Collections.Stack()
tStack.Push(TreeNode("root"))
visitorType.GetField("stack").SetValue(visitorInstance, tStack);

// Parse a file an run the visitor
compileUnit = BooParser.ParseFile(argv[0])
compileUnit.Accept(visitorInstance)

//Initialize a form an show the results
frm = Form(Text: "AST content", Width: 800, Height: 600)
outerSplit = SplitContainer(Dock: DockStyle.Fill, Orientation: Orientation.Vertical)
innerSplit = SplitContainer(Dock: DockStyle.Fill, Orientation: Orientation.Horizontal)
definitionText = TextBox(Dock: DockStyle.Fill, Multiline: true, 
    ScrollBars: ScrollBars.Both, Font: Font("Courier New", 8.0))
xmlText = TextBox(Dock: DockStyle.Fill, Multiline: true, 
    ScrollBars: ScrollBars.Both, Font: Font("Courier New", 8.0))

tv = TreeView(Dock: DockStyle.Fill)

tv.AfterSelect += def (sender as object, e as TreeViewEventArgs):
    if e.Node.Tag != null:
        writer = StringWriter()
        BooPrinterVisitor(writer).Visit(e.Node.Tag as Node)
        definitionText.Text = writer.ToString()
        
        serializer = XmlSerializer(e.Node.Tag.GetType())
        writer = StringWriter()
        serializer.Serialize(writer, e.Node.Tag)
        xmlText.Text = writer.ToString()

tv.Nodes.Add(tStack.Pop() as TreeNode)
frm.Controls.Add(outerSplit)
outerSplit.Panel1.Controls.Add(tv)
outerSplit.Panel2.Controls.Add(innerSplit)
innerSplit.Panel1.Controls.Add(definitionText)
innerSplit.Panel2.Controls.Add(xmlText)
Application.Run(frm)

Some highlights:

There is a BooPrinterVisitor class in Boo.Lang.Compiler.Ast.Visitors namespace, that converts AST to source code. Unfortunately this class doesn’t support quasiquotes, so passing a file with quasiquotes will result in exception.
The file passed to the application doesn’t necessarily have to be compilation error free. It can contain macros, that are not yet defined. This way you can write your desired syntax, and then use this application to see how it is parsed. This should make macro implementation easier.

An example of IQuackFu usage in Boo

2008-12-14T12:40:00.001+01:00

The IQuackFu interface lets you intercept calls to a duck type when called method or property doesn’t exist. You can use this to achieve many interesting things. In Boo’s examples there is a class for accessing XML data and also a class that allows dynamic inheritance. The IQuackFu interface has three methods:

QuackInvoke(name as string, args as (object)) as object – intercepts method calls
QuackSet(name as string, parameters as (object), value) as object – intercepts property setter calls
QuackGet(name as string, parameters as (object)) as object – intercepts property getter calls

As you can see each method has a name argument, which contains a name of the called method or property. This name can be parsed and appropriate action can be taken. My example will be a simple class for accessing data in a database table. Methods called on this class should be in form of:

Get <table name> By <column 1> [And <column 2> And …] ( <value 1> [, <value 2>, …] )

The method name specifies which table will be queried and what columns will be used in the WHERE clause. Arguments of the method specify values for the filter. The method returns a DataSet. Here is the code:

import System
import System.Text.RegularExpressions
import System.Collections.Generic
import System.Data.SqlClient
import System.Data

class Repository(IQuackFu):
    
    def QuackInvoke(name as string, args as (object)) as object:

        match = /^Get(?<table>\w+)By(?<column>\w+?)(And(?<column>\w+?))*$/.Match(name)
        if match != null and match.Success:
            table = match.Groups["table"].Value
            
            params = List[of string]()
            for c as Capture in match.Groups["column"].Captures:
                params.Add(c.Value)
            
            if len(args) == len(params):
                return RunQuery(table, params.ToArray(), args)
            else:
                raise InvalidOperationException("Column count doesn't match parameter count")
        else:
            raise InvalidOperationException("Method name has incorrect format")
    
    private def RunQuery(table as string, params as (string), args as (object)):
        paramString = ""
        for i in range(len(params)):
            paramString += " and ${params[i]} = @p${i}"
        commandText = "select * from ${table} where (1 = 1)${paramString}";
        
        using connection = SqlConnection("<your connection string here>"):
            command = SqlCommand(commandText, connection)
            
            for i in range(len(args)):
                command.Parameters.AddWithValue("@p${i}", args[i])

            connection.Open()
            
            dataSet = DataSet()
            SqlDataAdapter(command).Fill(dataSet)
            return dataSet
    
    def QuackSet(name as string, parameters as (object), value) as object:
        pass
    
    def QuackGet(name as string, parameters as (object)) as object:
        pass

Sample usage of this class is:

repository = Repository()
customers = repository.GetCustomersByCountry("Germany") as DataSet
products = repository.GetProductsByUnitsInStockAndReorderLevel(0, 0) as DataSet

As you can see, the result of the method is casted to DataSet. Without this, the result would be of the duck type, which means its members would be resolved at runtime. While this may be OK, you should consider two things:

Duck typing is approx. 160 times slower than static typing.
There is a known issue with Boo 0.8.2 causing AmbiguousMatchException to be thrown when an overloaded indexer is accessed on a duck type. This means that accessing indexer of Tables collection of the DataSet class through duck typing will result in this exception.

Boo language with SharpDevelop

2008-12-13T14:04:00.001+01:00

Recently I had urge to wander off the track laid by almighty Microsoft and take a peek into open source community. And there it is – the Boo language for CLI. It is announced as a “wrist friendly” language, which basically means that it has smooth syntax and other facilities that ease the hard life of a developer. The syntax is Python-like (or Ruby-like if you prefer), but the language itself is statically typed. It imitates dynamically typed language by type inference (where possible). The greatest power of Boo lies in extensibility of the language and the compiler, but let’s start at the beginning.

True programmers use the notepad, but lazy ones prefer to have some IDE. There is Boo support built into open source SharpDevelop IDE. You get project templates, syntax highlighting, debugger and NUnit integration for free.

This is no Visual Studio, but still impressive.

If you would like to learn about Boo, just visit it’s home page. As for this post I would like to bullet point some features that make programming in Boo more pleasant than C#:

First class functions. No need to declare a class, even for entry point.

Puts your code on a diet. It’s this:

class Calculator:
    def Add(a as int, b as int):
        return a + b

print Calculator().Add(1, 2)

versus this:

public class Calculator
{
    public int Add(int a, int b)
    {
        return a + b;
    }
}

public static class Program
{
    public static void Main()
    {
        Console.WriteLine(new Calculator().Add(1, 2));
    }
}

Builtin literals for arrays, lists and hashes (like Hashtable)

myArray = (1, 2, 4)
myList = [1, 2, "apple"]
myHash = { "key1" : "value1", "key2" : "Value2" }

print myArray
print myList
print myHash

// outputs:
// System.Int32[]
// [1, 2, apple]
// Boo.Lang.Hash

Generators, which are basically expressions for defining a sequence based on condition. Those sequences are not stored in memory, but rather executed on demand (yeah, you can do this with LINQ too)

squaresForEvenNumbers = i*i for i in range(10) if i % 2 == 0
print join(squaresForEvenNumbers, ', ')
fruits = ("apple", "bannana", "strawberry", "raspberry")
berries = "Berry: " + fruit for fruit in fruits if fruit.Contains("berry")
print join(berries, ', ')

// outputs:
// 0, 4, 16, 36, 64
// Berry: strawberry, Berry: raspberry

String interpolation (no more String.Format)
```
name = "Marcin"
print "Hello ${name}"
```
Builtin literals for regular expressions.
```
print /\d{2}-\d{3}/.IsMatch("12-333")
```

Slicing

numbers = (1, 2, 3, 4, 5)
print join(numbers[1:3])
name = "Marcin"
print join(name[-3:])

// outputs
// 2 3
// c i n

Duck typing, which is what you will get with the dynamic keyword in C# 4. This means being able to resolve member names at runtime. But Boo also implements mechanism of missing methods (also present in Ruby for example). When a method is not found on object implementing IQuackFu interface, then a special method of this interface is called and the name of missing method along with its parameters are passed. Developer can implement some behavior basing on the method’s name or its parameters. I hope to cover this in more detail in future.
Boo allows creating syntactic macros which you can use to extend language. While implementing a macro, developer has access to object tree representation of code (AST – abstract syntax tree) and is able to modify this tree. This opens a lot of possibilities. For examle Ayende Rahien uses this feature to create DSLs. Also you can build your own compiler steps – an example is to check the naming convenstions during compilation or automatically include assembly references for used types.

Hopefully I’ll be able to write more on Boo in future. For now I encourage you to explore it on your own.

How to get property name from lambda expression

2008-12-04T20:45:00.001+01:00

Just a quick tip here. If you need a property name for reflection or any other purpose, the simplest solution is to pass it as a parameter of your method. This unfortunately has a serious disadvantage of not being influenced by automatic refactoring. Alternate way is to take advantage of lambda expressions and expression trees:

public void UseProperty<T, K>(Expression<Func<T, K>> propertySelector)
{
    if (propertySelector.Body.NodeType != ExpressionType.MemberAccess)
    {
        throw new ArgumentException(
            "A property selector expression has an incorrect format");
    }

    MemberExpression memberAccess = propertySelector.Body as MemberExpression;

    if (memberAccess.Member.MemberType != MemberTypes.Property)
    {
        throw new ArgumentException("A selected member is not a property");
    }

    string property = memberAccess.Member.Name;
    Console.WriteLine(property);
}

The usage:

var myClass = new MyClass();
myClass.UseProperty((Exception ex) => ex.Message);

Notice there is no need to explicitly provide parameter types as they are inferred from the lambda expression, but you have to explicitly type the lambda expression. If your class operates on a specific type, you can make it generic and move type declaration to the point, where an instance is created. This is useful if you have more methods using lambda to retrieve property name.

public class MyClass<T>
{
    public void UseProperty<K>(Expression<Func<T, K>> propertySelector)
    {
        // same as previously
    }
}

The usage:

var myClass = new MyClass<Exception>();
myClass.UseProperty(ex => ex.Message);

Hope this helps.

Some thoughts on LLBLGen

2008-12-03T00:21:00.001+01:00

While there are many popular ORMs, the LLBLGen doesn’t get too much buzz on the blogs. Possibly because it is a commercial product and people rather use free tools if available. I think it is one of the first .NET ORMs on the market and it surely is a mature product. I had an opportunity to work a little with LLBLGen so I’d like to sum up some of my observations.

An overall impression I got was that LLBLGen is very ADO.NET-like. There’s no equivalent of session (or object context) unless explicitly specified, so the Identity Map pattern is not used by default, which is OK in most scenarios (and more intuitive in my opinion). Instead of session, you use a DataAccessAdapter class, which behaves like a smart connection to a database. There’s the EntityCollection class which is basically something like a DataSet – a data container, that tracks changes . There are typed lists which allow to create a custom view over a set of related entities.

The pros of LLBLGen are:

Allows you to just get things done quickly. Maybe not in the best style, but still.

Has a nice designer and a very sophisticated text templating engine with ORM oriented syntax. Take a look at this:

/// <summary> 
/// CTor which initializes the DTO with values from its corresponding entity 
/// </summary> 
/// <param name="entityInstance">The entity instance which holds the values for this DTO</param> 
public <[CurrentEntityName]>DTO(<[CurrentEntityName]>Entity entityInstance)<[ If IsSubType ]> 
    : base(entityInstance)<[ EndIf]> 
{ 
    <[Foreach EntityField CrLf]> _<[CaseCamel EntityFieldName]> = 
        entityInstance.<[EntityFieldName]>;<[NextForeach]> 
}

I consider this to be a nice example of DSL.

Has an advanced LINQ support together with extensions allowing you to define prefetch paths on IQueryable<>
Extensive predicate mechanism (Specification pattern) when not using LINQ
Supports both adapter mode (the usual with entities loaded and saved using external object, adapter) and self-serviced mode(Active record pattern)
Has some advanced features including auditing, authorization and own dependency injection, but presumably you will use specialized libraries for this aspects anyway.

While the cons are:

Entities generated by default templates are cluttered with ORM related code and really heavyweight.

The syntax imposed by runtime libraries is clumsy and awkward. An example:

EntityCollection customers = new EntityCollection(new CustomerEntityFactory());
DataAccessAdapter adapter = new DataAccessAdapter();
adapter.FetchEntityCollection(customers, null);
customers.Sort((int)CustomerFieldIndex.CompanyName, ListSortDirection.Descending);

Fortunately this is remedied to some extent by usage of LINQ.

Does not support lazy loading.
While prefetching related entities, LLBLGen generates separate SELECT statement for each entity type instead of performing a join.
While having a very nice templating engine and template designer, there is virtually no repository for community created templates. Sure, there is a forum section, but it is unstructured and you can browse posts from last year only!

Still, overall impression is good. I hope to get some comments on this :)

Using MetaLinq to simulate NHibernate’s DetachedCriteria in Entity Framework

2008-11-28T00:28:00.002+01:00

NHibernate’s DetachedCriteria allow you to specify a query filter outside session context and then pass it to code that actually uses a session in order to retrieve data. What’s more, DetachedCriteria is serializable, so you could imagine a scenario in which a client forms criteria and passes it to a service in order to query data.

Entity Framework has no such thing as DetachedCriteria, but you can use LINQ and expression trees in similar fashion. The only problem is that the expression trees are immutable and nonserializable. There is however a project called MetaLinq that copies an expression tree’s structure into mutable clone (built from custom set of classes that mirror the System.Linq.Expressions namespace) and enables you to both edit the tree and serialize it.

So I performed a little experiment. Following is the code of WCF service operation.

public Customer[] GetCustomersBy(EditableLambdaExpression filter)
{
    var filterLambda = filter.ToExpression() as LambdaExpression;
    var filterDelegate = filterLambda.Compile() as Func<Customer, bool>;

    using (var context = new NorthwindContext())
    {
        return context.CustomerSet.Where(filterDelegate).ToArray();
    }
}

The method receives a filter in a parameter. It is of type EditableLambdaExpression, which is one of MetaLinq’s expression types. By calling it’s ToExpression method you get the regular expression tree. Then there’s a compilation resulting in a delagate of type Func<Customer,bool> (it’s a quiet assumption) that is finally passed to the Where method of ObjectContext. Quite straightforward.

So is the client code:

using (Service1Client client = new Service1Client())
{
    var filter = EditableExpression.CreateEditableExpression(
        (Customer c) => c.CustomerID == "ALFKI") as EditableLambdaExpression;

    var customers = client.GetCustomersBy(filter);
}

The filter expressed as a lambda has to be converted to MetaLinq’s EditableLambdaExpression. Then it can be serialized and passed to the service. And this works just fine. What also works is:

var filter = EditableExpression.CreateEditableExpression(
    (Customer c) => c.CustomerID.StartsWith("A")) as EditableLambdaExpression;

What does not work is for example:

var filter = EditableExpression.CreateEditableExpression(
    (Customer c) => c.Orders.Any(o => o.Freight > 100.0M)) 
    as EditableLambdaExpression;

It turns out MetaLinq (as for now) is having trouble deserializing expressions with generic and extension methods.

This was an interesting experiment and now there is time for:

The criticism

I wouldn’t recommend using this idea in any production environment. It has several disadvantages:

MetaLinq itself is just an experiment, it definitely is not finished. There potentially are some hidden problems and you can’t express more advanced scenarios (unless of course you fix the code yourself :).
Building mutable expressions, serializing them, then deserializing and then compiling again imposes a performance penalty.
You can’t enforce the lambda passed to your service to be Func<Customer, bool>. You always has to check it and throw when it’s not.
And last but not least – it threats security of your service. What you’re doing is just executing some code that came from outside the service’s boundaries. This could allow someone to perform, let’s say, LINQ injection :)

[Book] Application Architecture Guide 2.0

2008-11-24T00:17:00.001+01:00

I just skimmed through Application Architecture Guide 2.0 by Patterns & Practices (beta). So I think it's a nice checklist for your project, whatever that is. The guide lists patterns and consideration points categorized both by layers and application types. Yet, if your not already familiar with those concepts, you might have trouble learning them from that book, because there are only short explanations without any examples and there are no specific technologies listed. It's a good starting point for your research. I myself was a little disappointed, because I expected more insight into advanced topics.

Should transaction management be implemented as an aspect?

2008-11-20T19:11:00.001+01:00

As usually the right answer is: that depends :) But let's think think about this a little.

When considering aspect oriented programming, there are a lot of very good examples of cross-cutting concerns whose implementation as an aspect is a very good idea (logging, exception handling, etc). But is transaction management such? Suppose there is situation like following:

(the service here is not to be understood as a part of distributed system, but rather as fragment of transaction script business logic)

Now, what are the pros of opening transaction in an advice (a handler) for DoSomethingTransactional():

You don't have to do it manually
Each new method and class matching a pointcut (or matching rules in Enterprise Library terminology) gets a "free" transaction without any effort
Transaction creation code is centralized and easy to modify
When the Service calls another service, say Service2, the operation should be implicitly enlisted to transaction that was created for Service1

Now the cons:

Transactions are a very serious matter. If for some reason your pointcut contains an error, some operations can execute without opening database transaction, which leads to violated data integrity in case of an unexpected error. This can cause some serious damage and is not easily detectable. I suppose you could roll out some unit tests for that.
Your transactions sometimes cover too much. Suppose your DoSomethingTransactional1 method does two things: updates a database record and then send a notification email. If sending an email fails, whole operation is rolled back. In some cases this is not a desirable behavior. A notification might be significantly less important than the business operation which was successful. This business operation might be your holiday trip reservation when there's just one place left :) Surely you could explicitly handle this error, but your code has to be aware of that specific error.

So you may consider using explicit transactions instead of AOP. Also remember, that .NET's TransactionScope is a nice way of determining what portion of code should run in a transaction.

A digression on UML modeling

2008-11-15T15:46:00.001+01:00

What do you need UML diagrams for?

UML is a language and a language is used for communication. This is also true for programming languages because through them you are communicating a compiler what you want to achieve. So anyway, the communication is the goal.

Who do you communicate with? This depends, really. For one, you may be trying to describe your application's structure in technical documentation. This means you're communicating with people that will require knowledge on your application in the future. Then there are design meetings. You can use UML to express your thoughts more clearly than in regular spoken language. Once other people get your idea, they can discuss, suggest changes and modify your diagrams.

How accurate should the diagrams be? This also depends. While writing a science book you would want to be a little more accurate than when making notes for your next grocery shopping. But this doesn't make your shopping list worse than a science book in terms of communication. It's accurate ENOUGH.

What I'm driving at is that you only need to make your UML diagrams (or any other diagrams) as good as it gets to communicate your ideas to your audience. In my opinion, once this is done, diagrams can be disposed. Well if you need them for future reference, that's fine, but treat them as a way of communication. As analogy I might ask you if you keep transcript of every meeting.

Finally, the language itself. Do you really need UML? Think twice before you use it, because maybe a few boxes connected with arrows will tell your audience more than an elaborate UML diagram. Not everyone can tell the difference between solid and dashed line, but that doesn't make them less able to comprehend your ideas.

nServiceBus

2008-11-11T16:54:00.001+01:00

Last time I wrote about asynchronous communication as a way to improve availability and capacity of a distributed system. To dig further on this topic I played a little with the nServiceBus project by Udi Dahan. This is a communications framework built upon a concept of a bus. It promotes asynchronous communication model both in one way and publish/subscribe models.

As the documentation is close to none I had to look into source code to get a grasp on what's going on in there. There are some samples provided, in order to run which you need to install Microsoft Message Queuing component. Samples demonstrate basic messaging, publish/subscribe model and the concept of saga.

The saga is a set of related messages. This takes asynchronous messaging a step further allowing you to deal with workflows. Suppose you have to make two calls to external system: first one to get a order status, then depending on this status you would either cancel the order or change it's priority. In traditional approach you would have a method similar to following:

orderStatus = externalSystemPoxy.GetOrderStatus(orderId) if (orderStatus == Status.Postponed) operationStatus = externalSystemProxy.CancelOrder(orderId) else operationStatus = externalSystemProxy.SetLowPriority(orderId) // notify user about operationStatus

EDIT: As Udi pointed out, I should explicitly say, that the code above is time-bound. Execution of the GetOrderStatus method is blocking, which implies that if the external system is too busy to process the request (or it's not available for other reason), whole operation fails. Also if the response is taking long, a thread on client side is waiting idle instead of serving another request. If client crashes during execution of this "workflow", all progress is lost. The reason to use the asynchronous model here is to remove that time dependency.

When migrating this code to asynchronous model the situation gets complicated because you can't tell when the response to GetOrderStatus() will arrive. So you need to have an event handler for message arrival event, but you also need to maintain a state of the particular workflow instance (in this case the orderId value). A saga is a nice way to encapsulate such workflow. It contains a state and also handles for any of messages involved in the workflow. So with saga you would:

Define the class deriving from ISaga<OrderStatusResponse> and ISaga<OperationStatusResponse>
Define the state inside the class, ex. public property OrderId
Define message handlers for OrderStatusResponse and OperationStatusResponse

The saga itself is persisted in a database using implementation of ISagaPersister, I think you need to roll out your own. Speaking of which, nServiceBus provides some extensibility points like for example the transport. There are two provided in the project: MSMQ and HTTP and you can add your own by implementing ITransport.

So to sum up, nServiceBus is worth looking at when dealing with asynchronous communication. Hope it gets better documentation soon.