Outbox Design Pattern

Yazının Türkçe versiyonuna bu link aracılığı ile ulaşabilirsiniz.

The applications we develop may need to communicate with many external components. Operations such as taking service over web, communicating with the messaging system, interacting with the database can be listed as examples. It may not always be possible to use multiple external components together in a single transaction. In case we develop by ignoring the integrity, the reliability of our application decreases. The subject of this article will be how we can use different components transactional through outbox design pattern.

Let’s imagine that we have built an account management system. In this project, we send a verification mail to the e-mail address given by the user to ensure that the e-mail adress belongs to the user. After storing a user into the database, we can try to send e-mail via mail service to achieve verification requirement.

We see that there are many problems that can occur because the transaction integrity is not provided. For example; our application may close unexpectedly after saving user into the database; after saving to the database, the machine on which our application runs may experience network problems and cannot reach the mail service; another case is that the mail service may not be working at that moment. In all these scenarios, we stored the user info into the db, but we were not able to carry out all the steps we had to carry out. In other words, we added the user to the db but could not complete the user registration process.

There are similar problems in distributed architectures such as microservice. Raising the events that occur within the domain to the outside world is mostly done via the messaging system. If the occurrence of the event within the domain and the transmission of the event to the messaging system cannot be done by maintaining the transactional integrity, the system will start to experience disruptions.

At this point, the outbox design pattern emerges to save us. In fact, it works with a very simple logic. Solution is that the operation to be carried out after making changes on the database is stored into the database as a task. In other words, in the same transaction, we execute the action we want to do in the database and also leave a mark for the action we will carry out after the db operation. We scan the tasks periodically with a background application during implemantation of outbox design pattern. If a new task is found, we process it through the background service and mark it as completed. This is the summary of the outbox design pattern.

If we consider our example about the messaging system; while storing a user into the database, we will also save the UserCreated message to the database in the same transaction. In order to send this message to the messaging system, we will add a task to the database while recording the message. Our background service checks the tables periodically and contacts the messaging system for unprocessed tasks.

In this method, transmission of the message to the messaging system is guaranteed at least once. If the task is taken and processed but cannot be marked as completed in the database, this task will be executed again. When using the outbox design pattern, we should not forget about considering this situation.

Let’s create an example of how we can solve the problems we may encounter as stated above via the library called MessageStorage.

So let’s start by creating an application with .NetCore. After this step, let’s download the MessageStorage.Db.MsSql, MessageStorage.Db.MsSql.DI.Extensions and MessageStorage.AspNetCore packages via NuGet and proceed to the coding section.

1- To establish the database connection, let’s continue by creating an instance of DbRepositoryConfiguration class.

2- After this step, let’s use the line below to create database tables.

3- Let’s add our background tasks, which we name Handler, to our system as dependencies.

4- Let’s add dependencies such as the interface that we will use to add messages, the background service that will carry out the tasks we recorded, and the monitor object that will enable us to access the number of tasks registered in the system.

With these 4 steps, we have completed the adjustments in our Startup class. After making our adjustments, we can define the messages and the tasks we want to create for these messages. We can start by creating message types called AccountEvent and AccountCreatedEvent.

We will define the tasks to be carried out for these messages through an instance of Handler. More than one Handler may process a message. For example; since an object of the AccountCreatedEvent type also matches the AccountEvent type, both AccountCreatedEventHandler and AccountEventHandler will work for the same message.

As can be seen in the example below, when saving an account via EntityFramework, a message stating that membership is created will also be recorded in the database. With this message recording, both tasks will be recorded in the database.

The output will be as follows.

You can use the “Issues” section on GitHub for your requests and suggestions. You can also send a pull-request for open issues :)