• Insights

Kentico & RabbitMQ Integration: Part 1 – Outbound Integration

Preface – Kentico’s integration options

Kentico has an excellent, built-in message queuing integration system called the Integration Bus. The Integration Bus does its job very well and I encourage anyone thinking of an integration between Kentico and some external system to consider it.

It does come with some downsides though. The integration tasks created by the connectors are processed within the scope of Kentico’s scheduled task system. This means that, depending on your configuration, the tasks may be processed or triggered within the context of a web request.

Depending on the nature of your integration, the amount of data that you’re synchronising or the number of records, this can lead to timeouts or to other tasks being blocked. You could avoid this by running the relevant tasks using the Windows Service task runner that Kentico provides but in practice I find that almost no one does this.

Additionally, I find that the handling of errors within integration connectors is uninspiring. In most cases I see, the errors are simply logged to the Kentico event log as “Error and skip” or just “Error” by Kentico and the task left in an “failed” state. While this might be fine if checked regularly and dealt with they often are simply ignored by developers and clients alike at this point. This leads to a lot of noise in the event logs that can obscure other, more meaningful errors.

In this series of articles, I’m going to explore an alternative that can provide a more scalable solution to much higher volumes of data as well as providing better handling and notification of errors in the integration.


Assumptions

In this article I’ll assume that you know how to create custom modules in Kentico and have a custom module in place into which the code in this article can be placed.

To use RabbitMQ as a messaging server you will first need to either set it up on your own infrastructure or purchase a cloud-hosted instance.

I’m not going to cover these processes in this article. I’ll also be assuming that you already have access to a RabbitMQ server. 

If you need information on how to setup your own RabbitMQ server please refer to the RabbitMQ documentation

If you would prefer to purchase a cloud-hosted instance, there are many providers available. A quick Google search will help you find the right one.


Setup

Adding RabbitMQ to your solution

Once you have a RabbitMQ instance you will need to install the .Net client package into your Kentico solution or module class library project. Use the following command in the package manager console to install the package:

Install-Package RabbitMQ.Client

Once installed you need to add some configuration to our custom module class. 

First, add the “using” statement to the top of your class file:

using RabbitMQ.Client; 

Then, in the module classes OnInit() method, add the setup code below:

ConnectionFactory factory = new ConnectionFactory();
factory.Uri = new Uri("amqp://user:pass@hostName:port/vhost");

IConnection conn = factory.CreateConnection();

Hint: You may want to keep your credentials in Kentico Settings keys or in your web.config appsettings.


Next, you need to create a “channel”. In code this is also called a “model”. I will be storing the model object in a private class field so that it can be accessed inside other methods of the module class:

_channel.ExchangeDeclare(exchangeName, ExchangeType.Direct);
_channel.QueueDeclare(queueName, false, false, false, null);
_channel.QueueBind(queueName, exchangeName, routingKey, null);

You can declare multiple queues and bind them all to the same exchange in this way. Be sure to take note of the routingKey, though – you’ll need this to determine which queues messages will be sent to later.


Creating a message model

Before you can send any messages, you will need to define a custom message model. This model will be serialised into the message that you send to RabbitMQ.

I use an interface for the model to allow me to have a base set of properties that must be set, allowing me to determine the type of message on the other end.

public interface IRabbitMqMessageModel<T>
    where T : class
{
    Type ModelType { get; }
    T ModelData { get; set; }
}

You also need an implementation of this interface. I use generics here to reduce code duplication:

public class UserMessageModel<T> : IRabbitMqMessageModel<T>
        where T : class
{
    public Type ModelType => typeof(T);
    public T ModelData { get; set; }
}

Next, you can create models that implement this interface for our three use-cases – Insert, Update and Delete.

public class InsertUserModel
{
    public int UserID { get; set; }
    public string Username { get; set; }
    public string FullName { get; set; }
    public string Email { get; set; }
}
public class UpdateUserModel
{
    public int UserID { get; set; }
    public string Username { get; set; }
    public string FullName { get; set; }
    // No email - can't be changed in this integration
}
public class DeleteUserModel
{
    public int UserID { get; set; }
}

Sending a message

In this example I’ll be using the Kentico UserInfo object. You can use this method with any Kentico Info object that has global events.

First you need to define some global event handlers within the OnInit method:

UserInfo.TYPEINFO.Events.Insert.After += UserInsertOnAfter;
UserInfo.TYPEINFO.Events.Update.After += UserUpdateOnAfter;
UserInfo.TYPEINFO.Events.Delete.After += UserDeleteOnAfter;

Next, create the methods themselves within the module class (insert shown here):

private void UserInsertOnAfter (object sender, ObjectEventArgs e)
{

}

Now you can fill out the integration details.

private void UserInsertOnAfter(object sender, ObjectEventArgs e)
{
    var user = e.Object as UserInfo;
    if (user == null) return; // You might want to log this

    var messageModel = new UserMessageModel<InsertUserModel>();
    messageModel.ModelData = new InsertUserModel
    {
        UserID = user.UserID,
        Email = user.Email,
        FullName = user.FullName,
        Username = user.UserName
    };

    var message = Newtonsoft.Json.JsonConvert.SerializeObject(messageModel);

    byte[] messageBodyBytes = System.Text.Encoding.UTF8.GetBytes(message);
    _channel.BasicPublish(exchangeName, routingKey, null, messageBodyBytes);
}

You should replace “exchangeName” and “routingKey” with the values required for your exchange and queue.


Firing the integration

Now that the code is all done, head into the CMS and create a new user. You should see a message appear in your RabbitMQ instance. 


RabbitMQ - Message driven development - Distinction Thoughts


Congratulations! You’ve published your first message from Kentico to your RabbitMQ instance. 

By following the same process for each type of InfoObject you would like to export to your external system , you can easily send messages out from Kentico.


Handling the message

Your external system then needs to subscribe to the same message queues and handle the data in each message on the other end. 

Alternatively, you could create a worker service (Windows service, Azure WebJob , AWS Lambda, etc) that would subscribe to the queue and deal with pushing the changes into your external system – in this way you can decouple the integration from Kentico and run it in a separate process.

Summary

RabbitMQ opens up the possibility of incredibly resilient integrations. Any messages that fail will get automatically retried. You can also configure your queues to pass messages that have failed too many times into a different queue so that they can be exported or retried in your development environment.

Next time…

In the next article of this series I’ll be looking at how to subscribe to message queues from within Kentico to pick up messages sent by an external system. The same principles can be applied to worker services or external systems that pick up the messages that you send out from Kentico.

Author:Lee Conlin

More insights