• Insights

Kentico & RabbitMQ Integration: Part 2 – Inbound Integration

Senior developer Lee Conlin walks us through part 2 of his exploration into RabbitMQ integration to receive data into Kentico from an external source.

In my previous article I talked about how to send data out of Kentico using RabbitMQ as the message queuing server.

In this article I’m going to look at how you can receive data into Kentico from an external source via a RabbitMQ message queuing server.

Setup

I’ll assume for the purposes of this article that you have already followed the “Setup” section in Part 1 and that the inbound messages from the third-party system that you’ll be receiving will follow the same structure and model as those that we sent in Part 1.


Polling vs subscription

There are two methods by which you can receive messages from a RabbitMQ server: polling and subscription. 


Polling

message driven development - polling | Rabbit MQ and Kerntico integration | Distinction Thoughts


Polling involves your application making a request to the RabbitMQ server to retrieve each message (or no message if none are in the queue). This can be a good way to ensure that the messages are retrieved if you need to ensure that your application controls how and when the messages are retrieved.


Subscription

message driven development - subscription | Rabbit MQ and Kerntico integration | Distinction Thoughts

With subscription, your application simply opens a connection to RabbitMQ and subscribes to one or more queues. The RabbitMQ server will then push new messages down to your application as and when they arrive in the queue and your application is ready to receive them.


Which is better?

The answer is that it really depends on your situation.

If you need to control the exact timing of the message processing, then the polling method might work better for you – for example, if you only wanted to process the information at certain times of day.

If you need to get the messages processed as quickly as possible then the subscription method might be a better fit since your workers will get the messages as soon as they are available.

With either method, it is possible to have multiple workers to consume the messages from the queue, allowing for far more scalability.

Personally, I prefer the subscription method. It takes the management of connections and timing out of your hands and lets RabbitMQ handle it, but this is only a personal preference based on my experience with RabbitMQ and the projects I have used it in. I’ll describe both methods below so that you can use the method that best suits your situation.


A quick note on web farms

If you use Kentico’s web farms feature, you can process messages on each server in parallel – improving performance.

It should be noted though that you may need to also create web farm tasks to ensure caches are cleared or actions are taken on all web farm servers depending on the work being done.


Configure polling

To use polling, you will need some way of running your polling code on a schedule. Fortunately, Kentico has a great scheduled task system built-in.

Create a new scheduled task class as per the Kentico documentation. Configure your task to execute every 1 minute (the shortest time interval at which they can run by default).

Next (because you will now need access to the RabbitMQ channel object in both your module class and this new scheduled task) you will need to abstract your setup code out of your module’s OnInit method into a static helper method or singleton service that returns the _channel variable. It’s important that everywhere that you access the queue(s) that you use that they are configured in the same way – abstracting the code that performs the setup allows us to achieve this.

Then, within the Execute() method of your custom task you will need to add some code to request one or more messages from the queue and process them.

Receiving a message is as simple as this:

var message = _channel.BasicGet(_queueName, false);

The “Body” property on the message will contain a byte-array which is the text contents of the message. If you were receiving one of the messages sent in part 1, this would be a JSON object which you could now deserialise as follows:

var messageModel = 
JsonConvert.DeserializeObject<UserMessageModel<InsertUserModel>>(Encoding.UTF8.GetString(message.Body));

Now that you have the model decoded you can do anything you need to do with it. You could also wrap the above code in a loop to process multiple messages in each pass or use the scheduled task tips & tricks I talked about in a previous article to enhance the processing capabilities of the task.

Finally, once you have completed the processing for each message you must send an acknowledgement back to the RabbitMQ server to let it know that it can release that message from the queue. Failure to do this may result in the message being delivered to you multiple times:

_channel.BasicAck(message.DeliveryTag, false);

It is also possible to return a “not acknowledged” response back to the server as well, to tell the server that something went wrong and that the message was not processed.

_channel.BasicNack(message.DeliveryTag, false, requeue: true);

The “requeue” parameter tells RabbitMQ whether the message should be requeued for another try. In places where you know the failure is permanent you should set this to false.

Configure subscription

To use subscription, you will need to do things a little differently. 

Instead of a scheduled task, you can register a consumer within your module’s OnInit method:

var consumer = new EventingBasicConsumer(_channel);

Next, you need to register an event handler for the “Received” event:

consumer.Received += ConsumerOnReceived;

Then you’ll need to write the method to handle the messages. This method will be called for each individual message that RabbitMQ delivers to your application:

private void ConsumerOnReceived(object sender, BasicDeliverEventArgs message)
{
    var messageModel = JsonConvert.DeserializeObject<UserMessageModel<InsertUserModel>>(Encoding.UTF8.GetString(message.Body));

    // Handle your message here

    // Then acknowledge the message
    _channel.BasicAck(message.DeliveryTag, false);
}

Finally, you need to register the consumer with the channel for the relevant queue:

_channel.BasicConsume("queueName", autoAck: false, consumer: consumer);

Now, RabbitMQ will send messages directly to your application whenever there are messages in the queue. By default, it will initially deliver 10, then as each message is acknowledged it will deliver one more to replace it.

It’s worth noting that you can have your consumer automatically acknowledge the messages it receives by setting the autoAck parameter to true, however doing so can present a risk that messages that fail to process correctly are removed from the queue and lost. I find that it’s always better to manually acknowledge the messages once the process has successfully completed.


Wrapping up

Now that you know how to send messages to RabbitMQ and how to receive them you’re all set to create new integrations that are more scalable and can handle much larger volumes of data as well as being able to communicate out to potentially unknown external systems.

In the next article I’ll take the scalability aspect of this process further and detail an architecture that can scale in multiple ways to more easily accommodate surges in usage.

At Distinction, we are Kentico specialists and certified gold partners. If you are interested in working with us, get in touch today to see how we can help your business.

Author:Lee Conlin

More insights