How to choose prefetch count value for RabbitMQ

In this article I am going to present few rules I use when selecting prefetch count value for RabbitMQ. Those rules are based on the experience I gained when working with RabbitMQ. You should treat them as a guidance and starting point, remembering that each application, network and queue topology is different. Let’s start with quick explanation what prefetch count is.

Prefetchcount

RabbitMQ allows consumers to specify the size of the limit of unacknowledged messages on a queue basis. This value is used to specify how many messages is send to the consumer and cached by RabbitMQ client library. Once the buffer is full the RabbitMQ will wait with delivering new messages to that consumer until it sends ACKs / NACKs. The main purpose for pre fetching messages is to optimise message delivering. The ideal situation is where there is a message delivered and waiting for the processor to be consumed.
Of course, caching messages on the consumer side has repercussions. All pre-fetched messages are removed from the queue and become invisible to other processors. Hence setting prefetch count to a high value will skew message distribution between different consumers.

Consumer types

Below are rules for selecting prefetch count depending on the consumer type.

Fast, single consumer

If there is only one consumer processing messages quickly you definitely want all messages to be pre-fetched. The optimum value will probably be something higher than 20, but setting it to anything above 50 will rarely delivery more value. Remember that pre-fetched messages are cached on the client side consuming its resources. High value will also restrict your ability to peek messages.

Multiple fast consumers

When you have multiple consumers which are processing messages fast you need to find value which keeps messages waiting for consumers as well as allows for balanced message delivery between consumers. Low value will penalise consumers as they need to wait for messages. High value will damage load balancing. I usually use anything between 20 and 30.

Slow consumers

For slow consumers message delivery time is negligible. My advice is to set prefetch count to 1 and let RabbitMQ load balance message between consumers.

Application crash and buffered messages

Let’s see what happens to buffered messages when the application crashes. Because RabbitMQ automatically puts all not ACKed messages back to the queue, if the application crashes then all cached messages will simply go back to the top of the queue.

Setting prefetch count

To set ‘prefetch’ count in RabbitMQ you have to execute basic.qos command1.

Setting prefetch count in EasyNetQ

You can set prefetch count value in EasyNetQ by including it in the bus’ connection string. Default value used by EasyNetQ is 50.
Below example sets prefetch count to 30:

var bus = RabbitHutch.CreateBus("host=localhost;prefetchcount=30");

Resources

You may find more information about prefetch count in following articles:

Some queuing theory: throughput, latency and bandwidth
RabbitMQ Performance Measurements, part 2


  1. For example of usage check Work Queues tutorial. 
Advertisements

How to process messages in parallel using EasyNetQ

The IBus interface from EasyNetQ framework has SubscribeAsync() method which allows easily to span message handling between different threads. It uses standard TPL Tasks for delegating handler execution. Depending on the application, you may create Tasks with LongRunning flag, which provides a hint to the scheduler that additional thread may be required1 and it should avoid using a ThreadPool2. Below example shows how to register parallel consumer:
Continue reading

Tell EasyNetQ bus what Queue name to use for your messages

When you consume messages from EasyNetQ bus you do not need to worry about naming and creating queues or exchanges. That’s done for you. The framework will use a pattern when generating name for the queue which is based on strongly typed name of the message type and consumer’s subscription id. Similar happens when generating exchange name. But is there anything you can do to change this behaviour?
Continue reading

How to allow EasyNetQ finish processing message on application stop

This post was written as an answer to one of the questions on EasyNetQ user group.

The main principle of EasyNetQ bus is simplicity. It greatly abstracts nuances of communicating with RabbitMQ giving developers time to concentrate on writing the core application. The framework is great and makes processing messages really simple. But there are few scenarios when this simplicity becomes a small pain. One of those times is when you want to gracefully stop the application allowing it to finish processing current message.

This post is rather lengthy but I wanted to provide good explanation.
Continue reading

Using complex type messages with EasyNetQ

This is the second post in series about EasyNetQ. In the first post I am explaining how to install RabbitMQ and write simple application which publishes and consumes string messages.

Complex type message

Usually you will need to send more complex types than string. Fortunately it is no harder than sending string messages. All you need to do is to create library project which will contain your message types and share it between publishers and consumers.

Load the example application you made in the first post and add new Class Library project to the solution named Messages. Add new class PersonName to that project which will represent persons name:

Next, add new class CreateAccountMessage to represent a message which is raised when user created new account:

Consuming complex type messages

There really is no difference in consuming complex type message from simple type message. All dirty work related to routing, serialisation and deserialisation is handled by EasyNetQ. All you have to do is to add reference to Messages project from your ConsumerApplication project and change Subscribe registration to

bus.Subscribe<CreateAccountMessage>("consumer1", message =>
    Console.WriteLine("Creating account for {0} {1}, age {2}.", message.Name.FirstName, message.Name.LastName, message.Age)
);

Publishing complex type messages

Again, as you would expect, nothing complicated here. Just add reference to Messages project from the PublisherApplication and change the call to Publish method to:

bus.Publish(new CreateAccountMessage("Joe", "Smith", 26));

Now run the ConsumerApplication by right clicking project name in Solution Explorer and selecting Debug > Start new instance. Next run the PublishedApplication project in the same way and you should see ConsumerApplication receiving the message.

Why do I have to share Messages

As you could see there is a need for sharing project with messages between publisher and consumer project. This is dictated by the way EasyNetQ is handling message delivery. Under the hood, the EasyNetQ is registering exchange and queue in RabbitMQ server using message strongly type names, which in our case will be Messages.CreateAccountMessage:Messages. If the consumer will subscribe to the bus with message type residing in different assembly (such as ConsumerApplication) then it won’t match the type of published message, even though they have the same definition.
Anyway, the requirement for sharing message types is not that bad as it will prevent problems with different versions of messages being published and consumed. It also allows for sharing message’s behaviour, as only data is carried over the message bus.

EasyNetQ with RabbitMQ – easy way to messaging

Few people have asked me already how to start using RabbitMQ in .NET, so I decided to write a short post about it.

Installation

First, visit erlang’s page, download erlang distribution according to your system (either 32 or 64 bit) and install it. If you have 64 bit machine (and you really should have) than get the 64 bit distribution, otherwise erlang process won’t be able to address more than 2GB of memory.

Next download RabbitMQ server and install it. I strongly recommend to install Management Plugin, which comes with RabbitMQ installation. All you need to do is run following command from command prompt (you may have to navigate to RabbitMQ’s sbin folder):

rabbitmq-plugins enable rabbitmq_management

There are detailed instructions on Management Plugin and its installation at http://www.rabbitmq.com/management.html.

Additionally, you may find useful RabbitMQTools – a power shell module for managing RabbitMQ servers.

EasyNetQ

Although pivotal provides a great .NET library for working with RabbitMQ, there is even better solution which will be sufficient in most cases – EasyNetQ. EasyNetQ provides nice abstraction over some low level concepts of RabbitMQ, such as topology registration, messages serialisation or error handling. It also gives an easy interface to work with some popular messaging patters such as Publish / Subscribe or Message Bus.

Hello World!

Lets create our first application. Start new console application project in Visual Studio and name it ConsumerApplication.
Add EasyNetQ nuget package:

Install-Package EasyNetQ

and replace Main.cs code with following:

Now, add new console application project to the solution and call it ProducerApplication. You also have to add EasyNetQ nuget package to this project.
Replace Main.cs code with following:

To run the example, first right click on the ConsumerApplication project and select Debug > Start new instance. This will run the ConsumerApplication. Next, do that same on the PublisherApplication and you should see that in the ConsumerApplication window there is a lot of diagnostics information printed and right at the bottom there is the Hello World! message.

How it works

Under the bonnet EasyNetQ is doing quite a work to make all this happen. When publishing the message, EasyNetQ performs a check to see whether the related Exchange exists in RabbitMQ and if it doesn’t than creates it. On the other hand, call to Subscribe method will verify whether both exchange and queue exists and they are bound together. As you can see in the console windows, EasyNetQ runs quite verbose diagnostics which are priceless when it comes to debugging. If there was a problem with consuming the message, such as exception thrown by consumer code, then it will be automatically moved to error queue for further analysis. You can also rely on EasyNetQ to manage connections to RabbitMQ server so you do not need to worry when the connection was dropped etc.

Summary

As you could see from the above example it is very easy to start with messaging using combination of RabbitMQ and EasyNetQ.
From serialising / de-serialising messages, diagnostics and error handling, managing connection to managing topology – all handled automatically by EasyNetQ so you can concentrate your efforts on making awesome applications. As you will see in the second article, working with complex type messages is no more complicated.