How to choose prefetch count value for RabbitMQ

In this article I am going to present few rules I use when selecting prefetch count value for RabbitMQ. Those rules are based on the experience I gained when working with RabbitMQ. You should treat them as a guidance and starting point, remembering that each application, network and queue topology is different. Let’s start with quick explanation what prefetch count is.

Prefetchcount

RabbitMQ allows consumers to specify the size of the limit of unacknowledged messages on a queue basis. This value is used to specify how many messages is send to the consumer and cached by RabbitMQ client library. Once the buffer is full the RabbitMQ will wait with delivering new messages to that consumer until it sends ACKs / NACKs. The main purpose for pre fetching messages is to optimise message delivering. The ideal situation is where there is a message delivered and waiting for the processor to be consumed.
Of course, caching messages on the consumer side has repercussions. All pre-fetched messages are removed from the queue and become invisible to other processors. Hence setting prefetch count to a high value will skew message distribution between different consumers.

Consumer types

Below are rules for selecting prefetch count depending on the consumer type.

Fast, single consumer

If there is only one consumer processing messages quickly you definitely want all messages to be pre-fetched. The optimum value will probably be something higher than 20, but setting it to anything above 50 will rarely delivery more value. Remember that pre-fetched messages are cached on the client side consuming its resources. High value will also restrict your ability to peek messages.

Multiple fast consumers

When you have multiple consumers which are processing messages fast you need to find value which keeps messages waiting for consumers as well as allows for balanced message delivery between consumers. Low value will penalise consumers as they need to wait for messages. High value will damage load balancing. I usually use anything between 20 and 30.

Slow consumers

For slow consumers message delivery time is negligible. My advice is to set prefetch count to 1 and let RabbitMQ load balance message between consumers.

Application crash and buffered messages

Let’s see what happens to buffered messages when the application crashes. Because RabbitMQ automatically puts all not ACKed messages back to the queue, if the application crashes then all cached messages will simply go back to the top of the queue.

Setting prefetch count

To set ‘prefetch’ count in RabbitMQ you have to execute basic.qos command1.

Setting prefetch count in EasyNetQ

You can set prefetch count value in EasyNetQ by including it in the bus’ connection string. Default value used by EasyNetQ is 50.
Below example sets prefetch count to 30:

var bus = RabbitHutch.CreateBus("host=localhost;prefetchcount=30");

Resources

You may find more information about prefetch count in following articles:

Some queuing theory: throughput, latency and bandwidth
RabbitMQ Performance Measurements, part 2


  1. For example of usage check Work Queues tutorial. 
Advertisements

How to process messages in parallel using EasyNetQ

The IBus interface from EasyNetQ framework has SubscribeAsync() method which allows easily to span message handling between different threads. It uses standard TPL Tasks for delegating handler execution. Depending on the application, you may create Tasks with LongRunning flag, which provides a hint to the scheduler that additional thread may be required1 and it should avoid using a ThreadPool2. Below example shows how to register parallel consumer:
Continue reading

How to allow EasyNetQ finish processing message on application stop

This post was written as an answer to one of the questions on EasyNetQ user group.

The main principle of EasyNetQ bus is simplicity. It greatly abstracts nuances of communicating with RabbitMQ giving developers time to concentrate on writing the core application. The framework is great and makes processing messages really simple. But there are few scenarios when this simplicity becomes a small pain. One of those times is when you want to gracefully stop the application allowing it to finish processing current message.

This post is rather lengthy but I wanted to provide good explanation.
Continue reading

EasyNetQ – methods for delivering messages

When designing your system you have few options on how the messages will be delivered to your consumers:

  1. messages are delivered in round-robin fashion – this is a great option for distributed, load balanced processing;
  2. messages are delivered to all consumers – this is the option to use if there are different processors for given message;
  3. hybrid solution where some consumers are getting messages in round robin fashion and other are getting all messages;

Round-robin delivery

In the round-robin delivery, the messages are automatically distributed between all consumers. By manipulating prefetchcount setting you can easily achieve load balancing. This scenario is great when there are many consumers doing the same type of processing. The example will be an order fulfilling system with few processors receiving orders. In the heavy load times you can spin extra consumers to cope with additional work, and the system will automatically deliver next message to free consumer using round-robin fashion.

When subscribing to the bus you have to specify the same subscriptionId for all consumers. You also need to set prefetchCount1 to small arbitrary value. This could be 1 or some other small value if buffering is necessary.

The above example shows 3 consumers subscribed to new order message. One of the consumers (no. 2) is processing messages slow, and due to load balancing will receive every 7th message (whenever it is ready to consume next message). Other consumers are processing messages with the same speed and are receiving them in round robin fashion.

All consumers delivery

This delivery mode will deliver every message to each subscribed consumer. This is useful in scenarios where there are many different processors interested in given message. For example, in the order fullfilment system, there may be one processor for verifying stock availability, another for billing and another for auditing and logging. When client places an order, new message is generated and copy of it is delivered to each processor. This allows for parallel processing, as each consumer may run on separate thread or machine. It also makes system open for extension – each new processor just registers interest in the message by subscribing to the bus.

When subscribing to the bus, each processor is using a different subscriptionId. When deciding on prefetchcount1 you may go with a higher value to get better performance.
As long as consumer code for each processor runs on different thread or different machine, the processing will be in parallel. Each consumer receives messages independently whenever it is ready.
The speed of processing messages by each processor doesn’t matter to other processors2. Messages are simply buffered in the queue.
Extending the system is simple. You create new consumer which subscribes to the bus with new subscriptionId and run it. During first run, EasyNetQ automatically creates and binds the queue, and messages get delivered to it.
When designing the system you can create conditional consumer. This kind of consumer will process messages only when some condition is met, such as: order value is over certain amount. When the condition isn’t met you return from the consuming method and the message is simply removed from the queue.

Hybrid solution

There is nothing to prevent you from using both delivery methods for the same message type. You can scale out each processor type by adding extra instances of it, so if you find for example that stock processor and billing processor are getting behind then scale those processors out.
If the system load varies than you can implement auto scaling – a system which will monitor queue sizes and automatically increase / decrease number of consumers to cope with the workload.



  1. prefetchcount is set when creating a bus. It’s value determines number of messages being cached by consumer for faster delivery. 
  2. There may be some performance hit with delivering a message when one queue gets big backlog of messages and RabbitMQ has to spend resources on persisting those. 

Working with REST services in PowerShell

With PowerShell 3.0 comes a wonderful Invoke-RestMethod cmdlet which much simplifies using REST services. The cmdlet handles all communication as well as data serialisation/deserialisation. Below gist shows how easy it is to make a call to get some data from the REST service:

Depending on the result type (atom, JSON, XML), the cmdlet will appropriately deserialise data and return richly object (or collection of objects).

Often access to the service requires authorization. In that case all you need to do is pass and instance of the PSCredential object. Depending on the case you can either use Get-Credential cmdlet which will prompt user for credentials, or create new instance of PSCredential object. Below example, taken from RabbitMQTools module (at the time of writing this module is in very early stage), shows how to list all Virtual Hosts registered with RabbitMQ server, using credentials of guest user:

To create new entity in REST service you use PUT method passing extra data in the body. For example, to create new Virtual Host in RabbitMQ with tracing enabled, you will call method PUT passing hosts name in the Url. Notice that variable $body is converted to JSON object and the content type on REST call is specified as JSON. as well:

Deleting entity means calling REST service and using DELETE method:

Summary

As you can see working with REST Api services in PowerShell is very easy. The Invoke-RestMethod cmdlet fully abstracts all communication and serialisation from the developer. The result comes as rich object and can be piped to other cmdlets like select or where.

I’d like to thank Jason Helmick and Jeffrey Snover for great Getting Started with PowerShell 3.0 Jump Start and Advanced Tools & Scripting with PowerShell 3.0 Jump Start modules on Microsoft Virtual Academy which helped me getting started with PowerShell.

And to find more about creating PowerShell modules come back to this blog, where I plan to write about my experiences at creating RabbitMQTools – the PowerShell module for administering RabbitMQ servers.