This is a technical post from one of our developers. 

It has been 4 months since I’ve joined TINT as a developer and pushed code to production on my first day. I’m having a lot of fun building features and squashing bugs — and learning new things. I wanted to take the time to talk about a tool that was new to me when I joined TINT: Message Queues.

Message queues allow different parts of a system to communicate and process operations asynchronously. As I learned about them, I thought it would be helpful to have a resource explaining them in more detail and how we use them at TINT.

Why Do We Need Message Queues?

It’s easy to think that message queues are unnecessary and cause a lot of overhead because configuring and setting up a queueing system can be a lot of work. A simple synchronous system will just receive a request from the client, perform an operation (anything from retrieving some data from the server to uploading an image) and return a response. This is not entirely wrong for a small scaled project with low traffic or small operations. It is actually “faster” since its instantly processes the request and returns a response.

But of course here comes the caveat;

• What if your operations took more than 100-200 milliseconds?
• What if you had operations that took seconds to complete and also you were receiving a huge amount of traffic?
• Would instant HTTP request/response method still be the best practice?
• What if you want persistence so that when a method fails, it is queued up until it’s successfully executed?

That’s where the message queue comes into effect. The idea behind the message queue is to connect operation producers with executing methods that’ll process the request and return the result. Once the request has been made, it’ll be inserted in the queue in which will be processed by the executing method. Usually, a cron task periodically checks for pending operations and runs them with given class or method.

Reviewing Message Queues

There are both awesome and not-so-awesome aspects of using message queues:

Strengths

• Persistence

In synchronous systems when an operation fails, if there are no additional components setup for data recovery, all the information is lost. Since the request data is stored on a thread in memory, data recovery is quite difficult. Message queues come with a simple solution for this problem. When we are enqueuing operations, with a quick customization we can ensure their successful execution by persistence (in case of crashing, recover the data and until it’s successfully execute re-enqueued the operation). With most queue libraries the operation will run even under extreme circumstances — such as server restarting or shutting off. Even though queue systems use a database, it is wise to use replicas not to lose queues if anything goes wrong with the master database.

• Management

Queue libraries often provide a clear web interface for you to review important statistics such as:

• Which queues are currently active (multiple queue logic will be explained in detail)
• Which operations are currently being processed
• Which operations have failed, optionally waiting to be re-enqueued
• Hardware statistics and many more

Without queues you need to use additional libraries and tracking tools for the same functionality.

• Separation Logic

You can enqueue operations into different queues. Each queue’s purpose can be defined according to the statistics such as the hardware resource that operation type requires, time interval defined in the cron job, custom persistence strategies etc.

There are many other features that are included with using message queues such as time independent connection, horizontal scalability, ability to prioritize operations and more.

Weaknesses

• Overhead

Sir Tony Hoare, one of the most important computer scientists ever, once said ”Premature optimization is the root of all evil”. Using message queues in simple light-weight applications will result in unnecessary overhead such as going through the libraries to choose the best option, setting up the queues for deployment, configurations and many more. Don’t use message queues if your system does not yet require it.

• Error Handling

If you are using a basic message queue, you’ll need to setup a custom error handling mechanism which usually takes quite a while customize according to your needs. Check out this article from Tuts+ that explains how to handle exceptions properly.

Queues & TINT

Being a developer in a hot startup comes with it’s responsibilities including thinking about optimization constantly. As of now we’re receiving a fair amount of traffic (9 million visitors last month) and already having a diverse user plan structure with update times starting from ~30 minute update times to ~1-5 min update times did make us think and apply queues into our system. In fact it would be accurate to say that queues are now the backbone of our backend structure.

We have a relatively simple, clean and scalable architecture. We’re using RoR on our backend, MySQL for our database and of course we’re using queues. When it comes to Rails, you can find a library/gem for almost anything so among plenty of queue libraries we decided to go with Resque.

Let’s take a deeper look at how everything works step by step;

1. We have predefined cron tasks to check database for feeds that can be enqueued in also predefined time intervals. We adopted an amazing library called whenever (https://github.com/javan/whenever) that helps you express cron tasks in neat, clear ruby syntax.
2. After checking each type of feed (whether it’s Facebook, Twitter, Instagram etc.) we store them in Resque, using Redis in order to store the queues, making it super fast in terms adding and removing items into queues. While we are querying our feeds we take our last processed parameter into consideration (sort accordingly) in order to make sure that each feed gets processed.
3. When Resque needs to process an operation, it pops the operation and processes it with the class provided in our configuration file. Resque uses method naming conventions such as before_perform and perform to execute the operation. After the operation, we save whether it’s successful or an exception has been thrown to the database. This helps us to filter successful events from the failed ones so that we don’t queue failed operations over and over.

So what?

I tried to explain message queues as simply as I can and hopefully got you excited about them as much as I am right now. It’d be smart to remember that each system might require a different strategy and time to time a message queue might be the last component you would like to incorporate to your system. However most of the time, queues are simple communication interfaces which are both versatile and also robust in terms of processing operations.