Lessons learned from running Redis at scale

At Mattermost, we recently were able to scale to 100K users. But we didn’t want to stop there, we wanted to push even further. This post details how that effort led to introducing Redis to our architecture, and the lessons we learned running Redis at scale.

Historically, we have been using a simple in-mem LRU cache for all our caching needs. This has served us well so far, but we started to run into some scaling bottlenecks as we strived to scale further and further.

A major issue that we faced was to do with cache invalidations. Because of our in-mem cache architecture, every app node in a Mattermost cluster would have its own cache. And when something would get invalidated, it would need to happen on all the nodes. This means that every node has to make its own DB call separately to re-populate the cache entry later. This led to a scaling problem where the more you add nodes in a cluster, the worse the problem becomes. Introducing Redis as an external cache solves this because all nodes would reach out to Redis, so only one DB call needs to be made.

Introducing Redis also allowed us to use its powerful pub-sub mechanism to efficiently perform cluster broadcasts at very high scales.

But implementing Redis wasn’t as simple as swapping out the LRU cache with a Redis cache. Naively, I thought it would be. It turned out to be much more tricky. Let’s look at some of the lessons learned during this process.

Choice of library matters 📚

We chose the https://github.com/redis/rueidis library instead of the more popular https://github.com/redis/go-redis.

The primary reason is that rueidis does a lot more by default, whereas go-redis relies on the user to do a lot of the heavy lifting. For example:

• Automatic pipelining of concurrent queries
• Client-side cache

These are some of the things which are supported by the library. Additionally, the library is more tuned towards performance by exposing additional knobs like MaxFlushDelay. Which brings me to my next point.

Batching is not optional 🍞

Redis has a feature called pipelining which allows sending multiple commands in a single call. This is a must-have if you want to use Redis efficiently, and one of the reasons why we chose rueidis instead of go-redis. With the latter, you would need to manually construct the pipeline yourself, but rueidis automatically pipelines your queries if they are coming from separate goroutines.

This reduces not only the round-trip time (RTT) but also the syscall overhead on Redis itself.

For example, let’s consider a syscall overhead of X and the actual userspace work to be Y for a single Redis command. This is purely server side, and we ignore the network latency from the client. So to process n commands, Redis has to do n(X + Y) amount of work. But now if those n commands are pipelined, they will just have to do one syscall, therefore the total work becomes X + nY, which can have a massive impact on reducing CPU usage!

Use client-side caching for best of both worlds 💺

Even if we batch outgoing requests, reaching out to the network has a cost we cannot ignore. Fortunately, Redis allows such a feature. Allocating a small amount of memory to be used as a client-side cache can have a big impact on reducing your application latencies. But care must be taken to decide what is the optimal size of the cache. If it is too small, then there might not be enough of an improvement. And if it’s too large, then you will run into extra network overhead in trying to invalidate the caches. We ended up using 32MiB for each connection. Analyze your application to find out the most used cache objects and come up with a reasonable number.

Use MGET/MDELETE if you have tight loops with GET/DELETE ➰

For some API calls, we were seeing consistently poor performance for no good reason. It took us quite a while to figure this out, but we finally understood it was due to using Redis GETcommands in a tight loop. Our scenario was something like this:

var itemsToQueryFromDB []string
var foundItems []Struct
for _, item := range items {
    // call redis GET(item)
    // if not found
    // append to itemsToQueryFromDB slice
    // else
    // append to foundItems slice
}
// query DB(itemsToQueryFromDB)
// append to foundItems slice
return foundItems

Whereas this worked fine with an in-mem cache, this was disastrously bad with Redis. Because the network latency for each call would keep getting accumulated for every item. And the more items, the longer it took to return!

We fixed it by using MGET to get all elements in a single call, like this:

var itemsToQueryFromDB []string
var foundItems []Struct
// call redis MGET(items)
// find items not found in cache, and prepare itemsToQueryFromDB slice
// query DB(itemsToQueryFromDB)
// append to foundItems slice
return foundItems

You can apply the same logic for DELETEas well.

Be careful of high volumes of SCAN calls 📈

We had some caches, where we needed to delete some objects for which we needed to iterate the entire cache. For example, our session cache would be keyed by the session ID, but to delete all session objects by userID, we need to iterate all the session objects, find which ones have ID equal to our userID and then delete them. It was something like:

var toDelete []string
err := sessionCache.Scan(func(keys []string) error {
	sessions, errs := sessionCache.GetMulti(keys)
	for i, err := range errs {
		// error handling
		if sessions[i].UserId == userID { 	// userID is passed from a higher-level function
			toDelete = append(toDelete, keys[i])
		}
	}
})
// error handling
err = sessionCache.RemoveMulti(toDelete)
// error handling

This generated considerable CPU load on Redis simply because of the sheer volume of SCAN calls. We tried playing around with the batch size, but that didn’t really have much of a difference. Of course, a huge batch size of >1000 would not be ideal.

At the end, we had to keep using our in-mem cache for some of these caches which had this characteristic.

Ensure not to self-send a message while using pub-sub 🖇️

While using pub-sub, remember that Redis will send every message to every subscriber, even if it is the publisher! So, if you have a scenario where a node is both a publisher and a subscriber and you don’t want the publisher node to receive the just-published event, you need to add metadata to the event to indicate which node it’s coming from and then ignore it if it is from the originating node.

Finishing words

This goes without saying, but using any software requires a deep understanding of how it works and how to best integrate it with your product. Every software has its own unique workload and requirements. While yours might be different than ours, I hope you can still have some key takeaways from this exercise.

Redis is a great piece of software and works really well if efficiently used. We are also looking to extend usages of Redis to other areas of the codebase as we explore further.

If you have any comments/thoughts on this, please don’t hesitate to reach out to me on our Community server at https://community.mattermost.com. My user id is @agniva.de-sarker.