Distributed Systems With Node.js: Part 8 Distributed Primitives

Introduction

In this series, I will be working through this book on Distributed Systems using Node.js. The book is quite large and in depth, but in these articles I will be distilling the big picture pieces of the book into bite size tutorials and walk throughs (for those following along in the book, this is actually chapter 9).

In this section, we will learn about distributed primitives in Redis. We will explore how we can use the distributed state stored in Redis to preform our business logic in an atomic manner.

The code for this demo is found in this repo at the commit here

GitHub - mattmacf98/distributed-systems-node-js at 99593e31965e97a43b0f1163c33fdc782c61144d

Redis

Redis is used as our foundational service for elevating our types to a distributed level. Redis is able to manage numbers, strings, sets, lists and maps for us in a distributed manner, it also takes care of some common distributed primitives like incrementing counters, moving an item from one list to another and adding a value to a map if it is not present.

Our book goes through a common problem, an ID generation software where each ID must be unique across all IDs in a distributed setting. We struggle with how to do this with multiple processes and a local file, trying to read and write a count number to the file and manage our locks on the file ourselves. Managing our own locking logic could cause deadlocking issues and adds a significant amount of complexity to our application when adding distributed state (and this is just for a basic counter). Luckily, Redis takes care of all this for us!

We can spin up a docker container to run a Redis image

docker run -it --rm --name distnode-redis -p 6379:6379 redis:6.0.5-alpine

Then we can use it in our application.

#!/usr/bin/env node
const Redis = require('ioredis');
const redis = new Redis('localhost:6379');

(async () => {
    await redis.set('foo', 'bar');
    const result = await redis.get('foo');
    console.log('result:', result);
    redis.quit()
})();

This application will set the key ‘foo’ to have value ‘bar’ in the Redis image and then get the value back from Redis using ‘foo’. We could then also read this value from another process and retrieve the same data back.

Summary

• Redis can store our state in a distributed manner
• Redis has multiple types like strings, numbers and maps

Atomic Actions

An atomic action is an indivisible and irreducible series of database operations such that either all occur, or none occur.

With Redis, we now have distributed state management, meaning all our instances can do singular actions in an atomic safe way. However, most of the logic we may want to do is not a simple key value set or list add, but a more complex set of actions, how can we ensure another process cannot interfere with our Redis state while we are trying to complete our business logic?

We have three options: using atomic commands, using transactions or using Lua scripts.

Atomic Commands

Redis has a few Atomic commands for very common functionality like…

• INCR: this operation gets a key, increments the value and then sets that value back to the key. Normally this would require three separate operations and three separate network calls. We also risk another process touching our state during the process and breaking our business logic. With INCR, all three steps are done atomically without risk of interruption.
• RPOPLPUSH: this operation pops a value from the source list and pushes it onto the destination list, again in a single atomic call.

Transactions

Sometimes, the actions we want to preform do not exist in Redis’s list of special atomic commands. In cases like this, we can see if our business logic can be handled by a transaction.

#!/usr/bin/env node
const Redis = require('ioredis');
const redis = new Redis('localhost:6379');

(async () => {
    const [res_srem, res_hdel] = await redis.multi()
        .srem("employees", "42")
        .hdel("employee-42", "company-id")
        .exec();
    console.log('srem?', !!res_srem[1], 'hdel?', !!res_hdel[1]);
    redis.quit();
})();

By chaining multiple commands together and then executing them in this fashion, we tell Redis to treat this list of operations as atomic. This particular transaction removes employee 42 from the employees set and then removes the company-id key from employee-42’s hash map. This kind of operation does not have a built in atomic Redis command. However, by using transactions we can build our own!

Lua Scripts

Finally, there are some cases where even transactions will not do the job — for instance if we needed to use some state retrieved from a previous command and use it, or branch our logic based on the value we retrieve during our previous logic. In instances like this, we must reach for a Lua script, Lua let’s us send and run business logic directly on the Redis server in an atomic way.

In our example we will be making a waiting lobby for a game, users will join the lobby and when four users are present they will be removed from the lobby and added to a game.

local LOBBY = KEYS[1] -- Set
local GAME = KEYS[2] -- Hash
local USER_ID = ARGV[1] -- String

redis.call('SADD', LOBBY, USER_ID)

if redis.call('SCARD', LOBBY) == 4 then
    local members = table.concat(redis.call('SMEMBERS', LOBBY), ",")
    redis.call('DEL', LOBBY) -- empty LOBBY
    local game_id = redis.sha1hex(members)
    redis.call('HSET', GAME, game_id, members)
    return {game_id, members}
end

return nil

This Lua script will take in three arguments from the caller, the lobby to be added to, the game the lobby is for and the user id that has just entered the lobby.

We first add the user to the lobby, then we check the cardinality of the lobby.

If it is not four we finish.

If it is four, we we create a list of the four lobby members, empty the lobby, and create a new game in our game hashset with a generated game id and the four users we just removed from the lobby.

This makes our lobby atomic, so we won’t get issues like having three lobby members and then having two new users added at the same time (causing SCARD to be 5). Issues like this would occur if we did this logic on our node process instead of directly on Redis.

#!/usr/bin/env node

const Redis = require('ioredis');
const redis = new Redis('localhost:6379');

redis.defineCommand("addUser", {
    numberOfKeys: 2,
    lua: require('fs').readFileSync(__dirname + '/add-user.lua')
});

const LOBBY = 'lobby', GAME = 'game';

(async () => {
    console.log(await redis.addUser(LOBBY, GAME, 'alice'));
    console.log(await redis.addUser(LOBBY, GAME, 'bob'));
    console.log(await redis.addUser(LOBBY, GAME, 'cindy'));
    const [gid, players] = await redis.addUser(LOBBY, GAME, 'matt');
    console.log('GAME ID', gid, 'PLAYERS', players.split(','));
    redis.quit();
})();

This script defines the script in Redis (Redis caches the script so it will only do this when this is first run on a new Redis instance). The script then adds four players. On the fourth, Redis will return a game id and list of player which we can extract and log.

Summary

• We can achieve Redis atomicity by using Redis atomic commands, Transactions or Lua Scripts
• In general, if you can do something using atomic commands it is more performant than using transactions and transactions are more performant than Lua scripts

Conclusion

In this section, we explored how Redis can serve as the backbone for storing and managing our primitive states in a distributed system. We explored what we can do to ensure atomicity of our business logic by using Redis commands, transactions and Lua scripts.