Better way to write a simple Node redis loop (using ioredis)?

Question

So, I'm stilling learning the JS/Node way from a long time in other languages.

I have a tiny micro-service that reads from a redis channel, temp stores it in a working channel, does the work, removes it, and moves on. If there is more in the channel it re-runs immediately. If not, it sets a timeout and checks again in 1 second.

It works fine...but timeout polling doesn't seem to be the "correct" way to approach this. And I haven't found much about using BRPOPLPUSH to try to block (vs. RPOPLPUSH) and wait in Node....or other options like that. (Pub/Sub isn't an option here...this is the only listener, and it may not always be listening.)

Here's the short essence of what I'm doing:

var Redis = require('ioredis');
var redis = new Redis();

var redisLoop = function () {
    redis.rpoplpush('channel', 'channel-working').then(function (result) {
        if (result) {
            processJob(result); //do stuff

            //delete the item from the working channel, and check for another item
            redis.lrem('channel-working', 1, result).then(function (result) { });
            redisLoop();
        } else {
            //no items, wait 1 second and try again
            setTimeout(redisLoop, 1000);
        }
    });
};

redisLoop();

I feel like I'm missing something really obvious. Thanks!

Can you elaborate on why pub/sub isn't an option? It's perfectly fine to publish when there aren't any listeners subscribed. You'd have the process that pushes work into Redis also publish. If this app is subscribed, it will wake it up. If not, nothing happens. — Wander Nauta
– Wander Nauta, Commented Nov 5, 2016 at 0:16
From what I've seen (without more complex workarounds) if a message is published with no subscribers, the message is lost. This is more of a message queue model, with only one listener as well -- it's most important that the items in the channel sit around forever until popped, be that seconds or days (if the process dies, machine goes offline, etc). — XediDC
– XediDC, Commented Nov 5, 2016 at 0:23
That's true, the message is lost. That's why you also publish: the work-generating app would put the work in the queue, and only then PUBLISH something (say, an empty string) so that this app wakes up in case it's online but sleeping. — Wander Nauta
– Wander Nauta, Commented Nov 5, 2016 at 0:25
Ah, okay, I see what you mean. Yes, I did read an article about the Publish + Queue method to achieve additional reliability over both methods. I don't have full control of the upstream end, but I'll see about that too. Thanks! — XediDC
– XediDC, Commented Nov 5, 2016 at 0:31

Igor Raush · Accepted Answer · 2016-11-05 00:30:55Z

5

BRPOPLPUSH doesn't block in Node, it blocks in the client. In this instance I think it's exactly what you need to get rid of the polling.

var Redis = require('ioredis');
var redis = new Redis();

var redisLoop = function () {
    redis.brpoplpush('channel', 'channel-working', 0).then(function (result) {
        // because we are using BRPOPLPUSH, the client promise will not resolve
        // until a 'result' becomes available
        processJob(result);

        // delete the item from the working channel, and check for another item
        redis.lrem('channel-working', 1, result).then(redisLoop);
    });
};

redisLoop();

Note that redis.lrem is asynchronous, so you should use lrem(...).then(redisLoop) to ensure that your next tick executes only after the item is successfully removed from channel-working.

edited Nov 5, 2016 at 0:30

answered Nov 5, 2016 at 0:25

Igor Raush

15.3k1 gold badge38 silver badges58 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

XediDC Over a year ago

Thank you! I think I may have come close to that....but then went off in another direction before I got there. It looks really obvious now looking at my trimmed down example and then what you wrote. :)

Igor Raush Over a year ago

No problem! BTW, it's important that no I/O is blocking in Node here. You can do other work concurrently. However, if you try to reuse the same redis client to make calls to Redis concurrently, then your loop will interfere. I mention because this bit me in a recent project and I realized I needed a second "non-blocking" client to run commands while the first one blocks.

Collectives™ on Stack Overflow

Better way to write a simple Node redis loop (using ioredis)?

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest