async operations in nodejs express

Question

I am having a route

app.get('/posts', function (req, res) {
  res.json([
    // data
  ]);

  // what would happen to the code below
  // Suppose it is processing a million records
  processMillionRecords();
});

function processMillionRecords () {
  // process million records
}

Once the response is sent, another function is called which is quite an expensive operation. What would happen if this continues? Are there any implications?

I know the ideal way of doing it is using background processes. So I replaced it with the below.

res.json([
  // data
]);

var child = require('child_process');
child.fork('worker.js');

// worker.js
function processMillionRecords () {
  // process million records
}

Which one is more preferred and in which cases?

How is forking another process not using background processes? — Pointy
– Pointy, Commented Dec 12, 2014 at 15:09
yeah you are right, may be I should rephrase the question properly. — Madhusudhan
– Madhusudhan, Commented Dec 12, 2014 at 15:10
OK - well I don't think there's anything wrong with using a background process, assuming you're OK with the overhead (and in Node it's not like you have many choices). Overhead is not really important if you're really doing a large amount of work, since the fork only happens once (per work batch). — Pointy
– Pointy, Commented Dec 12, 2014 at 15:12

Timothy Strimple · Accepted Answer · 2014-12-12 15:15:54Z

2

Node is built to handle long asynchronous operations. Unless it's very CPU heavy, you probably don't need to fork a new worker process. Just write your processMillionRecords to be async and make sure it doesn't block the event loop.

answered Dec 12, 2014 at 15:15

Timothy Strimple

23.1k6 gold badges73 silver badges76 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

rdegges · Accepted Answer · 2014-12-12 17:52:00Z

0

What will happen in the situation described above (as mentioned by Timothy), is this:

Your app will return data to the user quickly.
Your Node process will continue to run the async code behind the scenes until the process has finished.

If you're building an app that is expecting a lot of traffic, or a lot of CPU intensive workload, it's best to move this stuff into a separate server specifically meant for processing long-running tasks. This reduces the chance that:

Your web server reboots -- destroying your background processing stuff.
Your web server is processing too many background tasks to properly handle incoming user requests.

Hope that helps.

answered Dec 12, 2014 at 17:52

rdegges

34.1k22 gold badges88 silver badges109 bronze badges

Collectives™ on Stack Overflow

async operations in nodejs express

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related