running many parallel http requests using node.js

Question

I've been tasked with splitting a process into many smaller processes running in parallel, distributed to a number of slave machines. The request comes in via HTTP, the server splits it up into a number of child processes issued to slave machines, waits for all the slave requests to return responses, and then collates the aggregate results into a single data object returned as the result of the top-level request. I thought node.js would be useful, but since it's single threaded I can't quite tell if this is possible, or if it will block waiting for each request to come back before moving on to the next one. Is this possible with node.js? If so, could someone point me in the right direction? i.e. a node module to use or outline how it would be done?

Thanks for any help.

Raynos · Accepted Answer · 2011-04-15 09:31:42Z

5

You can spawn child processes manually.

var util  = require('util'),
    spawn = require('child_process').spawn,
    node = spawn('node', ['slave.js']);

node.stdout.on('data', function (data) {
  console.log('stdout: ' + data);
});

node.stderr.on('data', function (data) {
  console.log('stderr: ' + data);
});

node.on('exit', function (code) {
  console.log('child process exited with code ' + code);
});

node.stdin.write("do my work!");

We are anticipating a Web Worker API being implemented to handle child processes on a higher abstraction

edited Apr 15, 2011 at 9:31

answered Apr 15, 2011 at 9:25

Raynos

170k57 gold badges359 silver badges399 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

clee · Accepted Answer · 2011-04-15 08:41:49Z

1

You want to read about WebWorkers. Specifically you should read http://developer.yahoo.com/blogs/ydn/posts/2010/07/multicore_http_server_with_nodejs/

answered Apr 15, 2011 at 8:41

clee

11.2k6 gold badges38 silver badges28 bronze badges

Comments

yojimbo87 · Accepted Answer · 2011-04-15 09:29:29Z

1

Try to look at node.js cluster module which is a multi-core server manager.

answered Apr 15, 2011 at 9:29

yojimbo87

68.8k26 gold badges128 silver badges133 bronze badges

Comments

generalhenry · Accepted Answer · 2011-04-15 09:26:34Z

0

if even a single one of the smaller processes had a synchronous interface being single threaded would be an issue.

Fortunately node is designed around asynchronous interfaces.

such as the child process http://nodejs.org/docs/v0.4.6/api/child_processes.html#child_process.exec

or http requests http://nodejs.org/docs/v0.4.6/api/http.html#http.request

answered Apr 15, 2011 at 9:26

generalhenry

17.3k4 gold badges51 silver badges63 bronze badges

1 Comment

Jake Over a year ago

In node's Http.request documentation, it says: "Node maintains several connections per server to make HTTP requests.". My understanding of this is that I can just issue http requests in a for loop, keeping track of all the results as they come back, and then call my callback when they've all returned. However, this approach seems dependent on the number of connections node keeps open as stated in that sentence, meaning that it will parallelize the process until I hit the number of available requests in the pool. Anyone know how I can increase that number to say 100?

jake · Accepted Answer · 2011-09-20 10:38:03Z

0

I think this blog post might interesting in terms how async helps with parallel execution, and how to do parallel execution with Node.

http://blog.mixu.net/2011/02/01/understanding-the-node-js-event-loop/

answered Sep 20, 2011 at 10:38

jake

2,4191 gold badge21 silver badges31 bronze badges

Collectives™ on Stack Overflow

running many parallel http requests using node.js

5 Answers 5

Comments

Comments

Comments

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

Comments

Comments

Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related