Calling multiple HTTP requests in a single HTTP request in Node.js

Question

I am trying to call multiple URL in a single URL call and push it's json response in an array and send that array in response to the end user.

My code look like this:

var express = require('express');

var main_router = express.Router();

var http = require('http');

urls = [
"http://localhost:3010/alm/build_tool",
"http://localhost:3010/alm/development_tool",
"http://localhost:3010/alm/project_architecture"];

var responses = [];

main_router.route('/')

.get(function (req, res) {

var completed_requests = 0;

for (url in urls) {

  http.get(url, function(res) {

    responses.push(res.body);

    completed_request++;

    if (completed_request == urls.length) {

        // All download done, process responses array
    }
  });
}
res.send(responses);
});

I have also tried this using npm request module. When i run this code it only return NULL or some random output that have only headers.

My aim is to call multiple URL's in a single node get request and append it's JSON output on a array and send to the end user.

Thanks

Yo have to use callbacks or I can suggest the Async module github.com/caolan/async — michelem
– michelem, Commented Dec 23, 2015 at 13:15
See this answer. You should actually wait on the end of the response, by using res.on('end', function... — toasted_flakes
– toasted_flakes, Commented Dec 23, 2015 at 13:15

yoogeeks · Accepted Answer · 2015-12-23 14:45:00Z

23

Here, try this code,

const async = require('async');
const request = require('request');

function httpGet(url, callback) {
  const options = {
    url :  url,
    json : true
  };
  request(options,
    function(err, res, body) {
      callback(err, body);
    }
  );
}

const urls= [
  "http://localhost:3010/alm/build_tool",
  "http://localhost:3010/alm/development_tool",
  "http://localhost:3010/alm/project_architecture"
];

async.map(urls, httpGet, function (err, res){
  if (err) return console.log(err);
  console.log(res);
});

Explanation : This code uses async and request node packages. async.map by definition takes 3 params, first one being an array, second being the iterator function you want to call with each element of that array, and the callback function, called when async.map has finished processing.

map(arr, iterator, [callback])

Produces a new array of values by mapping each value in arr through the iterator function. The iterator is called with an item from arr and a callback for when it has finished processing. Each of these callback takes 2 arguments: an error, and the transformed item from arr. If iterator passes an error to its callback, the main callback (for the map function) is immediately called with the error.

Note: All calls to iterator function are parallel.

Inside your httpGet function, you are calling request function with passed url, and explicitly telling the response format to be json. request, when finished processing, calls the callback function with three params, err - if any, res - server response, body - response body. In case there is no err from request, async.map collects the results from these callbacks as an array, and passes that array at the end to its third, callback function. Otherwise,if (err) is true, the async.map function stops the execution and calls its callback with an err.

edited Dec 23, 2015 at 14:45

answered Dec 23, 2015 at 14:08

yoogeeks

9658 silver badges24 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

birdoftheday Over a year ago

what's request here? how about async? using libraries in answers is fine but only if you specify what libraries you're using :P

yoogeeks Over a year ago

Well, asker seems to be already aware of these packages. Anyhow, updating for other users.

Suhaib Over a year ago

thanks for this. Is it possible to merge all the data bodies into one ? This is how it looks for me if I downlad JSON form multiple URls i.imgur.com/nX1IRla.png. Would you know how to merge them under 1 data body ?

yoogeeks Over a year ago

you can simply loop through each response body and add it to your final body. However, be careful if you're using pagination.

Ita · Accepted Answer · 2015-12-23 14:27:58Z

4

I suggest to use the async library.

async.map(urls, http.get, function(err, responses){
  if (err){
    // handle error
  }
  else {
    res.send responses
  }
})

The snippet above will perform a http.get call for each of the urls in parallel, and will call your callback function with the results of all of the calls after all the responses were received.

If you want to call the urls in series, you can use async.mapSeries instead. If you want to limit the number of concurrent requests you can use async.mapLimit.

edited Dec 23, 2015 at 14:27

answered Dec 23, 2015 at 13:18

Ita

9117 silver badges21 bronze badges

2 Comments

Vaibhav Jain Over a year ago

Thanks for the comment. If you are having any available code, please share. I Have tried following code but does not get anything in response. main_router.route('/') .get(function (req, res) { var test = async.each(urls, http.get, function(err, responses, callback){ if (err){ // handle error } else { return responses; } }) res.send(test); res.end(); });

yoogeeks Over a year ago

Note: async.each callback doesn't handle responses. If you need the responses at the end of call, you should use async.map which will have response of each HTTP request in the form of an array in your callback function.

Collectives™ on Stack Overflow

Calling multiple HTTP requests in a single HTTP request in Node.js

2 Answers 2

4 Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related