node.js - stream http response from multiple calls

Question

In my express app, I want to make multiple calls to an API and stream each response back to the client as and when I receive it, instead of waiting for all of them.

For example, if I make requests to yelp for restaurants in San Francisco, Berkeley and Palo Alto in parallel, I shouldn't have to wait for all the responses to come back and be able stream them as they're available. How would I do this?

If you are going to restructure the API data server side for your response, then you must wait for the data from both external APIs. — hexacyanide
– hexacyanide, Commented Sep 15, 2013 at 23:05
If you want JSON data, you'll need to make multiple calls to the server and fetch them in batches (as you can't partially send JSON back), or use sockets/long polling to "push" the data as it's completed. — WiredPrairie
– WiredPrairie, Commented Sep 15, 2013 at 23:40
What does your response object look like? JSON? What is receiving it? There are streaming specifications for JSON-like data available. — Brad
– Brad, Commented Sep 16, 2013 at 4:09
Thanks for your help. I have another question. Since it's kind of similar to the earlier one, I just updated that. — tldr
– tldr, Commented Sep 17, 2013 at 5:36

Daniel Beardsley · Accepted Answer · 2013-09-17 07:26:22Z

Since browsers wait till the entire response is received before passing the result to javascript, this isn't directly possible. You could, on the other hand, do it with websockets.

Possible architecture:

Server: Establish websocket connection with client
Server: Fire off 4 requests to APIs in parallel
Server: As data arrives on each connection, pass each packet to the websocket with something like {api: "yelp", data: ... }
Client: Keep appending incoming data to strings representing each api's response.
Server: When a connection is done, send a done message { api: "yepl", done: true }
Client: Upon receiving a done message, you have a full response from that API.

I highly doubt this is a good idea. It's far more complex and you'd be better off using 4 parallel requests from the client, or if possible, querying the apis directly from the browser.

Collectives™ on Stack Overflow

node.js - stream http response from multiple calls

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related