How to execute synchronous HTTP requests in JavaScript

Question

I need to execute unknown number of http requests in a node.js program, and it needs to happen synchronously. only when one get the response the next request will be execute. How can I implement that in JS?

I tried it synchronously with the requset package:

function HttpHandler(url){

  request(url, function (error, response, body) {
     ...
  })

}

HttpHandler("address-1")
HttpHandler("address-2")
...
HttpHandler("address-100")

And asynchronously with request-promise:

async function HttpHandler(url){

  const res = await request(url)
  ...

}

HttpHandler("address-1")
HttpHandler("address-2")
...
HttpHandler("address-100")

Non of them work. and as I said I can have unknown number of http request over the program, it depends on the end user.

Any ideas on to handle that?

There is no such thing as synchronous http requests in nodejs (all networking in nodejs is asynchronous). Do you perhaps mean you need to run them in sequence one after another, even though they are asynchronous? And, please show what you're trying to do with the responses from these multiple calls as that will effect how the code should be written. Right now, they don't seem dependent upon one another so they could be run in parallel, without sequencing them in a strict order. — jfriend00
– jfriend00, Commented Jan 22, 2021 at 22:26
Also, the request() package has been deprecated and is not recommended for new code. You can pick from a list of alternatives here. My favorite is got() which will also make it easier to sequence your calls by using promises and await. — jfriend00
– jfriend00, Commented Jan 22, 2021 at 22:28
@jfriend00 The http requests can be asynchronous, I'm fine with that. I just need the program to be synchronous. for example to get notify when one http got a response and then send the next one, I guess I can use some form of callback, but I'm not sure how to implement that, since I don't have a list of all the requests before runtime. It will be totally dynamic. — dev3dev
– dev3dev, Commented Jan 22, 2021 at 22:44
Please show the real logic of your problem so we can offer a complete solution to the problem. — jfriend00
– jfriend00, Commented Jan 22, 2021 at 22:52

jfriend00 · Accepted Answer · 2021-01-23 07:48:35Z

4

Use the got() library, not the request() library because the request() library has been deprecated and does not support promises. Then, you can use async/await and a for loop to sequence your calls one after another.

const got = require('got');
let urls = [...];    // some array of urls

async function processUrls(list) {
    for (let url of urls) {
        await got(url);
    }
}

processUrls(urls).then(() => {
    console.log("all done");
}).catch(err => {
    console.log(err);
});

You are claiming some sort of dynamic list of URLs, but won't show how that works so you'll have to figure out that part of the logic yourself. I'd be happy to show how to solve that part, but you haven't given us any idea how that should work.

If you want a queue that you can regularly add items to, you can do something like this:

class sequencedQueue {
    // fn is a function to call on each item in the queue
    // if its asynchronous, it should return a promise
    constructor(fn) {
        this.queue = [];
        this.processing = false;
        this.fn = fn;
    }
    add(...items) {
        this.queue.push(...items);
        return this.run();
    }
    async run() {
        // if not already processing, start processing
        // because of await, this is not a blocking while loop
        while (!this.processing && this.queue.length) {
            try {
                this.processing = true;
                await this.fn(this.queue.shift());
            } catch (e) {
                // need to decide what to do upon error
                // this is currently coded to just log the error and
                // keep processing.  To end processing, throw an error here.
                console.log(e);
            } finally {
                this.processing = false;
            }
        }
    }
}

edited Jan 23, 2021 at 7:48

answered Jan 22, 2021 at 23:36

jfriend00

711k104 gold badges1.1k silver badges1k bronze badges

Sign up to request clarification or add additional context in comments.

10 Comments

Brad Over a year ago

I'd also recommend node-fetch, as it's compatible with the Fetch API used in browsers, and makes portability a bit easier.

jfriend00 Over a year ago

@Brad - It is a matter of opinion which API to use here. For programming in nodejs, the fetch() API is far, far less convenient than something like got() in a number of different ways (streams, parsing of responses, general feature support, etc...). Sure, if you want compatibility with browser programming, go for node-fetch, but if not, it's not the best option in my opinion.

dev3dev Over a year ago

That's will work perfectly. but there is one more thing. there are new requests to send all the time. and I know can push them to an array and run through it every X seconds, but I was hoping to find a better way.

jfriend00 Over a year ago

@dev3dev - As I've said multiple times, if you show more of how new urls arrive, I can offer a solution for that, but can't code for something I don't know anything about. If it's a continuous process, you can use a queue rather than an array (similar concept). Or run this on all the URLs you have and put any newly arrive urls in a new array. When this array is done, start processing the array of newly arrived ones. There are hundreds of ways to do some sort of queue. You could use a while loop on a queue. while (queue not empty) { await got(next queue item) }

dev3dev Over a year ago

@jfriend00 Thank you. but if I`m getting your example right, so if the queue will become empty once, the while loop will stop, and new URLs that will inserted to the queue will not be handled isn't it? that's why I thought about setting timer to check every few seconds for a new URLs in the array/queue, but I feel like it's not the right solution. and I'm sorry for not showing the full code, it's 10 files of project and I'm trying to be keep the question as simple as possible .

|

Collectives™ on Stack Overflow

How to execute synchronous HTTP requests in JavaScript

1 Answer 1

10 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

10 Comments

Your Answer

Sign up or log in

Post as a guest

Related