Handling of large array of items in nodejs

Question

I have array of more then 100 items. Each item hits a url. After getting response from the specific url of the item I do insert option in the database.

If I use async await, then I have to wait till the 100 items get processd. But if I do asynchronoulsy, then I got response but as asynchronus behavior the data in db will not be visible untill all process get completed.

Example 1: using async await

var loc = async(locations){
    return new Promise((resolve,reject) => {

        try{
            for(let item of locations){
                var response = await html(location.url);
                await insertDB(response);
            }
        }catch(e){
            console.log('error : ',e);
        }

        resolve('All operation completed successfully');
    })
}

loc(locations);

locations is array of items In this I will get response only when all the requests get completed.

Example 2: asynchronously

var loc = (locations){
    return new Promise((resolve,reject) => {

        locations.forEach(location => {
            html(location.url,(res) => {
                insertDB(response,(rows) => {

                });
            });
        })
        resolve('All operation completed successfully');
    })
}

loc(locations)

Here I will immediately get the response, but the process on hitting and insertion will be processing in the background.

Considering example 1, is it possible that we can divide the looping request into child process so that make it execute fast in nodejs or any other way to do it?

LHJ · Accepted Answer · 2020-09-12 15:00:40Z

1

Is this what you want? It will work like your 2nd solution, but it will be resolved after every insert complete

var loc = (locations){
    return new Promise((resolve,reject) => {
        const promises = locations.map(async (location) => {
            try { 
                var response = await html(location.url);
                await insertDB(response);
            } catch(e) {
                console.log('error : ',e);
            }
        })
        await Promise.all(promises);
        resolve('All operation completed successfully');
    })
}

Should have difference. Can you do me a favor to test the performance time.
solution1() as your first solution function
solution2() as my solution function

console.time('solution1');
await solution1(locations);
console.timeEnd('solution1');

console.time('solution2');
await solution2(locations);
console.timeEnd('solution2');

assume insertDB cost 100ms ~ 200ms, avg = 150ms
Solution 1:
run loop 1, wait insertDB -> run loop 2, wait insertDB
time = 150ms + 150ms = 300ms
Solution 2:
run loop 1, without wait insertDB -> run loop 2, without wait insertDB
till all insertDB complete, then resolve
time = 150ms
because all insertDB start without waiting another insertDB
credit: https://stackoverflow.com/a/56993392/13703967

test: https://codepen.io/hjian0329/pen/oNxdPoZ

edited Sep 12, 2020 at 15:00

answered Sep 12, 2020 at 13:22

LHJ

6925 silver badges16 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Manoj Thakur Over a year ago

What I want is like I have 1000 request and I have 10 persons and wanted to dived these 1000 request to these 10 people so that I can completed the 1000 request.

LHJ Over a year ago

I think you are looking for multithreading. Too bad i didn't try this before. After some search, here's are something you can read for blog.logrocket.com/…

LHJ Over a year ago

By the way, my answer above should have better performance compare to your 1st solution. According to stackoverflow.com/questions/56992054/…

Manoj Thakur Over a year ago

There is no so much difference between your answer and the solution for example 1. Still I have to wait for all the task to get completed only then final response will be delivered to the user.

Collectives™ on Stack Overflow

Handling of large array of items in nodejs

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related