How to iterate through an array of promises in node.js?

Question

In javascript, I have an non flat array of Promises:

const array = [
  [ promise11, promise12.. promise1n ],
  [ promise21, promise22.. promise2n ],
  [ promise31, promise32.. promise3n ],
  ...  
  [ promisek1, promisek2.. promisekn ]
];

How do iterate through the array in such a fashion, where the next array of promises would start resolving ONLY after the previous array of promises have resolved? For instance I would like the script to wait until [ promise11, promise12.. promise1n ] promises resolve in parrallel, before calling on to the next array of promises.

I am looking for a general solution as my application cannot know in advance how many calls it'll have to make (it's a webscraper).

Graham · Accepted Answer · 2017-09-23 00:44:44Z

4

You could use a variation on the Array#reduce pattern:

array.reduce(
    (p, subArray) => p.then(() => Promise.all(subArray)),
    Promise.resolve()
);

That sets up a promise chain where each entry in the chain is the result of the promise from Promise.all for the subarrays in the array.

...but:

...where the next array of promises would start resolving ONLY after the previous array of promises have resolved?

That's not how promises work. Once you have the promise, the action it promises a result for is already in progress. You don't "start" it.

So instead, you'd want to make array an array of functions that start the processes in question and return promises, which is slightly different:

function get(label) {
  console.log(label + "- starting");
  return new Promise(resolve => {
    setTimeout(() => {
      console.log(label + "- resolving");
      resolve(label);
    }, Math.random() * 500);
  });
}
const array = [
  [ get.bind(null, "1a"), get.bind(null, "1b"), get.bind(null, "1c") ],
  [ get.bind(null, "2a"), get.bind(null, "2b") ],
  [ get.bind(null, "3a"), get.bind(null, "3b"), get.bind(null, "3c"), get.bind(null, "3d") ]
];

array.reduce(
  (p, subArray) => p.then(() => Promise.all(subArray.map(f => f()))),
  Promise.resolve()
).then(() => {
  console.log("All done");
});

.as-console-wrapper {
  max-height: 100% !important;
}

Notice how the array is now an array of functions, which when called "start" the process and then return a promise. Our Array#reduce loop is only slightly modified: We use Array#map to call the functions for this subarray and get an array of their promises, which is what we feed into Promise.all. So all of the functions for a given subarray are called to start the work for that subarray, and then we wait unti they're all complete before continuing to the next subarray and having it start its work.

edited Sep 23, 2017 at 0:44

Graham

7,86020 gold badges67 silver badges92 bronze badges

answered Feb 1, 2017 at 15:19

T.J. Crowder

1.1m201 gold badges2k silver badges2k bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

simonas88 Over a year ago

The second solution you posted worked brilliantly, but I still have to understand how. Anyway, thank you very much!

T.J. Crowder Over a year ago

@simonas88: If there's anything I can explain further, please don't hesitate to ask.

simonas88 Over a year ago

(p, subArray) => p.then(() => Promise.all(subArray.map(f => f()))) This reducer seems to be the key in the solution. Am I correct to assume, that p.then() returns (therefore completing a single subArray) when Promise.all() resolves all promises in it's parameter?

T.J. Crowder Over a year ago

@simonas88: then returns immediately. What it returns is the promise returned by Promise.all, which is settled later when all of the promises created in the map operation are settled. The reduce builds a chain of those "all" promises, each waiting for the one before it to resolve before calling its then callback and starting the next batch.

simonas88 Over a year ago

Ok, so, the reducer finishes really quickly, and returns a chain of unfulfilled promises? And since on the outside each link in the 'chain' contains a Promise.all() I get the sequential/parallel behavior that I was after?

Collectives™ on Stack Overflow

How to iterate through an array of promises in node.js?

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related