using async function into loop with nodejs?

Question

I have to check if foreignKey exist, but I can't make a loop with my asynchronous query function

function checkAllFK(tables, foreignKeys) {
  let index = -1;
  for (var key in foreignKeys) {
    index++;
    QueryFunction(tables[index], key, foreignKeys[key])
    .then(result => {
    if(result == null) {
      //here, if result is null, that's mean the foreignKey doesn't exist, then, we have to stop the loop and return false;
      return false;
    }
    else if(index == (tables.length - 1)) {
      //here, that's mean we are at the end of the loop, and we doesn't break it with the previous if, that's mean all foreignKey exist, then, we return true;
      return true;
    }
  }

the problem is that at the end of the first iteration, you exit the function and the result of the return depends only on the first iteration: false if the if condition is met at the first iteration, null if not

even having looked at many similar topics here, I haven't found a solution to my problem.

How about wrapping a Promise around your QueryFunction so you can simply call it from an async function while using await in your loop? — Marc
– Marc, Commented Jun 27, 2019 at 12:33
Don't write these kinds of one-by-one loops over SQL tables in the first place. This is extremely inefficient. SQL has set operations for that, these things can be found out in a single SQL statement. — Tomalak
– Tomalak, Commented Jun 27, 2019 at 12:37

Tomalak · Accepted Answer · 2019-06-27 13:17:26Z

2

Your operation "check all foreignKeys against all tables" can be written in one line.

function checkAllFK(tables, foreignKeys) {
  return Promise.all(tables.map(t => Promise.all(foreignKeys.map(k => QueryFunction(t, k))));
}

This function returns a promise that resolves when all queries are done, so you call it like

checkAllFK(tables, foreignKeys)
    .then(/* success */)
    .catch(/* error */);

However, depending on how many foreignKeys and tables there are and how complex QueryFunction is, this can put enormous stress on the database server. If there are 10 tables and 1000 foreign keys, this would attempt to run 10,000 queries in parallel against the database server. This is not a smart thing to do.

SQL is made to handle these situations. Instead of running 10,000 queries for one thing each, you can decide to run one query for 10,000 things. Or 10 queries for 1000 things each. Both are obviously better than hammering the database server with 10,000 requests.

For example, this returns all foreign keys that do not exist in table_1 in one step.

SELECT
  k.key_column
FROM
  foreign_keys k
  LEFT JOIN table_1 t ON t.key_column = k.key_column
WHERE
  t.key_column IS NULL

It depends on what you do in your QueryFunction how the actual SQL needs to look like.

The fact that you have more than one table to check the same foreign keys against is worrying as well, this usually is an indication of poor database design.

answered Jun 27, 2019 at 13:17

Tomalak

339k68 gold badges547 silver badges635 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

random_user_name Over a year ago

Yes, very much. At least use Promise.all, or better is a single query if possible. Nice answer!

libik · Accepted Answer · 2019-06-27 12:38:13Z

0

there is few common begginer mistakes. Lets start with the tricky one and it is using var keyword in a for-loop in asynchronous context. As you can see, this will return you only 10s, not 1, 2, 3.

for (var i=0; i < 10; i++) {
  setTimeout(() => console.log(i), 100);
}

Fix is easy in this case - just use let, which has different scope than var and works as you would expect.

    for (let i=0; i < 10; i++) {
      setTimeout(() => console.log(i), 100);
    }

Second is asynchronous context - the for-cycle ends before you execute the async context inside the promise that is returned by QueryFunction. If you can use newer version of Node.js then async/await is your saviour, just mark your function as async and have

const result = await QueryFunction(tables[index], key, foreignKeys[key])

However be awared - once you have something in Promise/Asynchronous context, you basically cant get back to the synchronous context. So all your logic needs be aware that you are in asynchronous part. That basically means that all the results will be promises and you will need to then them or await them. Its not bug or something, its behaviour and you need to count on that.

answered Jun 27, 2019 at 12:38

libik

23.1k10 gold badges52 silver badges94 bronze badges

1 Comment

Tomalak Over a year ago

And the third issue is that the OP would be much better off with a single SQL statement that uses a JOIN or a EXISTS operation instead of checking keys one-by-one from the client end.

aoumeior · Accepted Answer · 2019-06-27 12:45:13Z

0

you can do same like print a message in the console or write in a file, can get the visual result.

if you want to get a result, use 'wait', please.

answered Jun 27, 2019 at 12:45

aoumeior

1

1 Comment

random_user_name Over a year ago

This really isn't an answer, and would be better suited as a comment. When you gain more reputation, you'll be able to add comments....

Collectives™ on Stack Overflow

using async function into loop with nodejs?

3 Answers 3

1 Comment

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related