3

EDIT: I'm using node v8.0.0

I just started learning how to access SQL databases with node-postgres, and I'm having a little bit of trouble accessing multiple databases to collect the data in a work able format, particularly with executing multiple queries within forEach loops. After a few tries, I'm trying async/await, but I get the following error:

await client.connect()
  ^^^^^^
SyntaxError: Unexpected identifier

When I tried using a pool or calling .query sequentially, I would get something along the lines of

1
[]
could not connect to postgres Error: Connection terminated

Here is an abbreviated version of my code:

const { Client } = require('pg');
const moment = require('moment');
const _ = require('lodash');
const turf = require('@turf/turf');

const connString = // connection string
var collected = []
const CID = 300
const snaptimes = // array of times
var counter=0;
const client = new Client(connString);

function createArray(i,j) {
     // return array of i arrays of length j
}

await client.connect()

snaptimes.forEach(function(snaptime){
  var info = {}; // an object of objects
  // get information at given snaptime from database 1
  const query1 = // parametrized query selecting two columns from database 1
  const result1 = await client.query(query1, [CID,snaptime]);
  var x = result1.rows;
  for (var i = 0; i < x.length; i++) {
     // store data from database 1 into info
     // each row is an object with two fields
  }

  // line up subjects on the hole
  const query2 = // parametrized query grabbing JSON string from database 2
  const result2 = await client.query(query2, [CID,snaptime]);
  const raw = result2.rows[0].JSON_col;
  const line = createArray(19,0); // an array of 19 empty arrays
  for (var i = 0; i < raw.length; i++) {
     // parse JSON object and record data into line 
  }

  // begin to collect data
  var n = 0;
  var g = 0;
  // walk down the line
  for (var i = 18; i > 0; i--) {
    // if no subjects are found at spot i, do nothing, except maybe update g
    if ((line[i] === undefined || line[i].length == 0) && g == 0){
      g = i;
    } else if (line[i] !== undefined && line[i].length != 0) {
      // collect data for each subject if subjects are found
      line[i].forEach(function(subject){
        const query 3 = // parametrized query grabbing data for each subject 
        const result3 = await client.query(query3,[CID,subject,snaptime]);
        x = result3.rows;
        const y = moment(x[0].end_time).diff(moment(snaptime),'minutes');
        var yhat = 0;
        // the summation over info depends on g
        if (g===0){
          for (var j = i; j <= 18; j++){
            yhat = moment.duration(info[j].field1).add(yhat,'m').asMinutes();
          }
        } else {
          for (var j = i; j <= 18; j++){
            if (i<j && j<g+1) {
              yhat = moment.duration(info[j].field2).add(yhat,'m').asMinutes();
            } else {
              yhat = moment.duration(info[j].field1).add(yhat,'m').asMinutes();
            }
          }
        }
        collected.push([y,yhat,n,i]);
      });
    }
    n+=line[i].length;
    g=0;
  }
  // really rough work-around I once used for printing results after a forEach of queries
  counter++;
  if (counter===snaptimes.length){
    console.log(counter);
    console.log(collected);
    client.end(); 
  }
});

3 Answers 3

4

The problem is caused by your forEach callback not being async:

snaptimes.forEach(function(snaptime){

should be:

snaptimes.forEach(async function (snaptime) {

for the await to be recognizable at all.

Keep in mind that an async function returns immediately and it returns a promise that gets eventually resolved by return statements of the async function (or rejected with uncaught exceptions raised inside the async function).

But also make sure your Node version supports async/await:

  • Since Node 7.6 it can be used with no --harmony flag.
  • In Node 7.x before 7.6 you have to use the --harmony flag.
  • It was not available in Node before 7.0.

See: http://node.green/#ES2017-features-async-functions

Also note that you can use await only inside of functions declared with the async keyword. If you want to use it in the top level of your script or module then you need to wrap it in an immediately invoked function expression:

// cannot use await here
(async () => {
  // can use await here
})();
// cannot use await here

Example:

const f = () => new Promise(r => setTimeout(() => r('x'), 500));

let x = await f();
console.log(x);

prints:

$ node t1.js 
/home/rsp/node/test/prom-async/t1.js:3
let x = await f();
              ^
SyntaxError: Unexpected identifier

but this:

const f = () => new Promise(r => setTimeout(() => r('x'), 500));

(async () => {
  let x = await f();
  console.log(x);
})();

prints:

$ node t2.js 
x

after 0.5s delay, as expected.

On versions of Node that don't support async/await the first (incorrect) example will print:

$ ~/opt/node-v6.7.0/bin/node t1.js 
/home/rsp/node/test/prom-async/t1.js:3
let x = await f();
              ^
SyntaxError: Unexpected identifier

and the second (correct) example will print a different error:

$ ~/opt/node-v6.7.0/bin/node t2.js 
/home/rsp/node/test/prom-async/t2.js:3
(async () => {
       ^
SyntaxError: Unexpected token (

It's useful to know because Node versions that don't support async/await will not give you a meaningful error like "async/await not supported" or something like that, unfortunately.

Sign up to request clarification or add additional context in comments.

3 Comments

Putting async before any function calling await cleared up the issue very nicely. However, when I try adding more snaptimes to the the snaptime array on which the first forEach is called, I either get Error: Connection terminated unexpectedly or Error: read ECONNRESET. Going off of print statements, these errors occur immediately on query1. Moving client.end() to after the forEach loop just causes the code to not run
@HansyPiou When you're using async/await then I would recommend using for (let snaptime of snaptimes) { ... } instead of snaptimes.forEach(function (snaptime) { ... }); because that way you will not call another functions and it will be easier to synchronize the iteration steps.
The whole things working great now! Using for() loops meant I had to wrap them all into a larger (async) function, but that ultimately cleared up issues with node-postgres since I could connect the client before collecting the data and close it when I was done. Thank you for all your help!
0

Make sure that you should use async block outside like:

async function() {
  return await Promise.resolve('')
}

And it is default supported after node 7.6.0. Before 7.6.0, you should use --harmony option to work for it.

node -v first to check your version.

Comments

0

First of all, you don't know enough about async-await just yet. don't worry, it's actually quite easy; but you need to read the documentation to be able to use that stuff.

More to the point, the problem with your code is that you can only await inside async functions; you're doing that outside of any function.

First of all, here's the solution that is closest to the code you wrote:

const { Client } = require('pg');
const moment = require('moment');
const _ = require('lodash');
const turf = require('@turf/turf');

const connString = // connection string
var collected = []
const CID = 300
const snaptimes = // array of times
var counter=0;
const client = new Client(connString);

function createArray(i,j) {
    // return array of i arrays of length j
}

async function processSnaptime (snaptime) {
    var info = {}; // an object of objects
    // get information at given snaptime from database 1
    const query1 = // parametrized query selecting two columns from database 1
    const result1 = await client.query(query1, [CID,snaptime]);
    var x = result1.rows;
    for (var i = 0; i < x.length; i++) {
      // store data from database 1 into info
      // each row is an object with two fields
    }

    // line up subjects on the hole
    const query2 = // parametrized query grabbing JSON string from database 2
    const result2 = await client.query(query2, [CID,snaptime]);
    const raw = result2.rows[0].JSON_col;
    const line = createArray(19,0); // an array of 19 empty arrays
    for (var i = 0; i < raw.length; i++) {
      // parse JSON object and record data into line
    }

    // begin to collect data
    var n = 0;
    var g = 0;
    // walk down the line
    for (var i = 18; i > 0; i--) {
      // if no subjects are found at spot i, do nothing, except maybe update g
      if ((line[i] === undefined || line[i].length == 0) && g == 0){
        g = i;
      } else if (line[i] !== undefined && line[i].length != 0) {
        // collect data for each subject if subjects are found
        line[i].forEach(function(subject){
          const query 3 = // parametrized query grabbing data for each subject
          const result3 = await client.query(query3,[CID,subject,snaptime]);
          x = result3.rows;
          const y = moment(x[0].end_time).diff(moment(snaptime),'minutes');
          var yhat = 0;
          // the summation over info depends on g
          if (g===0){
            for (var j = i; j <= 18; j++){
              yhat = moment.duration(info[j].field1).add(yhat,'m').asMinutes();
            }
          } else {
            for (var j = i; j <= 18; j++){
              if (i<j && j<g+1) {
                yhat = moment.duration(info[j].field2).add(yhat,'m').asMinutes();
              } else {
                yhat = moment.duration(info[j].field1).add(yhat,'m').asMinutes();
              }
            }
          }
          collected.push([y,yhat,n,i]);
        });
      }
      n+=line[i].length;
      g=0;
    }
    // really rough work-around I once used for printing results after a forEach of queries
    counter++;
    if (counter===snaptimes.length){
      console.log(counter);
      console.log(collected);
    }
}

async function run () {
  for (let snaptime of snaptimes) {
    await processSnaptime(snaptime);
  }
}

/* to run all of them concurrently:
function run () {
  let procs = [];
  for (let snaptime of snaptimes) {
    procs.push(processSnaptime(snaptime));
  }
  return Promise.all(procs);
}
*/

client.connect().then(run).then(() => client.end());

client.connect returns a promise and I use then to call run once it's resolved. When that part is over, client.end() can be called safely.

run is an async function, therefore it can use await to make the code more readable. The same goes for processSnaptime.

Of course I can't actually run your code, so I can only hope I didn't make any mistakes.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.