async / await run async functions sequentially in nodejs, puppeteer

Question

Problem

I have 3 functions which are dependent of one another and therefore I would want them to run sequentially. The function called are async and return promise and it takes time to run.

My Code - Index.js

#Function One
scrapA.scrap().then(data => {
    readWriteToFile.writeToFile(data, 'scrapA-data-'+timestamp, 'json');
});

#Function Two
compare.compareData('./output/json/data.json', './output/json/scraped-data-'+timestamp+'.json').then(data => {

    let dataWithStockData = data;
    let mapDatas = dataWithStockData.map( obj => { 

        let finalObj = {
            manufacturer: obj.manufacturer,
            grade: obj.condition,
            carrier: obj.carrier,
            stock: obj.stock,
        }
        return finalObj;
    });

#Function Three
readWriteToFile.writeToFile(data, 'data-with-stock-'+timestamp, 'csv'); });

My Code - compare.js

async function compareData(models, competitorsData) {

try {
    
    const modelsArr = JSON.parse(fs.readFileSync(models, { encoding: 'utf8', flag: 'r' }));
    const competitorsDataArr = JSON.parse(fs.readFileSync(competitorsData, { encoding: 'utf8', flag: 'r' }));

    //Loop through both array of json data
    modelsArr.forEach((device, deviceKey) => {

        competitorsDataArr.forEach((scrapedData, scrapedDataKey) => {

            
            if (
                    scrapedData.manufacturer.toLowerCase() === data.manufacturer.toLowerCase()
            ) {

                 modelsArr[dataKey]['stk'] = scrapedData.stk;

            }
        })
    });

    return modelsArr;

} catch (error) {

    console.log("Error", error);
    throw error;
}

}

My Code - readWriteToFile.js

async function writeToFile(data, filename, type) {

let finalData = '';

if (type == "json") {

    finalData = JSON.stringify(data);

} else if (type == "csv") {

    finalData = arrayToCSV(data);

} else {
    console.log("File type not supported");
    process.exit(1);
}

fs.writeFile("./output/" + type + '/' + filename + '.' + type, finalData, (err) => {
    if (err) {
        console.log(err);
    } else {
        console.log("File written successfully\n");
    }
});

}

My solution

I simply wrapped all my function calls in async and added await at the front. I does not work and it does not throw any errors.

(async () => {
    
    await scraperA.scrap().then(data => {
            console.log("DATA", data);
            readWriteToFile.writeToFile(data, 'scraped-data-'+timestamp, 'json');
        });
    
    await compare.compareData('./output/json/data.json', './output/json/scraped-data-'+timestamp+'.json').then(data => {
        let dataWithStockData = data;
        mapData = dataWithStock.map( obj => { 
    
            let finalObj = {
                manufacturer: obj.manufacturer,
                grade: obj.condition,
                stock: obj.stock,
            }
            return finalObj;
        });
    });

    await readWriteToFile.writeToFile(mapData, 'data-with-stock-'+timestamp, 'csv');

});

what is the expected outcome? and where (if at all?) is mapData declared in your solution? — Fre Timmerman
– Fre Timmerman, Commented Dec 13, 2021 at 10:06
You are not calling the function. You have to call the function to make an async IIFE. — user16540390
– user16540390, Commented Dec 13, 2021 at 10:11
I'm not sure if this is a simple typo (missing () after the "IIFE") — user16540390
– user16540390, Commented Dec 13, 2021 at 10:17
You could avoid the async IIFE by updating your Node.js version. Current versions of Node.js support top-level await: v8.dev/features/top-level-await — user16540390
– user16540390, Commented Dec 13, 2021 at 10:21
If you don't have limit about adding dependencies, [async][1] library offer a very good way to manage these kind of stuff. In your case i thinks its suitable using the waterfall method, that enables you to use the result data from one call into another, as explained in the docs. [1]: caolan.github.io/async/v3/docs.html#waterfall — MatteoPHRE
– MatteoPHRE, Commented Dec 13, 2021 at 10:22

Jaime Blázquez · Accepted Answer · 2021-12-13 10:27:50Z

2

It's difficult to answer without knowing which of the functions you are calling are synchronous or not. But making the following assumptions:

readWriteToFile.writeToFile returns a Promise (or it is async)
mapData is defined elsewhere

You must:

Add a return statement to the last line of the first .then(...) block
As jabaa said, you have to actually invoke the function.

(async () => {
    
    await scraperA.scrap().then(data => {
            console.log("DATA", data);
            return readWriteToFile.writeToFile(data, 'scraped-data-'+timestamp, 'json');
        });
    
    await compare.compareData('./output/json/data.json', './output/json/scraped-data-'+timestamp+'.json').then(data => {
        let dataWithStockData = data;
        mapData = dataWithStock.map( obj => { 
    
            let finalObj = {
                manufacturer: obj.manufacturer,
                grade: obj.condition,
                stock: obj.stock,
            }
            return finalObj;
        });
    });

    await readWriteToFile.writeToFile(mapData, 'data-with-stock-'+timestamp, 'csv');

})();

answered Dec 13, 2021 at 10:27

Jaime Blázquez

3281 silver badge9 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Murlidhar Fichadia Over a year ago

sorry my bad, the "compare.compareData()" call is not async neither is "readWriteToFile.writeToFile()". let me show the code in other 2 files

Jaime Blázquez Over a year ago

Seeing the other code, while the functions are not asynchronous because of the content (inside there are only synchronous code), you have made them asynchronous because you marked them as async. As such, they return a Promise, and you must still await for them. And I think the solution (adding the return before the first call to writeToFile) is still relevant. Did it work? It's good to give feedback.

score 2 · Accepted Answer · 2021-12-13 11:52:03Z

The code structure

(async () => {
    
    /* your code */

});

suggests you want to create an async Immediately Invoked Function Expression (IIFE). But you have to invoke it (() at the end):

(async () => {
    
    /* your code */

})();

Using current versions of Node.js you don't need an async IIFE. Node.js supports top-level await.

Since readWriteToFile.writeToFile and compare.compareData don't return a promise, you can't call .then and await doesn't make sense.

Starting with Node.js v14.8, you can write

await scraperA.scrap().then(data => {
    console.log("DATA", data);
    readWriteToFile.writeToFile(data, 'scraped-data-'+timestamp, 'json');
});

const dataWithStockData = compare.compareData('./output/json/data.json', './output/json/scraped-data-'+timestamp+'.json')
mapData = dataWithStock.map( obj => { 

    let finalObj = {
        manufacturer: obj.manufacturer,
        grade: obj.condition,
        stock: obj.stock,
    }
    return finalObj;
});

readWriteToFile.writeToFile(mapData, 'data-with-stock-'+timestamp, 'csv');

After you fixed the problem, you could consider using either .then or await / async to clean up your code. Mixing them is usually considered bad code style.

const data = await scraperA.scrap();
console.log("DATA", data);
readWriteToFile.writeToFile(data, 'scraped-data-'+timestamp, 'json');

const dataWithStockData = compare.compareData('./output/json/data.json', './output/json/scraped-data-'+timestamp+'.json');

mapData = dataWithStock.map( obj => { 

  let finalObj = {
    manufacturer: obj.manufacturer,
    grade: obj.condition,
    stock: obj.stock,
  }
  return finalObj;
});

readWriteToFile.writeToFile(mapData, 'data-with-stock-'+timestamp, 'csv');

Collectives™ on Stack Overflow

async / await run async functions sequentially in nodejs, puppeteer

Problem

My Code - Index.js

My Code - compare.js

My Code - readWriteToFile.js

My solution

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

Problem

My Code - Index.js

My Code - compare.js

My Code - readWriteToFile.js

My solution

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related