How to obtain the output after parsing the csv using csv-parser?

Question

I am parsing a huge csv (1.2GB) using csvparser and trying to obtain certain column data from the parsed csv. I am trying to push the data to the array after processing, but only getting empty array as output. How can I fix this code?

var parse = require('csv-parse');

var output = []
var parser = parse({
   delimiter: '\t',
   columns: true
}, function(err, csvLine) {
   for (var l = 0; l < csvLine.length; l++) {
      output.push(csvLine[l].id)
   }
});

console.log(output)

fs.createReadStream('file.csv', {
   encoding: 'utf8'
}).pipe(parser);

The output at console.log(output) is always an empty array. Please help me solve this.

I tried to understand the post here - Save csv-parse output to a variable. But I was not able to understand and fix the code.

That looks asynchronous. Try moving console.log(output) inside the function(err, csvLine){} right after the for loop — adiga
– adiga, Commented Feb 2, 2019 at 9:46

Mukesh Fulwariya · Accepted Answer · 2019-02-02 10:24:07Z

2

Because of asynchronous. The line console.log(output) runs after the declaration of parser variable. At that time output variable has no values in it.

You have to access output variable after the loop when all data read from csv as given below.

    var parse = require('csv-parse');
    var fs = require('fs');
    var output = []
    var parser = parse({
       delimiter: ',',
       columns: true
    }, function(err, csvLine) {
       for (var l = 0; l < csvLine.length; l++) {
          output.push(csvLine[l].visitortype)
       }
       console.log(output)
    }); 
    fs.createReadStream('file.csv', {
       encoding: 'utf8'
    }).pipe(parser);

answered Feb 2, 2019 at 10:24

Mukesh Fulwariya

463 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

csvb Over a year ago

Thank you @mukesh. This worked. The console.log() should be always at the function level. Is it the same when we read multiple files too?

luthfianto · Accepted Answer · 2019-03-28 15:14:59Z

0

Use fast-csv to parse the file as ReadableStream, then use async iteration (for await) to process it line by line. It's simpler.

const csv = require("fast-csv")

let myList=[];

const csvStream = csv.fromPath(filename);
for await(const row of csvStream) {
    myList.push(row[1]); // get the second column
}

answered Mar 28, 2019 at 15:14

luthfianto

1,8543 gold badges25 silver badges40 bronze badges

Collectives™ on Stack Overflow

How to obtain the output after parsing the csv using csv-parser?

2 Answers 2

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related