How to read Data from CSV file column wise in nodejs

Question

I've a CSV file, with number of columns (containing headers as column ID), under each column there are set of rows (contains data for each column) but not each column's rows are equal (eg: col1, could have 4 rows .. but col2 could have 8 rows.)

what i want to do i'm trying to read the data of each column and save it in a list to use it later in some processing.

I'm trying to use CSV-parser but i couldn't figure out how to access the data of a specific column only.

here is a sample of the csv

1 ,2
How do I change my password? ,Why we use bottels?
How can I change my password? ,Why you're lazy?
How do I reset my password? ,Why do I get the message that the name of my APK is in use?
How to do a password change?    
How do I do a password change?  
How can a password be changed?

I've been trying to this so far

fs.createReadStream('test.csv')
    .pipe(csv())
    .on('data', (row) => {
        //console.log('New row ',row);

        if (columns === null) {
            columns = [];
        
            Object.keys(row).forEach(function (c) {
                console.log(c) // this print headers only
                //columns.push(c);
            })
        
        }
        Object.entries(row).forEach((r)=>{
            console.log(r) // this prints the entire objects data

        })

What i want at the end is to have an array of arrays containing each columns data in a separate arr, (eg; Arr = [ [col1 data(6 rows)] , [col2 data(3 rows)] ]

My real csv file have a like 10000 column and in future it might be bigger.

Is your actual CSV like that? It should have commas between the sentences... — Alex Michailidis
– Alex Michailidis, Commented Jun 23, 2020 at 12:41
You should post the actual CSV (a part of it), so someone could do some tests with your code. — Alex Michailidis
– Alex Michailidis, Commented Jun 23, 2020 at 12:42
@AlexMichailidis first of all thank you for your help. Second, as i understand if i want to write whatever data to a CSV file i should pre-define columns and sent the data to these columns in stream.write(), but i was wondering can i make it column dynamically created, as whenever a new column is created in my list it should be created in the csv. (i am kinda new to JS) ( and i think i should open a question for this :d ?) — Doom
– Doom, Commented Jun 23, 2020 at 13:26

Alex Michailidis · Accepted Answer · 2020-06-23 12:56:11Z

1

Check below my approach to your problem.

const csv = require('csv-parser')
const fs = require('fs')
const results = [];

fs.createReadStream('data.csv')
  .pipe(csv())
  .on('data', (data) => {
    /* data would be something like :
        { '2': 'Why we use bottels?', '1 ': 'How do I change my password? ' }
    */
    Object.entries(data)
      .forEach(([key, value]) => {
        // key would be the column number (1 or 2)
        // value would be the data of the row
        // we "abuse" the fact that the column happens to be a number between 1 and 2 and we use that as the array index
        let index = parseInt(key) - 1;
        results[index] = results[index] || []
        results[index].push(value)
      })
  })
  .on('end', () => {
    console.log(results);
  });

answered Jun 23, 2020 at 12:56

Alex Michailidis

4,1531 gold badge18 silver badges35 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Alex Michailidis · Accepted Answer · 2020-06-23 13:09:23Z

Here is another approach that makes the necessary transformations only in the end in contrast to the previous one that made transformations on each row.

Although both of these solutions should work, I would do some benchmark tests and choose the one that performs better.

const csv = require('csv-parser')
const fs = require('fs')
let results = [];

fs.createReadStream('aou.csv')
  .pipe(csv({
    headers: false,
    skipLines: 1,
  }))
  .on('data', (data) => results.push(data)) // simply push the data in the results array
  .on('end', () => {
    // transform the results array into the desired format
    results = results.reduce((prev, curr) => {
      if (curr[0]) prev[0].push(curr[0])
      if (curr[1]) prev[1].push(curr[1])
      return prev
    }, [ [], [] ])
    
  });

Collectives™ on Stack Overflow

How to read Data from CSV file column wise in nodejs

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related