How to parse CSVs from s3 bucket to use in a javascript AWS Lambda function

Question

I'm trying to read in a CSV from an s3 bucket using the csvtojson library in AWS Lambda, but it's not working properly. Locally, my code works. But when I upload it to Lambda, it doesn't return anything. There are no errors in the Lambda console, so I'm having a hard time debugging. My code is below.

const AWS = require('aws-sdk');
const csvtojson = require('csvtojson');

const s3 = new AWS.S3();

const params = {
   Bucket: bucketName,
   Key: pathToFile 
};

const stream = s3.getObject(params).createReadStream();

csvtojson()
   .fromStream(stream)
   .then((json) => {
     console.log('Locally, this returns the CSV as JSON. On Lambda, it does not.');
   });

Does csvtojson not work on Lambda for some reason? Should I be using a different method to parse the CSV? Thanks!

Joey Kilpatrick · Accepted Answer · 2019-09-23 22:52:49Z

2

Your lambda is finishing before the Promise has completed. Replace the last section with this:

const json = await csvtojson().fromStream(stream);
console.log('Locally, this returns the CSV as JSON. On Lambda, it does not.');

answered Sep 23, 2019 at 22:52

Joey Kilpatrick

1,61212 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Cengizz · Accepted Answer · 2020-11-30 10:00:36Z

Use promise to solve this problem.

import * as AWS from 'aws-sdk';
const s3 = new AWS.S3();
const csv = require('@fast-csv/parse');


const params = {
    Bucket: 'bucket name',
    Key: 'uploads/img-links.csv'
};

const csvFile = s3.getObject(params).createReadStream();


let csvParsePromise = new Promise((resolve, reject) => {

    const parser = csv.parseStream(csvFile, { headers: true }).on("data", function (data) {
        parser.pause();  // can pause reading using this at a particular row
        console.log('One line from .csv >> ', data);
        parser.resume(); // to continue reading
    }).on("end", function () {
        resolve('csv parse process finished')
    }).on("error", function () {
        reject('csv parse process failed')
    });
});

try { await csvParsePromise; }
catch(err) {
    console.log('an error has occurred');
}```

Cedric Tanda · Accepted Answer · 2022-09-28 09:30:34Z

0

Get CSV content is simple
Try this

export async function getS3Object(
  bucket: string,
  key: string,
  s3: Pick<S3, "getObject">
): Promise<string> {
  const { Body } = await s3
    .getObject({
      Bucket: bucket,
      Key: key,
    })
    .promise();
  if (!Body) {
    throw new Error(`S3: Empty body for ${bucket} - ${key}`);
  }

  return Body.toString("utf-8");
}

answered Sep 28, 2022 at 9:30

Cedric Tanda

312 bronze badges

Collectives™ on Stack Overflow

How to parse CSVs from s3 bucket to use in a javascript AWS Lambda function

3 Answers 3

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related