AWS Api Gateway & CSV Upload Issue

Question

I am getting some meta-information / junk in CSV in lambda function. I need to remove that junk. If I directly save the file to s3 the junk is included. Could anyone guide me on how to remove this?

----------------------------362648820336892682391117   ***// remove this***
Content-Disposition: form-data; name="file"; filename="Book1.csv" ***// remove this***
Content-Type: text/csv ***// remove this***

o;?name,age // remove this o;?
andy,33
hello,34
----------------------------362648820336892682391117--  ***// remove this***

I can also upload directly to s3 using pre-signed URL however, that is not what I am looking for.

const AWS = require('aws-sdk');

exports.handler = async (event) => {
    try {
            console.log(JSON.stringify(event, 2, null));
            const data = new Buffer(event.body, 'base64');
            const text = data.toString('ascii');
            const s3 = new AWS.S3();
            const params = {Bucket: 'bucket', Key: 'key', Body: text};
        const d = await s3.upload(params).promise();
        return {
            statusCode: 200,
            body: JSON.stringify('uploaded successfully'),
        };
    } catch (e) {
        return {
            statusCode: 200,
            body: JSON.stringify('uploaded successfully'),
        };
    }
};

Thanks

Can you please post some code showing how you are saving the file into s3? — Josh Aguilar
– Josh Aguilar, Commented Feb 3, 2021 at 5:32

Josh Aguilar · Accepted Answer · 2021-02-03 08:06:57Z

1

I assume you are uploading the file using multipart/form-data. If so, you will need to do further processing of the request body. You can either do something very rudimentary like manually parsing the contents using regex or you could use a library like busboy which helps process HTML form data.

A quick example for your scenario could be something like this.

const Busboy = require('busboy');
const AWS = require('aws-sdk');

// This function uses busboy to process the event body and
// return an object containing the file data and other details.
const parseFile = (event) => new Promise((resolve, reject) => {
  let contentType = event.headers['content-type']
  if (!contentType) {
    contentType = event.headers['Content-Type'];
  }

  const busboy = new Busboy({ headers: { 'content-type': contentType } });
  const uploadedFile = {};
  busboy.on('file', (fieldname, file, filename, encoding, mimetype) => {
    file.on('data', data => {
      uploadedFile.data = data;
    });

    file.on('end', () => {
      uploadedFile.filename = filename;
      uploadedFile.contentType = mimetype;
    });
  });

  busboy.on('error', error => {
    reject(error);
  });

  busboy.on('finish', () => {
    resolve(uploadedFile);
  });

  busboy.write(event.body, event.isBase64Encoded ? 'base64' : 'binary');
  busboy.end();
});

exports.handler = async (event) => {
  // Use the parse function here
  const { data } = await parseFile(event);

  const s3 = new AWS.S3();
  const params = { Bucket: 'bucket', Key: 'key', Body: data };
  await s3.upload(params).promise();
  return {
    statusCode: 200,
    body: 'uploaded successfully',
  };
};

edited Feb 3, 2021 at 8:06

answered Feb 3, 2021 at 8:01

Josh Aguilar

2,2911 gold badge12 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Meet Over a year ago

thanks, this junk text does not come if I send it as binary in postman

Josh Aguilar Over a year ago

Can you then send it as a binary using your client app as well?

Meet Over a year ago

no, I don't think so, it's better to send it as multipart. Need to use busboy or multer. else directly upload to s3 and take action on s3 events, thanks

Josh Aguilar Over a year ago

Fair enough. If you think I've been able to answer your question, it would be great if you could mark it as accepted. :-)

Collectives™ on Stack Overflow

AWS Api Gateway & CSV Upload Issue

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related