COPY csv data into Redshift using LAMBDA

Question

I'd like to create a Lambda function that COPY on Redshift the data of a file PUT in a specific S3 Bucket, but I can't figure how to do that.

So far, I created a LAMBDA function that triggers whenever a .csv file is PUT on the S3 bucket and I managed to COPY the data from a local .csv file to Redshift.

Now I'd like to be helped on how to COPY the data using a Lambda function. I searched on the internet, but can't manage to have proper examples using Lambda.

Amazon provides a work solution for this: AWS Lambda Redshift Loader. — Jonathan
– Jonathan, Commented Jun 28, 2017 at 11:47
@Jonathan I've started to work with that already, should have mentionned it in the original post. I was looking for a "simpler" solution, since I will copy like 2 or 3 files per week, I do not need such a huge processing system for that. But thank you ! I am currently close to the solution using AWS Lambda Redshift Loader, still having some issues I can't deal with, but I'll manage to. — Prototype
– Prototype, Commented Jun 30, 2017 at 7:34
I understand, hopefully you can just take the parts you need from the example. Good luck. — Jonathan
– Jonathan, Commented Jun 30, 2017 at 8:14
Welp, managed to do what I wanted using another method : g14n.info/2016/04/s3-to-redshift-loader It works fantastically great and managed to deal with a lot of files ( I tried with 200 files at once, and every data has been processed. ) EDIT : Should I put this as an answear of my own question in order to help ? — Prototype
– Prototype, Commented Jun 30, 2017 at 9:32

CrazyNew · Accepted Answer · 2019-06-28 08:34:05Z

1

I use powershell to export data from SQL Server and upload it to an S3 bucket. I then have a Lambda function with an S3 put trigger that executes a stored procedure in Redshift that contains the copy command in a dynamic statement, in order to load data into different tables, and it will trigger's every time a file is uploaded, so multiple tables multiple times.

answered Jun 28, 2019 at 8:34

CrazyNew

311 bronze badge

Sign up to request clarification or add additional context in comments.

1 Comment

sam Over a year ago

How is the performance. When Copy command executed in Redshift stored procedure. I couldnt see any article in AWS site. appreciate your feedback

diptiranjan pradhan · Accepted Answer · 2018-03-14 07:55:45Z

0

The solution approach-

Lambda trigger works if you placed any file in S3

The data pipeline automated using below -

Flatfile - > S3 -> dynamodb copy comand - Lamdbda execute the script inside dynamodb - > load data on Redshift

answered Mar 14, 2018 at 7:55

diptiranjan pradhan

814 silver badges12 bronze badges

Collectives™ on Stack Overflow

COPY csv data into Redshift using LAMBDA

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related