Python script to load data from AWS S3 to Redshift

Question

Has anybody worked on creating a python script to load data from s3 to redshift tables for multiple files. How can we acheive it in AWS CLI. Your learnings and inputs on the same is appreciated.

just wrap a REDSHIFT COPY command in python script.thats it

theDbGuy
– theDbGuy

2018-09-17 13:16:14 +00:00
Commented Sep 17, 2018 at 13:16 — theDbGuy
– theDbGuy, Commented Sep 17, 2018 at 13:16

John Rotenstein · Accepted Answer · 2018-09-17 14:29:19Z

2

The COPY command is the best way to load data from Amazon S3 to Amazon Redshift. It can load multiple files in parallel into the one table.

Use any Python library (eg PostgreSQL + Python | Psycopg) to connect to Amazon Redshift, then issue the COPY command.

The AWS Command-Line Interface (CLI) does not have the ability to run the COPY command on Redshift because it needs to be issued to the database, while the AWS CLI issues commands to AWS. (The AWS CLI can be used to launch/terminate a Redshift cluster, but not to connect to the cluster itself.)

answered Sep 17, 2018 at 14:29

John Rotenstein

273k28 gold badges456 silver badges541 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python script to load data from AWS S3 to Redshift

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related