How to upload a data frame on AWS as CSV directly in python?

Question

i tried boto3 but no luck

import boto3
from botocore.exceptions import NoCredentialsError

ACCESS_KEY = 'access_key'
SECRET_KEY = 'secret_key'


def upload_to_aws(local_file, bucket, s3_file):
    s3 = boto3.client('s3', aws_access_key_id=ACCESS_KEY,
                      aws_secret_access_key=SECRET_KEY)

    try:
        s3.upload_file(local_file, bucket, s3_file)
        print("Upload Successful")
        return True
    except FileNotFoundError:
        print("The file was not found")
        return False
    except NoCredentialsError:
        print("Credentials not available")
        return False


uploaded = upload_to_aws('local_file', 'information-arch', 's3_file_name')

print("Done! with the uploud")

What errors do you get?

Marcin
– Marcin

2020-05-11 06:10:32 +00:00
Commented May 11, 2020 at 6:10 — Marcin
– Marcin, Commented May 11, 2020 at 6:10

Community · Accepted Answer · 2020-06-20 09:12:55Z

1

Hussein,

According to the boto3Documentation, you should upload your upload like this:

upload_file(Filename, Bucket, Key, ExtraArgs=None, Callback=None, Config=None)

Example:

import boto3
s3 = boto3.resource('s3')
s3.meta.client.upload_file('/tmp/hello.txt', 'mybucket', 'hello.txt')

Parameters

Filename (str) -- The path to the file to upload.

Bucket (str) -- The name of the bucket to upload to.

Key (str) -- The name of the key to upload to.

So on your upload_to_aws function called to pass the parameter like this way.

Thanks

edited Jun 20, 2020 at 9:12

CommunityBot

11 silver badge

answered May 11, 2020 at 6:24

Dipen Shah

2,4722 gold badges11 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Mayank Porwal · Accepted Answer · 2020-05-11 08:31:15Z

1

You can copy your dataframe directly to s3 like this:

Let's say you have a dataframe called df. You can use the to_csv option specifying your s3 path.

It will directly save the csv file on S3.

This works with pandas versions >= 0.24

df.to_csv(s3_path, index=False)

From pandas docs:

pandas now uses s3fs for handling S3 connections. This shouldn’t break any code. However, since s3fs is not a required dependency, you will need to install it separately, like boto in prior versions of pandas.

answered May 11, 2020 at 8:31

Mayank Porwal

34.2k9 gold badges45 silver badges65 bronze badges

Collectives™ on Stack Overflow

How to upload a data frame on AWS as CSV directly in python?

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related