Converting a python dataframe from AWS lambda to CSV in S3

Question

I am trying to convert a write a python Dataframe and store it in S3. I have added the required layers in Lambda (s3fs & fsspec) and have provided requisite permissions to lambda to write to s3. But not I am getting the below error:

"errorMessage": "module 's3fs' has no attribute 'S3FileSystem'"

Below are the relevant lines of my code:

s3 = boto3.client('s3')
df.to_csv('s3://buckets/<my-bukcet-name>/mydata.csv')

Any pointers on what could be the reason for this?

Regards, Dbeings

Mark B · Accepted Answer · 2022-03-06 16:06:19Z

4

Instead of including those layers, I would recommend including the Amazon provided AWS Data Wrangler layer.

Then you would use AWS Data Wrangler to write your dataframe directly to S3, like the following:

import awswrangler as wr

wr.s3.to_csv(df, 's3://buckets/<my-bukcet-name>/mydata.csv', index=False)

answered Mar 6, 2022 at 16:06

Mark B

203k27 gold badges339 silver badges332 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

dbeings Over a year ago

Thanks a lot, using wrangler for writing fixed it! The below code worked. wr.s3.to_csv(df, 's3://<my-bukcet-name>/mydata.csv', index=False)

Mark B Over a year ago

Please mark the answer as accepted if it fixed your problem. Thanks.

PandaPhi Over a year ago

This library is must-have for anyone extracting data and dumping it somewhere in AWS

Collectives™ on Stack Overflow

Converting a python dataframe from AWS lambda to CSV in S3

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related