Unable to append data to an existing csv file in AWS S3 using python

Question

I have a csv file in my s3 bucket and configured aws cli in my local machine. I want to append the data to that csv file whenever I call my python script, but i am not able to do that.

s3_client = boto3.client('s3')
df = pd.DataFrame(data_list)
bytes_to_write = df.to_csv(None, header=None, index=False).encode()
file_name = 'test.csv'
# get the existing file
current_data = s3_client.get_object(Bucket='test-bucket', Key=file_name)
# append
appended_data = current_data + bytes_to_write
# overwrite
s3_client.put_object(Body=appended_data, Bucket='test-bucket', Key=file_name)
enter code here

I have tried the above code but unfortunate couldn't complete the action, i got the following error

Traceback (most recent call last):
  File "script.py", line 17, in <module>
    appended_data = current_data + bytes_to_write
TypeError: unsupported operand type(s) for +: 'dict' and 'bytes'

Any solution for this? please help me!

What I can understand from this error log is that you don't have your current_data and bytes_to_write in the same format. You need to convert you current_data to data frames and then trying to upload it. — spectre009
– spectre009, Commented Jul 10, 2021 at 17:43
how can i convert to a data frame, I have tried this "df2 =pd.DataFrame(current_data)" but still having the same error — ADARSH K
– ADARSH K, Commented Jul 10, 2021 at 18:53

Antonin Riche · Accepted Answer · 2021-07-10 18:25:20Z

2

current_data = s3_client.get_object(Bucket='test-bucket', Key=file_name)

As described here, get_object return a dict

You are probably looking for

s3_client.get_object(Bucket='test-bucket', Key=file_name)['Body'].read().decode("utf-8")

answered Jul 10, 2021 at 18:25

Antonin Riche

5986 silver badges11 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

ADARSH K Over a year ago

Hi, thank you for the response I have changed the line "current_data = s3_client.get_object(Bucket='test-lambda-shyam', Key=file_name)['Body'].read().decode("utf-8")". but I got an error like this "Traceback (most recent call last): File "script.py", line 19, in <module> appended_data = current_data + bytes_to_write TypeError: can only concatenate str (not "bytes") to str"

Antonin Riche Over a year ago

.decode("utf-8") is transforming bytes to str - just remove it

Collectives™ on Stack Overflow

Unable to append data to an existing csv file in AWS S3 using python

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related