I have a csv file in s3 but I have to append the data to that file whenever I call the function but i am not able to do that,
df = pd.DataFrame(data_list)
bytes_to_write = df.to_csv(None, header=None, index=False).encode()
file_name = "Words/word_dictionary.csv" # Not working the below line
s3_client.put_object(Body=bytes_to_write, Bucket='recengine', Key=file_name)
This code is directly replacing the data inside the file instead of appending, Any solution?
s3 has no append functionality. You need to read the file from s3, append the data in your code, then upload the complete file to the same key in s3.
Check this thread on the AWS forum for details
The code will probably look like:
df = pd.DataFrame(data_list)
bytes_to_write = df.to_csv(None, header=None, index=False).encode()
file_name = "Words/word_dictionary.csv"
# get the existing file
curent_data = s3_client.get_object(Bucket='recengine', Key=file_name)
# append
appended_data = current_data + bytes_to_write
# overwrite
s3_client.put_object(Body=appended_data, Bucket='recengine', Key=file_name)
You can try using aws data wrangler library from awslabs to append , overwrite csv dataset stored in s3. Check out their documentation and tutorials from here link
You can utilize the pandas concat function to append the data and then write the csv back to the S3 bucket:
from io import StringIO
import pandas as pd
# read current data from bucket as data frame
csv_obj = s3_client.get_object(Bucket=bucket, Key=key)
current_data = csv_obj['Body'].read().decode('utf-8')
current_df = pd.read_csv(StringIO(csv_string))
# append data
appended_data = pd.concat([current_df, new_df], ignore_index=True)
appended_data_encoded = appended_data.to_csv(None, index=False).encode('utf-8')
# write the appended data to s3 bucket
s3_client.put_object(Body=appended_data_encoded,Bucket=bucket, Key=key)
