I'm trying to export a pandas dataframe to a csv file in a bucket on my google cloud storage, but the following code obviously isn't working for me:
my_df.to_csv(StringIO(file_io.FileIO('gs://mybucket/data/file.csv', mode='w+')))
How should this be rewritten? I'm getting the following error:
unbound method write() must be called within FileIO instance as first argument (got nothing instead)
Apologies if the answer is obvious, but I'm just starting to learn python.
Importing a file from gcloud to dataframe works when I code thus:
from tensorflow.python.lib.io import file_io
from pandas.compat import StringIO
import pandas as pd
def read_data(gcs_path):
file_stream = file_io.FileIO(gcs_path, mode='r')
data = pd.read_csv(StringIO(file_stream.read()), names=['various', 'column', 'names'])
return data
my_df = read_data('gs://mybucket/data/file.csv')
But I haven't been able to reverse the process.
GCS has worked for me, however:
import google.cloud.storage as gcs
client = gcs.Client()
bucket = client.bucket('my-bucket')
blobs = list(bucket.list_blobs(prefix='data/'))
my_df.to_csv('tmp.csv')
local_tmp_path = ('tmp.csv')
target_blob = bucket.blob('data/file.csv')
target_blob.upload_from_file(open(local_tmp_path, 'r'))
blobs?You can save your csv file in your VM and then use gsutil to save it on your bucket.
Python:
my_df.to_csv("data.csv")
Shell:
gsutil data.csv gs://my_bucket/