Writing databricks dataframe to S3 using python

Question

I have a databricks data frame called df. I want to write it to a S3 bucket as a csv file. I have the S3 bucket name and other credentials. I checked the online documentation given here https://docs.databricks.com/spark/latest/data-sources/aws/amazon-s3.html#mount-aws-s3 and it says to use following commands

dbutils.fs.mount(s"s3a://$AccessKey:$SecretKey@$AwsBucketName", s"/mnt/$MountName", "sse-s3")

dbutils.fs.put(s"/mnt/$MountName", "<file content>")

But what I have is a dataframe and not a file. How can I achieve it?

try df.write.csv('/mnt/mountname/filename.csv') or for more options see spark.apache.org/docs/latest/… — Laurens Koppenol
– Laurens Koppenol, Commented Aug 29, 2019 at 5:39

Antoine Krajnc · Accepted Answer · 2020-01-09 18:43:03Z

5

I had the same problem. I found two solutions

1srt

df
.write \
.format("com.databricks.spark.csv") \
.option("header", "true") \
.save("s3a://{}:{}@{}/{}".format(ACCESS_KEY, SECRET_KEY, BUCKET_NAME, DIRECTORY)))

Worked like a charm.

2nd

You can indeed mount an S3 Bucket and then write a file to it directly like this :

#### MOUNT AND READ S3 FILES
AWS_BUCKET_NAME = "your-bucket-name"
MOUNT_NAME = "a-directory-name"
dbutils.fs.mount("s3a://%s" % AWS_BUCKET_NAME, "/mnt/%s" % MOUNT_NAME)
display(dbutils.fs.ls("/mnt/%s" % MOUNT_NAME))

#### WRITE FILE 

df.write.save('/mnt/{}/{}'.format(MOUNT_NAME, "another-directory-name"), format='csv')

This is also going to sync to your S3 Bucket.

answered Jan 9, 2020 at 18:43

Antoine Krajnc

1,33315 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

dinu0101 Over a year ago

this line is missing that why unable to connect encoded_secret_key = secret_key.replace("/", "%2F"). After applying, successfully connected. It is mentioned on docs.databricks.com/spark/latest/data-sources/aws/…

Collectives™ on Stack Overflow

Writing databricks dataframe to S3 using python

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related