Spark - Write DataFrame with custom file name [duplicate]

Question

I have a Spark (2.4) DataFrame that I want to write as a Pipe separated file. It should be pretty straightforward like so

val myDF = spark.table("mySchema.myTable")
myDF.coalesce(1).write.format("csv").options("header", "true").options("delimiter", "|").save("/tmp/myDF")

I get a part-*.csv file in /tmp/myDF.

So far, so good. But I actually want the file name to be something specific, e.g. /tmp/myDF.csv

But giving this String in save will just create a dir called myDF.csv and create the part*.csv file in there.

Is there a way to write the DataFrame with a specific name?

SCouto · Accepted Answer · 2021-04-29 11:18:26Z

1

You can't do that with Spark

You can rename the file later accessing the fileSystem

val directory = new File(/tmp/myDF)

if (directory.exists && directory.isDirectory) {
   val file = directory.listFiles.filter(_.getName.endsWith(".csv")).head
   file.renameTo("myDF.csv")
}

answered Apr 29, 2021 at 11:18

SCouto

7,9465 gold badges37 silver badges52 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Spark - Write DataFrame with custom file name [duplicate]

1 Answer 1

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Linked

Related