Writing CSV file using Spark and scala - empty quotes instead of Null values

Question

I'm using spark 2.4.1 and scala, and trying to write DF to csv file. it seems that in case of null values ,the csv contains "". Is it possible to remove those empty quotes?

 val data = Seq(
      Row(1, "a"),
      Row(5, "z"),
      Row(5, null)
    )

    val schema = StructType(
      List(
        StructField("num", IntegerType, true),
        StructField("letter", StringType, true)
      )
    )

    var df = spark.createDataFrame(
      spark.sparkContext.parallelize(data),
      schema
    )
  df.write.csv("location/")

The output seems like:

1,a
5,z
5,""

And I want it will be:

1,a
5,z
5,

What should I do?

Thanks!

OldWolfs · Accepted Answer · 2022-06-10 20:09:50Z

9

You can use options of the writer see CSV specific options(SaveMode is not related to answer);

 df.write
   .option("nullValue", null)
   .mode(SaveMode.Overwrite)
   .csv("location/")

edited Jun 10, 2022 at 20:09

answered Aug 1, 2019 at 15:59

OldWolfs

6266 silver badges15 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

prosoitos · Accepted Answer · 2020-09-28 23:20:19Z

-1

Try this:

df.write.option("nullValue",None).save("location/")

edited Sep 28, 2020 at 23:20

prosoitos

7,6936 gold badges32 silver badges45 bronze badges

answered Sep 28, 2020 at 14:07

Lucas Vinicius Teodoro

111 bronze badge

1 Comment

Ivo Mori Over a year ago

Please include an explanation for your code as it makes your answer more helpful to future readers in general. In particular, it'd be great if you extend this answer with an explanation why this solution is a better approach than the one given in the accepted answer. See also the contribution guide for more details.

Collectives™ on Stack Overflow

Writing CSV file using Spark and scala - empty quotes instead of Null values

2 Answers 2

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related