Read spark csv with empty values without converting to null

Question

Consider a csv:

Name,Color
Apple,""

 val df = spark.read
.option("header", "true")
.option("inferSchema", "true")
.option("treatEmptyValuesAsNulls","false")
.csv(mycsv)

This still gives:

+--------+----------+
|Name    |Color     |
+--------+----------+
|   Apple|      null|
+--------+----------+

Expected was:

+--------+----------+
|Name    |Color     |
+--------+----------+
|   Apple|          |
+--------+----------+

then how will we read empty string as empty without having it as null — supernatural
– supernatural, Commented Oct 12, 2020 at 12:39

Oli · Accepted Answer · 2020-10-12 19:28:51Z

6

AFAIK, the option "treatEmptyValuesAsNulls" does not exist. See the doc for more details. Two other options may be of interest to you though. emptyValue and nullValue. By default, they are both set to "" but since the null value is possible for any type, it is tested before the empty value that is only possible for string type. Therefore, empty strings are interpreted as null values by default. If you set nullValue to anything but "", like "null" or "none", empty strings will be read as empty strings and not as null values anymore.

edited Oct 12, 2020 at 19:28

answered Oct 12, 2020 at 12:48

Oli

10.5k5 gold badges31 silver badges51 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Read spark csv with empty values without converting to null

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related