Custom Formatting of JSON output using Spark

Question

I have a dataset with a bunch of BigDecimal values. I would like to output these records to a JSON file, but when I do the BigDecimal values will often be written with trailing zeros (123.4000000000000), but the spec we are must conform to does not allow this (for reasons I don't understand).

I am trying to see if there is a way to override how the data is printed to JSON. Currently, my best idea is to convert each record to a string using JACKSON and then writing the data using df.write().text(..) rather than JSON.

Duy Nguyen · Accepted Answer · 2019-02-11 17:14:46Z

1

I suggest to convert Decimal type to String before writing to JSON.

Below code is in Scala, but you can use it in Java easily

import org.apache.spark.sql.types.StringType

# COLUMN_NAME is your DataFrame column name.

val new_df = df.withColumn('COLUMN_NAME_TMP', df.COLUMN_NAME.cast(StringType)).drop('COLUMN_NAME').withColumnRenamed('COLUMN_NAME_TMP', 'COLUMN_NAME')

answered Feb 11, 2019 at 17:14

Duy Nguyen

1,0256 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Custom Formatting of JSON output using Spark

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related