spark scala - convert json string to json struct

Question

I have a dataframe with schema like below

 |-- aaa: string (nullable = true)
 |-- bbb: long (nullable = false)
 |-- ccc: string (nullable = true)
 |-- ddd: string (nullable = true)
 |-- eee: long (nullable = false)

Already i have results in val for the above .

I want to convert the schema as below using spark scala

  |ddd:struct (nullable = true)
      |-- aaa: string (nullable = true)
      |-- bbb: string (nullable = true)
      |-- ccc: string (nullable = true)
      |-- eee: string (nullable = true)

Please help

Welcome to stackoverflow. Please give a more detailed description of your question. Include code when possible. — Sohum Sachdev
– Sohum Sachdev, Commented Sep 1, 2017 at 13:00

Anahcolus · Accepted Answer · 2017-08-31 07:21:35Z

2

You should use struct function

the first method is to use withColumn api and if you don't want the original columns you can drop them as

import org.apache.spark.sql.functions._
df.withColumn("ddd", struct(col("aaa"), col("bbb").cast(StringType).as("bbb"), col("ccc"), col("eee").cast(StringType).as("eee"))).drop("aaa", "bbb", "ccc", "eee")

the second way is to use select api as

df.select(struct(col("aaa"), col("bbb").cast(StringType).as("bbb"), col("ccc"), col("eee").cast(StringType).as("eee")).as("ddd"))

edited Aug 31, 2017 at 7:21

answered Aug 31, 2017 at 4:44

Anahcolus

42.1k6 gold badges75 silver badges101 bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

gayathri Over a year ago

Hi Harsh Thanks for your reply

gayathri Over a year ago

But it is appending

gayathri Over a year ago

i Have to create a new schema as above

Anahcolus Over a year ago

it does create a new schema :) see my updated answer

Anahcolus Over a year ago

and i am not Harsh @gayathri :P

|

Collectives™ on Stack Overflow

spark scala - convert json string to json struct

1 Answer 1

8 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

8 Comments

Your Answer

Sign up or log in

Post as a guest

Related