Creating Json structure from Pyspark Dataframe

Question

I have dataframe which is product of left join. Now I want to create json structure.

I tried using different option but I was not able create it. Here is my dataframe :

col1    col2    col3    col4
1111    name    aaa     bbb
1111    name    ccc     ddd
1111    name    iii     kkk
1112    name1   abcd    def
1112    name1   DEFG    ABXC

The desired json structure is:

{col1: 1111, col2: name, details: [{col3: aaa, col4: bbb}, {col3: ccc, col4: ddd}, {col3: iii, col4: kkk}]},
{col1: 1112, col2: name1, details: [{col3: abcd, col4: def}, {col3: DEFG, col4: ABXC}]}

Adam Dukkon · Accepted Answer · 2020-10-01 18:03:53Z

1

you can do it like this:

import pyspark.sql.functions as f

df = df.withColumn("details", f.to_json(f.struct("col3", "col4")))
df = df.groupBy(*["col1", "col2"]).agg(f.collect_list("details").alias("details"))

df.write.format('json').save('/path/file_name.json')

answered Oct 1, 2020 at 18:03

Adam Dukkon

2931 silver badge6 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Creating Json structure from Pyspark Dataframe

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related