creating json structure from dataframe using pyspark

Question

I have dataframe which is product of left join. Now I want to create json structure.

I tried using different option but I was not able create it. Here is my dataframe :

Col1    col2    col3    col4
1111    name    null    null
1112    name1   abcd    def
1112    name1   DEFG    ABXC

The desired json structure is :

{col1 : 1111, col2 : name , detial: {col3:,col4:}}
{col1 : 1112, col2 : name1 , detial: {{col3:abcd,col4:def},{col3:DEFG,col4:ABXC}}}

Yayati Sule · Accepted Answer · 2019-05-28 20:00:02Z

1

I have created the data from your sample dataset and created a dataframe out of it.You can use the following code:

from pyspark.sql import Row, DataFrame,SparkSession
from pyspark.sql.functions import *
import json
spark = SparkSession.builder.getOrCreate()

data = spark.createDataFrame([Row(col1=1111,col2="name",col3=None,col4=None),Row(col1=1112,col2="name1",col3="abcd",col4="def"),Row(col1=1112,col2="name1",col3="DEFG",col4="ABXC")])

ndata = data.select(struct("col1","col2",struct("col3","col4").alias("details")).alias("data"))

dataList = list(map(json.loads,ndata.toJSON().collect()))

The above code returns the result as a list of dictionaries in python.

edited May 28, 2019 at 20:00

answered May 22, 2019 at 9:25

Yayati Sule

1,62914 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Yayati Sule Over a year ago

You need to import struct() from pyspark.sql package. from pyspark.sql import struct

Yayati Sule Over a year ago

@Shane If your query was answered can you accept my answer and close this thread

Shane Over a year ago

I tried your solution but I didnot got the desired out put which was {col1 : 1111, col2 : name , detial: {col3:,col4:}} {col1 : 1112, col2 : name1 , detial: {{col3:abcd,col4:def},{col3:DEFG,col4:ABXC}}}` and your code gives me

[u'{"data":{"col1":1111,"col2":"name","details":{}}}',  u'{"data":{"col1":1112,"col2":"name1","details":{"col3":"abcd","col4":"def"}}}', u'{"data":{"col1":1112,"col2":"name1","details":{"col3":"DEFG","col4":"ABXC"}}}']

Yayati Sule Over a year ago

I updated my code. You will now get a list of dictionaries where every dictionary is of the structure you desire.

Shane Over a year ago

Now I dont get any output at all... I guess i need to group by the col1 and col2 and values of the details i need to see how to combine.

|

Collectives™ on Stack Overflow

creating json structure from dataframe using pyspark

1 Answer 1

7 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related