How to split a JSON array to multiple JSONs using scala spark

Question

I have an array JSON as below format

{
  "marks": [
    {
      "subject": "Maths",
      "mark": "80"
    },
    {
      "subject": "Physics",
      "mark": "70"
    },
    {
      "subject": "Chemistry",
      "mark": "60"
    }
  ]
}

I need to split each array object as separate JSON files. Is there any way to do this in spark shell.

mck · Accepted Answer · 2021-06-18 11:52:00Z

1

You can explode the marks array of structs, add an ID column, and write JSON files partitioned by the unique ID column.

df.selectExpr("inline(marks)")
  .withColumn("id", monotonically_increasing_id)
  .repartition(col("id"))
  .write
  .partitionBy("id")
  .json("output")

answered Jun 18, 2021 at 11:52

mck

42.7k13 gold badges44 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

ss301 Over a year ago

Where is the df coming from ? What if I just have the json .

Collectives™ on Stack Overflow

How to split a JSON array to multiple JSONs using scala spark

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related