How to convert Array<String> to Array<Struct> in spark scala?

Question

I have an Array of JSON String, which I need to parse and convert into a struct.

transDf schema:

root
 |-- logs: array (nullable = true)
 |    |-- element: string (containsNull = true)

This is the code I tried

  val logsSchema = new ArrayType(spark.read.json(transDf.select("logs").as[String]).schema, true)
  transDf = transDf.withColumn("logs", from_json(col("logs"), logsSchema))

but the above thing only works for string -> struct but not for Array struct.

How can I convert the array for JSON string into Array<Struct> without knowing the schema of the JSON.

Can you print a few rows of your transDf dataframe? This will help in helping you :) — Koedlt
– Koedlt, Commented Dec 12, 2022 at 9:21
I don't know if it is ok to add a dynamic link in the question with a lot of data. So, here are the transDf.select("logs").show(3, false) without table borders appp.me/aMhbvo Hoping, this will be helpfull — Mohit Kumar
– Mohit Kumar, Commented Dec 12, 2022 at 10:15
I suppose that you have n json representations in the array. Is the number of elements fixed in the array? — Emiliano Martinez
– Emiliano Martinez, Commented Dec 12, 2022 at 10:23
More importantly, do all elements of the array share the same schema? I.e. can you posexplode it, and then apply schema generated from one row to all of them? — mazaneicha
– mazaneicha, Commented Dec 12, 2022 at 20:24

Mohana B C · Accepted Answer · 2022-12-12 19:29:43Z

1

You can schema_of_json function to get schema from JSON string and pass it to from_json function get struct type.

  val logsSchema = schema_of_json(transDf.select(col("logs").cast("string")).as[String].first())
  transDf = transDf.withColumn("logs", from_json(col("logs"), logsSchema))

edited Dec 12, 2022 at 19:29

answered Dec 12, 2022 at 12:03

Mohana B C

5,4721 gold badge13 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Mohit Kumar Over a year ago

.as[String] throws error AnalysisException: Cannot up cast e.logs from array<string> to string.

Mohana B C Over a year ago

If you want to go ahead with this approach you can change the the type of array column to string.

Mohit Kumar Over a year ago

Then in the second line also I have to cast as string that will increase the compuataion. isn't it?

Collectives™ on Stack Overflow

How to convert Array<String> to Array<Struct> in spark scala?

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related