Convert Scala Dataframe to HashMap

Question

I have a scala dataframe like this

-----------------------------------
| Code | Name | Age | ID          |
-----------------------------------
| ABC  | Alan    | 22  | 111111   |
| ABC  | Bob     | 25  | 222222   |
| DEF  | Charlie | 29  | 333333   |
| GHI  | David   | 11  | 555555   |
-----------------------------------

I want to have an output HashMap like this:

{
 'ABC': [{'Name': 'Alan',    'Age': 22', 'ID': 111111} , {'Name': 'Bob', 'Age': 25', 'ID': 22222}], 
 'DEF': [{'Name': 'Charlie', 'Age': 29', 'ID': 333333}],
 'GHI': [{'Name': 'David',   'Age': 11', 'ID': 555555}] 
}

How can I efficiently do this?

that's probably a bad idea that defeat the spark design, work with rather than try to get out of — cchantep
– cchantep, Commented Aug 13, 2021 at 13:52

kanielc · Accepted Answer · 2021-08-13 12:56:05Z

1

Assuming your DataFrame is named ds, this should work:

ds.select('code, to_json(struct('name, 'age, 'id)) as "json")
  .groupBy('code).agg(collect_list('json))
  .as[(String, Array[String])]
  .collect.toMap

This will give you a Map[String, Array[String]]. If what you wanted was to turn the whole DataFrame into a single JSON, I wouldn't recommend that, but it would be doable as well.

answered Aug 13, 2021 at 12:56

kanielc

1,3221 gold badge12 silver badges15 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Convert Scala Dataframe to HashMap

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related