create map from dataframe in spark scala

Question

I have a json string as below in a dataframe

  aaa  |  bbb |  ccc |ddd | eee          
 --------------------------------------
   100 | xxxx |  123 |yyy|2017
   100 | yyyy |  345 |zzz|2017
   200 | rrrr |  500 |qqq|2017
   300 | uuuu |  200 |ttt|2017
   200 | iiii |  500 |ooo|2017

I want to get the result as

 {100,[{xxxx:{123,yyy}},{yyyy:{345,zzz}}],2017}
 {200,[{rrrr:{500,qqq}},{iiii:{500,ooo}}],2017}
 {300,[{uuuu:{200,ttt}}],2017}

Kindly help

Your title and question don't match at all.

philantrovert
– philantrovert

2017-09-06 12:33:51 +00:00
Commented Sep 6, 2017 at 12:33 — philantrovert
– philantrovert, Commented Sep 6, 2017 at 12:33
what should i mention

gayathri
– gayathri

2017-09-06 12:41:24 +00:00
Commented Sep 6, 2017 at 12:41 — gayathri
– gayathri, Commented Sep 6, 2017 at 12:41
The output that you're suggesting isn't a json.

vvg
– vvg

2017-09-06 12:53:50 +00:00
Commented Sep 6, 2017 at 12:53 — vvg
– vvg, Commented Sep 6, 2017 at 12:53

Bor Laze · Accepted Answer · 2019-08-10 08:03:08Z

4

This works:

 val df = data
    .withColumn("cd", array('ccc, 'ddd)) // create arrays of c and d
    .withColumn("valuesMap", map('bbb, 'cd)) // create mapping
    .withColumn("values", collect_list('valuesMap) // collect mappings
                 .over(Window.partitionBy('aaa)))
    .withColumn("eee", first('eee) // e is constant, just get first value of Window
                 .over(Window.partitionBy('aaa)))
   .select("aaa", "values", "eee") // select only columns that are in the question selected
   .select(to_json(struct("aaa", "values", "eee")).as("value")) // create JSON

Make sure you do

import org.apache.spark.sql.functions._
import org.apache.spark.sql.expressions._`

edited Aug 10, 2019 at 8:03

Bor Laze

2,52614 silver badges20 bronze badges

answered Sep 6, 2017 at 12:55

T. Gawęda

16.1k5 gold badges51 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

gayathri Over a year ago

Thanks Gaweda. But partitionby is not working. But now the requirement is changes. I have list as below

T. Gawęda Over a year ago

@gayathri What do you mean by "not working"? I've tested it on your data. If you want to have plain String list, you can do collect()

gayathri Over a year ago

Hi Gaweda, iam using IntelliJ(scala 2.10.6) .withColumn("valuesMap", map('bbb, 'cd)) , map is not recognized and aso over(Window.partitionBy('aaa)) not recognized. I have imported the sql.functions Pls help

T. Gawęda Over a year ago

map is inside functions object, so it should be visible

Ignacio Alorre · Accepted Answer · 2020-12-27 19:31:24Z

0

You can create a map defining the values as constants with lit() or taking them from other columns in the dataframe with $"col_name", like this:

val new_df = df.withColumn("map_feature", map(lit("key1"), lit("value1"), lit("key2"), $"col2"))

answered Dec 27, 2020 at 19:31

Ignacio Alorre

7,6558 gold badges65 silver badges104 bronze badges

Collectives™ on Stack Overflow

create map from dataframe in spark scala

2 Answers 2

4 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related