how to output multiple (key,value) in spark map function

Question

The format of input data likes below:

+--------------------+-------------+--------------------+
|           StudentID|       Right |             Wrong  |
+--------------------+-------------+--------------------+
|       studentNo01  |       a,b,c |            x,y,z   |
+--------------------+-------------+--------------------+
|       studentNo02  |         c,d |              v,w   |
+--------------------+-------------+--------------------+

And the format of output likes below():

+--------------------+---------+
|           key      |    value|
+--------------------+---------+
|     studentNo01,a  |       1 |
+--------------------+---------+
|     studentNo01,b  |       1 |
+--------------------+---------+
|     studentNo01,c  |       1 | 
+--------------------+---------+
|     studentNo01,x  |       0 | 
+--------------------+---------+
|     studentNo01,y  |       0 | 
+--------------------+---------+
|     studentNo01,z  |       0 | 
+--------------------+---------+
|     studentNo02,c  |       1 | 
+--------------------+---------+
|     studentNo02,d  |       1 | 
+--------------------+---------+
|     studentNo02,v  |       0 | 
+--------------------+---------+
|     studentNo02,w  |       0 | 
+--------------------+---------+

The Right means 1 , The Wrong means 0.

I want to process these data using Spark map function or udf, But I don't know how to deal with it . Can you help me, please? Thank you.

Do you want a data frame input and a RDD output?

Wilmerton
– Wilmerton

2016-10-13 11:55:18 +00:00
Commented Oct 13, 2016 at 11:55 — Wilmerton
– Wilmerton, Commented Oct 13, 2016 at 11:55

Wilmerton · Accepted Answer · 2016-10-13 12:00:25Z

Use split and explode twice and do the union

val df = List(
  ("studentNo01","a,b,c","x,y,z"),
  ("studentNo02","c,d","v,w")
  ).toDF("StudenID","Right","Wrong")

+-----------+-----+-----+
|   StudenID|Right|Wrong|
+-----------+-----+-----+
|studentNo01|a,b,c|x,y,z|
|studentNo02|  c,d|  v,w|
+-----------+-----+-----+


val pair = (
  df.select('StudenID,explode(split('Right,",")))
    .select(concat_ws(",",'StudenID,'col).as("key"))
    .withColumn("value",lit(1))
).unionAll(
  df.select('StudenID,explode(split('Wrong,",")))
    .select(concat_ws(",",'StudenID,'col).as("key"))
    .withColumn("value",lit(0))
)


+-------------+-----+
|          key|value|
+-------------+-----+
|studentNo01,a|    1|
|studentNo01,b|    1|
|studentNo01,c|    1|
|studentNo02,c|    1|
|studentNo02,d|    1|
|studentNo01,x|    0|
|studentNo01,y|    0|
|studentNo01,z|    0|
|studentNo02,v|    0|
|studentNo02,w|    0|
+-------------+-----+

You can convert to RDD as follows

val rdd = pair.map(r => (r.getString(0),r.getInt(1)))

Collectives™ on Stack Overflow

how to output multiple (key,value) in spark map function

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related