Spark: Converting WrappedArray of tuples to a string

Question

I have a Spark dataframe with column that has array<struct<_1:string,_2:string>> datatype and following sample data:

WrappedArray([Book1,Title1], [Book2,Title2], [Book3,Title3])

I want to convert this column from WrappedArray of tuples to a string

Here is the desired output:

Book1/Title1 Book2/Title2 Book3/Title3

I tried the following udf passing that column of the dataframe, but it did not work:

val unwraparr = udf ((x: mutable.WrappedArray[(String, String)]) => x.map { case (val1, val2) => val1 + "/" + val2 })

2 revs user6022341 · Accepted Answer · 2016-08-09 21:18:57Z

1

Try like this:

import org.apache.spark.sql.Row

(x: Seq[Row]) => x.map { case Row(val1: String, val2: String) => val1 + "/" + val2 })

edited Aug 9, 2016 at 21:18

community wiki

2 revs
user6022341

Sign up to request clarification or add additional context in comments.

2 Comments

user3803714 Over a year ago

Is this the complete UDF: val unwraparr = udf ((x: mutable.WrappedArray[(String, String)]) => x.map { case Row(val1: String, val2: String) => val1 + "/" + val2 })

Omkar Neogi Over a year ago

Is the way you accepted the input parameters to the udf correct? If there's an array of tuples, do you accept it as Seq[Row]?

Collectives™ on Stack Overflow

Spark: Converting WrappedArray of tuples to a string

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related