Difference between columns of ArrayType in dataframe

Question

I have dataframe with 2 ArrayType columns. I want to find the difference between columns. column1 will always have values while column2 may have empty array. I created following udf but it is not working

df.show() gives following records

SampleData:

["Test", "Test1","Test3", "Test2"], ["Test", "Test1"]

Code:

sc.udf.register("diff", (value: Column,value1: Column)=>{ 
                        value.asInstanceOf[Seq[String]].diff(value1.asInstanceOf[Seq[String]])          
                    })

Output:

["Test2","Test3"]

Spark version 1.4.1 Any help will be appreciated.

what was the result ?

Ram Ghadiyaram
– Ram Ghadiyaram

2016-12-15 09:12:47 +00:00
Commented Dec 15, 2016 at 9:12 — Ram Ghadiyaram
– Ram Ghadiyaram, Commented Dec 15, 2016 at 9:12
it gives all values of value

undefined_variable
– undefined_variable

2016-12-15 09:13:59 +00:00
Commented Dec 15, 2016 at 9:13 — undefined_variable
– undefined_variable, Commented Dec 15, 2016 at 9:13
can you paste sample data pls? ideally it should work

Ram Ghadiyaram
– Ram Ghadiyaram

2016-12-15 09:15:23 +00:00
Commented Dec 15, 2016 at 9:15 — Ram Ghadiyaram
– Ram Ghadiyaram, Commented Dec 15, 2016 at 9:15
I hope you have used collection.SeqLike.diff

Ram Ghadiyaram
– Ram Ghadiyaram

2016-12-15 09:18:35 +00:00
Commented Dec 15, 2016 at 9:18 — Ram Ghadiyaram
– Ram Ghadiyaram, Commented Dec 15, 2016 at 9:18
Please share example data and expected output.

mtoto
– mtoto

2016-12-15 09:19:18 +00:00
Commented Dec 15, 2016 at 9:19 — mtoto
– mtoto, Commented Dec 15, 2016 at 9:19

mtoto · Accepted Answer · 2016-12-15 13:34:13Z

1

You need to change your udf to:

val diff_udf = udf { ( a:  Seq[String], 
                       b:  Seq[String]) => a diff b }

Then this works:

import org.apache.spark.sql.functions.col
df.withColumn("diff",
  diff_udf(col("col1"), col("col2"))).show
+--------------------+-----------------+------------------+
|                col1|             col2|              diff|
+--------------------+-----------------+------------------+
|List(Test, Test1,...|List(Test, Test1)|List(Test3, Test2)|
+--------------------+-----------------+------------------+

Data

val df = sc.parallelize(Seq((List("Test", "Test1","Test3", "Test2"), 
                             List("Test", "Test1")))).toDF("col1", "col2")

edited Dec 15, 2016 at 13:34

answered Dec 15, 2016 at 10:11

mtoto

24.3k4 gold badges62 silver badges74 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Community · Accepted Answer · 2020-06-20 09:12:55Z

column1 will always have values while column2 may have empty array.

your comment : it gives all values of value – undefined_variable

Example1 :

lets see small example like this...

   val A = Seq(1,1)

 A: Seq[Int] = List(1, 1)

 val B = Seq.empty

 B: Seq[Nothing] = List()
    
A diff B

 res0: Seq[Int] = List(1, 1)

if you do a collection.SeqLike.diff then you will get A value as shown in example. As per scala, this is very much valid case since you told you are always getting value which is seq.

Also, reverse case is like this...

 B diff A

 res1: Seq[Nothing] = List()

if you use Spark udf for doing above as well then same results will come.

EDIT : (if one array not empty case as you modified your example )

Example2 :

 val p = Seq("Test", "Test1","Test3", "Test2")

 p: Seq[String] = List(Test, Test1, Test3, Test2)

 val q = Seq("Test", "Test1")

 q: Seq[String] = List(Test, Test1)

 p diff q

 res2: Seq[String] = List(Test3, Test2)

This is what your expected output which is coming as given in your example.

Reverse case : I think this is what you are getting which is not expected by you.

q diff p

 res3: Seq[String] = List()

Collectives™ on Stack Overflow

Difference between columns of ArrayType in dataframe

2 Answers 2

Comments

Example1 :

EDIT : (if one array not empty case as you modified your example )

Example2 :

Reverse case : I think this is what you are getting which is not expected by you.

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Example1 :

EDIT : (if one array not empty case as you modified your example )

Example2 :

Reverse case : I think this is what you are getting which is not expected by you.

Comments

Your Answer

Sign up or log in

Post as a guest

Related