how to use a custom function in query on Spark Dataframe using Scala

Question

I load data from database to Spark Dataframe,named DF,then I must to extract some records from the Dataframe which their ID has special condition. So, I define this function:

def hash_id(id:String): Int = {
val two_char = id.takeRight(2).toInt
val hash_result = two_char % 4
return hash_result
 }

Then, I use the function in this query:

DF.filter(hash_id("ID")===3)

But I receive this error:

value === is not a member of Int

DF has ID column.

Would you please guide me how to use a custom function in where/filter clause?

Any help would be really appreciated.

Vincent Doba · Accepted Answer · 2021-09-21 14:42:15Z

1

=== can only be used between Column objects. That's why you have an error value === is not a member of Int, as return type of your function hash_id is an Int, not a Column

To be able to use your function, you should convert it to an user-defined function and apply this function to a column object as follow:

import org.apache.spark.sql.functions.{col, udf}

def hash_id(id:String): Int = {
  val two_char = id.takeRight(2).toInt
  val hash_result = two_char % 4
  return hash_result
}

val hash_id_udf = udf((id: String) => hasd_id(id))

DF.filter(hash_id_udf(col("ID")) === 3)

edited Sep 21, 2021 at 14:42

answered Sep 21, 2021 at 14:07

Vincent Doba

5,1683 gold badges28 silver badges49 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

how to use a custom function in query on Spark Dataframe using Scala

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related