Applying function to Spark Dataframe Column

Question

Coming from R, I am used to easily doing operations on columns. Is there any easy way to take this function that I've written in scala

def round_tenths_place( un_rounded:Double ) : Double = {
    val rounded = BigDecimal(un_rounded).setScale(1, BigDecimal.RoundingMode.HALF_UP).toDouble
    return rounded
}

And apply it to a one column of a dataframe - kind of what I hoped this would do:

 bid_results.withColumn("bid_price_bucket", round_tenths_place(bid_results("bid_price")) )

I haven't found any easy way and am struggling to figure out how to do this. There's got to be an easier way than converting the dataframe to and RDD and then selecting from rdd of rows to get the right field and mapping the function across all of the values, yeah? And also something more succinct creating a SQL table and then doing this with a sparkSQL UDF?

stackoverflow.com/questions/29109916/…

Jean Logeart
– Jean Logeart

2016-02-05 15:29:53 +00:00
Commented Feb 5, 2016 at 15:29 — Jean Logeart
– Jean Logeart, Commented Feb 5, 2016 at 15:29

JAG · Accepted Answer · 2020-10-27 06:39:05Z

22

You can define an UDF as follows:

val round_tenths_place_udf = udf(round_tenths_place _)
bid_results.withColumn(
  "bid_price_bucket", round_tenths_place_udf($"bid_price"))

although built-in Round expression is using exactly the same logic as your function and should be more than enough, not to mention much more efficient:

import org.apache.spark.sql.functions.round

bid_results.withColumn("bid_price_bucket", round($"bid_price", 1))

2 Comments

Nambu14 Over a year ago

How can i parametrized the $"bid_price" call? suppose I have the column name stored in a variable. e.g. val tg_column = "bid_price" and then something like $tg_column

greatvovan Over a year ago

This approach is causing NotSerializableException in Databricks. The only way seems to be writing an arrow function and wrapping into udf(). If I reference any other function, I am getting this exception.

Collectives™ on Stack Overflow

Applying function to Spark Dataframe Column

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related