Assigning Spark SQL function to variable

Question

I want to assign a spark SQL function to a variable. For example:

    val func = org.apache.spark.sql.functions.max(_)

Now, this is the way for doing partial functions. But when I'm doing this, I get this error:

  Cannot resolve overloaded method max

Searched online, couldn't find a solution. Does anyone have an idea?

Terry Dactyl · Accepted Answer · 2018-10-29 10:00:09Z

2

Note max is an overloaded function:

So either:

import org.apache.spark.sql.Column

scala> val func: Column => Column = org.apache.spark.sql.functions.max
func: org.apache.spark.sql.Column => org.apache.spark.sql.Column = <function1>

or

scala> val func: String => Column = org.apache.spark.sql.functions.max
func: String => org.apache.spark.sql.Column = <function1>

Out of interest why do you want to do this?

answered Oct 29, 2018 at 10:00

Terry Dactyl

1,86814 silver badges24 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

J. Doe Over a year ago

Thank you! Why does the annotation makes it to work? How is it related of max being overloaded function?

Terry Dactyl Over a year ago

So there are two definitions of max, Column => Column and String => Column Defining the type tells the compiler which one we are aliasing.

J. Doe Over a year ago

Ok, I see. and what about the " _ " parameter? Doesn't it need to be provided?

Terry Dactyl Over a year ago

It doesn't have to be provided in this case.

Collectives™ on Stack Overflow

Assigning Spark SQL function to variable

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related