Selecting columns of Dataframe in Spark Scala

Question

If you want to select the first column of a dataframe this can be done:

df.select(df.columns(0))

df.columns(0) returns a string, so by giving the name of the column, the select is able to get the column correctly.

Now, suppose I want to select the first 3 columns of the dataset, this is what I would intuitively do:

df.select(df.columns.split(0,3):_*)

The _* operator would pass the array of strings as a varag to my understanding, and it would be the same as passing (df.column(1), df.column(2), df.column(3)) to the select statement. However this doesn't work and it is necessary to do this:

import org.apache.spark.sql.functions.col
df.select(sf.columns.split(0,3).map(i => col(i)):_*))

What is going on?

qaziqarta · Accepted Answer · 2022-09-02 10:12:36Z

I think in the question you meant slice instead of split.

And as for your question, df.columns.slice(0,3):_* is meant to be passed to functions with *-parameters (varargs), i.e. if you call select(columns:_*) then there must be a function defined with varargs, e.g. def select(cols: String*).

But there can only be one such function defined - no overloading here is possible. Example on why it's not possible to define two different functions with same vararg-parameter declaration:

def select(cols: String*): String = "string"
select() // returns "string"
def select(cols: Column*): Int = 3
select() // now returns 3

And in Spark, that one function is defined not for Strings but for Columns:

def select(cols: Column*)

For Strings, the method is declared like this:

def select(col: String, cols: String*)

I suggest you to stick to Columns, like you do now, but with some syntax sugar:

df.select(df.columns.slice(0,3).map(col):_*))

Or if there's a need to pass column names as Strings, then you can use selectExpr:

df.selectExpr(df.columns.slice(0,3):_*)

Collectives™ on Stack Overflow

Selecting columns of Dataframe in Spark Scala

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related