Spark DataFrame - select list of columns using Java API

Question

I'm trying to select List of columns from the DataFrame using Java API.

Sample Java Code:

List<String> colList = Arrays.asList(new String[] { "column1", "column2", "column3" });
df.selectExpr((String[])colList.toArray()).show();

In Java API, I have to use selectExpr instead of select. Is there any other way of selecting list of columns using Java API.

But in Scala, I can do something like below.

Sample Scala Code:

val colList = List("column1", "column2", "column3")
df.select(colList.head, colList.tail: _*).show

Alper t. Turker · Accepted Answer · 2017-12-12 13:31:47Z

13

You can use array of String:

String[] colList =  { "column1", "column2", "column3" };
String first = colList[0];
String[] rest =  Arrays.copyOfRange(colList, 1, colList.length);

logData.select(first, rest);

or array of Column:

import static org.apache.spark.sql.functions.col;
import org.apache.spark.sql.Column;

Column[] colList =  { col("column1"), col("column2"), col("column3") };
logData.select(colList);

answered Dec 12, 2017 at 13:31

Alper t. Turker

35.3k9 gold badges89 silver badges118 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Shankar Over a year ago

first approach is not array of string, you are programmatically building first and rest and also second approach is again something we need to build it from the list of column names as String.

Dror Over a year ago

Thanks a lot! This is a lovely and straightforward way. NOT.

Prakash Raj Over a year ago

@Alper, nice. My requirement is, columns will change, say sometimes I get col1, col2 and sometimes I get col1,col2,col3. How to dynamically do this in second approach. because I need to add column alias also from params. appreciate your inputs on this.

Collectives™ on Stack Overflow

Spark DataFrame - select list of columns using Java API

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related