Spark Scala Dynamic column selection from DataFrame

Question

I have a DataFrame which have different type of columns. Among those column, i need to retrieve specific column from that DataFrame. Hard coded DataFrame select statement will be like this:

val logRegrDF = myDF.select(myDF("LEBEL_COLUMN").as("label"),
col("FEATURE_COL1"), col("FEATURE_COL2"), col("FEATURE_COL3"), col("FEATURE_COL4"))

Where LEBEL_COLUMN and FEATURE_COLs will be dynamic. I have Array or Seq for those FEATURE Columns like this:

val FEATURE_COL_ARR = Array("FEATURE_COL1","FEATURE_COL2","FEATURE_COL3","FEATURE_COL4")

I need to use this Array of column collection with that SELECT statement in the 2nd part. In the select, 1st column will be one (LABEL_COLUMN) and rest will be dynamic list.

Can you please help me to make the select statement working in SCALA.

Note: The sample code given bellow is working, but i need to add column array in the 2nd part of the SELECT

val colNames = FEATURE_COL_ARR.map(name => col(name))
val logRegrDF = myDF.select(colNames:_*)  // it is not the requirement

I am thinking for 2nd part code will be like this, but it is not working:

val logRegrDF = myDF.select(myDF("LEBEL_COLUMN").as("label"), colNames:_*)

koiralo · Accepted Answer · 2017-07-27 11:52:04Z

2

If I understand your question, I hope this is what you are looking for

val allColumnsArr = "LEBEL_COLUMN" +: FEATURE_COL_ARR
result.select("LEBEL_COLUMN", allColumnsArr: _*)
  .withColumnRenamed("LEBEL_COLUMN", "label")

Hope this helps!

edited Jul 27, 2017 at 11:52

answered Jul 27, 2017 at 10:12

koiralo

23.2k6 gold badges57 silver badges77 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Yousuf Zaman Over a year ago

Hi @Shankar, Thanks a lot. Though your given suggestion is not working, but i got idea from your suggestion and solved the issue by this way val allColumnsArr = "LEBEL_COLUMN" +: FEATURE_COLUMNS val colNames = allColumnsArr.map(name => col(name)) myDF.select(colNames:_*).withColumnRenamed("LEBEL_COLUMN", "label")

koiralo Over a year ago

Updated the answer, please check. You don't need to columns, the strings are also accepted as arguments.

Yousuf Zaman Over a year ago

Yes this works. result.select("LEBEL_COLUMN", FEATURE_COL_ARR: _*) .withColumnRenamed("LEBEL_COLUMN", "label")

Yousuf Zaman · Accepted Answer · 2017-07-27 12:20:58Z

1

Thanks a lot @Shankar.

Though your given suggestion is not working, but i got an idea from your suggestion and solved the issue by this way

val allColumnsArr = "LEBEL_COLUMN" +: FEATURE_COL_ARR
val colNames = allColumnsArr.map(name => col(name)) 
myDF.select(colNames:_*).withColumnRenamed("LEBEL_COLUMN", "label")

Also this way without creating DataFrame column:

result.select(LEBEL_COLUMN, FEATURE_COL_ARR: _*) .withColumnRenamed(LEBEL_COLUMN, "label")

edited Jul 27, 2017 at 12:20

answered Jul 27, 2017 at 10:37

Yousuf Zaman

1911 gold badge5 silver badges11 bronze badges

1 Comment

Yousuf Zaman Over a year ago

result.select("LEBEL_COLUMN", FEATURE_COL_ARR: _*) .withColumnRenamed("LEBEL_COLUMN", "label") It woks this way also.

Collectives™ on Stack Overflow

Spark Scala Dynamic column selection from DataFrame

2 Answers 2

3 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related