Dropping multiple columns from Spark dataframe by Iterating through the columns from a Scala List of Column names

Question

I have a dataframe which has columns around 400, I want to drop 100 columns as per my requirement. So i have created a Scala List of 100 column names. And then i want to iterate through a for loop to actually drop the column in each for loop iteration.

Below is the code.

final val dropList: List[String] = List("Col1","Col2",...."Col100”)

def drpColsfunc(inputDF: DataFrame): DataFrame = { 
    for (i <- 0 to dropList.length - 1) {
        val returnDF = inputDF.drop(dropList(i))
    }
    return returnDF
}

val test_df = drpColsfunc(input_dataframe) 

test_df.show(5)

I am getting a compile error that could not resolve "returnDF". Can anyone please help to fix this. — Ramesh
– Ramesh, Commented Sep 30, 2016 at 8:14
Please make the question self-contained. Why do you put part of your question in the comments? How do I ask a good question? — Martin Nyolt
– Martin Nyolt, Commented Sep 30, 2016 at 8:37
Please edit your question with the additional information you have added in the comments ! — eliasah
– eliasah, Commented Sep 30, 2016 at 8:38
@ Martin and Eliasah-- Done the changes in the question. Thanks — Ramesh
– Ramesh, Commented Sep 30, 2016 at 13:10
The issue that i was facing with above code is, I am getting a compile error that "could not resolve 'returnDF'". Can anyone please help to fix this. — Ramesh
– Ramesh, Commented Sep 30, 2016 at 14:25

Ricky McMaster · Accepted Answer · 2017-11-15 09:57:38Z

32

If you just want to do nothing more complex than dropping several named columns, as opposed to selecting them by a particular condition, you can simply do the following:

df.drop("colA", "colB", "colC")

answered Nov 15, 2017 at 9:57

Ricky McMaster

4,6772 gold badges26 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Ram Ghadiyaram · Accepted Answer · 2016-11-28 03:01:01Z

21

Answer:

val colsToRemove = Seq("colA", "colB", "colC", etc) 

val filteredDF = df.select(df.columns .filter(colName => !colsToRemove.contains(colName)) .map(colName => new Column(colName)): _*)

edited Nov 28, 2016 at 3:01

Ram Ghadiyaram

29.4k16 gold badges102 silver badges133 bronze badges

answered Oct 2, 2016 at 15:38

Ramesh

1,5939 gold badges27 silver badges39 bronze badges

3 Comments

Venkat Over a year ago

Works fine. Could you please elaborate significance of "_*" here ?

ivankeller Over a year ago

on : _* : if you know python it's similar to the unpack unary operator * you put in front of a list for example. The expression above before : _* is a sequence of Column (more precisely an Array[Column]), but (one form of) select expects a varargs field of Column, ie. a variable number of Column objects. See for example: alvinalexander.com/scala/…

Anthony Awuley Over a year ago

df.drop(colsToRemove : _*) this is a simpler|cleaner solution.

kalehmann · Accepted Answer · 2019-05-03 09:14:03Z

12

This should work fine :

val dropList : List[String]  |
val df : DataFrame  |
val test_df = df.drop(dropList : _*)

edited May 3, 2019 at 9:14

kalehmann

5,0896 gold badges29 silver badges37 bronze badges

answered May 3, 2019 at 8:48

mr59

1211 silver badge2 bronze badges

Comments

Fahad Siddiqui · Accepted Answer · 2019-01-02 09:00:36Z

4

You can just do,

def dropColumns(inputDF: DataFrame, dropList: List[String]): DataFrame = 
    dropList.foldLeft(inputDF)((df, col) => df.drop(col))

It will return you the DataFrame without the columns passed in dropList.

As an example (of what's happening behind the scene), let me put it this way.

scala> val list = List(0, 1, 2, 3, 4, 5, 6, 7)
list: List[Int] = List(0, 1, 2, 3, 4, 5, 6, 7)

scala> val removeThese = List(0, 2, 3)
removeThese: List[Int] = List(0, 2, 3)

scala> removeThese.foldLeft(list)((l, r) => l.filterNot(_ == r))
res2: List[Int] = List(1, 4, 5, 6, 7)

The returned list (in our case, map it to your DataFrame) is the latest filtered. After each fold, the latest is passed to the next function (_, _) => _.

edited Jan 2, 2019 at 9:00

answered Jan 2, 2019 at 8:54

Fahad Siddiqui

1,8491 gold badge19 silver badges43 bronze badges

Comments

Nikunj Kakadiya · Accepted Answer · 2021-04-12 09:31:22Z

You can use the drop operation to drop multiple columns. If you are having column names in the list that you need to drop than you can pass that using :_* after the column list variable and it would drop all the columns in the list that you pass.

Scala:

val df = Seq(("One","Two","Three"),("One","Two","Three"),("One","Two","Three")).toDF("Name","Name1","Name2")
val columnstoDrop = List("Name","Name1")
val df1 = df.drop(columnstoDrop:_*)

Python: In python you can use the * operator to do the same stuff.

data = [("One", "Two","Three"), ("One", "Two","Three"), ("One", "Two","Three")]
columns = ["Name","Name1","Name2"]
df = spark.sparkContext.parallelize(data).toDF(columns)
columnstoDrop = ["Name","Name1"]
df1 = df.drop(*columnstoDrop)

Now in df1 you would get the dataframe with only one column i.e Name2.

Collectives™ on Stack Overflow

Dropping multiple columns from Spark dataframe by Iterating through the columns from a Scala List of Column names

5 Answers 5

Comments

3 Comments

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

Comments

3 Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related