25

I have a dynamic list which is created based on value of n.

n = 3
drop_lst = ['a' + str(i) for i in range(n)]
df.drop(drop_lst)

But the above is not working.

Note:

My use case requires a dynamic list.

If I just do the below without list it works

df.drop('a0','a1','a2')

How do I make drop function work with list?

Spark 2.2 doesn't seem to have this capability. Is there a way to make it work without using select()?

Improve this question

5 Answers 5

Reset to default
85

You can use the * operator to pass the contents of your list as arguments to drop():

df.drop(*drop_lst)
Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

Thanks! what does the * operator do? Whats its significance?
The star unpacks the contents of an iterator if you place it to its left, ie. it produces the individual elements of your list.
This does not work for me, it gives: TypeError: drop() takes exactly 2 arguments (92 given). I might have an old version?
To answer my own question: I just checked, and in my version (1.6.2), the list method described here does not work.
The solution works in python but not in scala for scala see the answer by @fox ghost beneath
14

You can give column name as comma separated list e.g.

df.drop("col1","col11","col21")
Improve this answer

Comments

4

This is how drop specified number of consecutive columns in scala:

val ll = dfwide.schema.names.slice(1,5)
dfwide.drop(ll:_*).show

slice take two parameters star index and end index.

Improve this answer

Comments

-1

Use simple loop:

for c in drop_lst:
   df = df.drop(c)
Improve this answer

1 Comment

A code-only answer is not high quality. While this code may be useful, you can improve it by saying why it works, how it works, when it should be used, and what its limitations are. Please edit your answer to include explanation and link to relevant documentation.
-6

You can use drop(*cols) 2 ways .

  1. df.drop('age').collect()
  2. df.drop(df.age).collect()

Check the official documentation DataFrame.drop

Improve this answer

1 Comment

what does .collect() do in this case?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.