Pyspark - Loop over dataframe columns by list

Question

New to pyspark. Just trying to simply loop over columns that exist in a variable list. This is what I've tried, but doesn't work.

column_list = ['colA','colB','colC']
for col in df:
   if col in column_list:
      df = df.withColumn(...)
   else:
      pass

It's definitely an issue with the loop. I feel like I'm missing something really simple here. I performed the df operation independently on each column and it ran clean ie.

df = df.withColumn(...'colA').withColumn(...'colB').withColumn(...'colC')

The error points to a bool issue in the logic which goes inside the withColumn. Do update question with the logic for the columns. — Nithish
– Nithish, Commented Dec 15, 2021 at 18:58
How can that be? If the alternative solution works then it can't be the logic, right? All I'm doing is literally substituting col for the actual field names in the loop — chicagobeast12
– chicagobeast12, Commented Dec 15, 2021 at 18:59
Can you provide the full code that produces the error? ie where the ......'s are? — BoomBoxBoy
– BoomBoxBoy, Commented Dec 15, 2021 at 19:02

Rahul Kumar · Accepted Answer · 2021-12-15 19:05:15Z

2

Use the following snippet

column_list = ['colA','colB','colC']
for col in df.columns:
   if col in column_list:
      df = df.withColumn(....)
   else:
      pass

answered Dec 15, 2021 at 19:05

Rahul Kumar

2,4055 gold badges27 silver badges56 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Pyspark - Loop over dataframe columns by list

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related