How to filter values in an array column in PySpark?

Question

I have a column of ArrayType in Pyspark. I want to filter only the values in the Array for every Row (I don't want to filter out actual rows!) without using UDF.

For instance given this dataset with column A of ArrayType:

|     A      |
______________
|[-2, 1, 7]  |
|[1]         |
|[-4, -1, -3]|

And I would like to have only positive values the output would be:

|     A      |
______________
|[1, 7]      |
|[1]         |
|[]          |

Surya · Accepted Answer · 2020-11-11 17:24:07Z

6

For Spark 2.4 and above,

from pyspark.sql.functions import expr

df.withColumn("A", expr("filter(A, x -> x > 0)")).show()

answered Nov 11, 2020 at 17:24

Surya

3,4293 gold badges22 silver badges36 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to filter values in an array column in PySpark?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related