Spark assign value if null to column (python)

Question

Assuming that I have the following data

+--------------------+-----+--------------------+
|              values|count|             values2|
+--------------------+-----+--------------------+
|              aaaaaa|  249|                null|
|              bbbbbb|  166|                  b2|
|              cccccc| 1680|           something|
+--------------------+-----+--------------------+

So if there is a null value in values2 column how to assign the values1 column to it? So the result should be:

+--------------------+-----+--------------------+
|              values|count|             values2|
+--------------------+-----+--------------------+
|              aaaaaa|  249|              aaaaaa|
|              bbbbbb|  166|                  b2|
|              cccccc| 1680|           something|
+--------------------+-----+--------------------+

I thought of something of the following but it doesnt work:

df.na.fill({"values2":df['values']}).show()

I found this way to solve it but there should be something more clear forward:

def change_null_values(a,b):
    if b:
        return b
    else:
        return a

udf_change_null = udf(change_null_values,StringType())

df.withColumn("values2",udf_change_null("values","values2")).show()

Does this answer your question? Filter Pyspark dataframe column with None value — shadow_dev
– shadow_dev, Commented Nov 6, 2019 at 0:01
@Mpizos Dimitris, check this out stackoverflow.com/questions/61964179/… — jgtrz
– jgtrz, Commented May 22, 2020 at 22:33

Dmytro Popovych · Accepted Answer · 2016-09-06 08:55:50Z

5

You can use https://spark.apache.org/docs/1.6.2/api/python/pyspark.sql.html#pyspark.sql.functions.coalesce

df.withColumn('values2', coalesce(df.values2, df.values)).show()

answered Sep 6, 2016 at 8:55

Dmytro Popovych

9547 silver badges2 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

vegetarianCoder Over a year ago

Can we ingest our own value like 0 or something ?

ZettaP · Accepted Answer · 2021-10-12 15:34:29Z

4

Following up on @shadow_dev's method :

    df.withColumn("values2", 
                  when(col("values2").isNull(), col("values1"))
                  .otherwise(col("values2")))

Dmytro Popovych's solution is still the cleanest.

If one needs more fancy when/otherwise logic :

df.withColumn("values2", when(col("values2").isNull() | col("values3").isNull(), col("values1"))
.when(col("values1") == col("values2"), 1)
.otherwise(0))

answered Oct 12, 2021 at 15:34

ZettaP

1,49412 silver badges15 bronze badges

Comments

shadow_dev · Accepted Answer · 2019-11-06 00:01:13Z

-3

You can use the column attribute .isNull().

df.where(col("dt_mvmt").isNull())

df.where(col("dt_mvmt").isNotNull())

This answer comes from this answer - I just don't have enough reputation to add a comment.

answered Nov 6, 2019 at 0:01

shadow_dev

1301 gold badge1 silver badge15 bronze badges

1 Comment

proutray Over a year ago

Doesn't provide complete solution to the issue. Only hints towards the solution.

Collectives™ on Stack Overflow

Spark assign value if null to column (python)

3 Answers 3

1 Comment

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related