How to replace specific columns multiple value in Spark Dataframe?

Question

I am trying to replace or update some specific column value in dataframe, as we know Dataframe is immutable, I am trying to transform in to new dataframe instead of Update or Replacement.

I tried dataframe.replace as explained in Spark doc, but it's giving me error as error: value replace is not a member of org.apache.spark.sql.DataFrame

I tried below option.For passing multiple value I am passing in array

val new_df= df.replace("Stringcolumn", Map(array("11","17","18","10"->"12")))

but I am getting error as

error: overloaded method value array with alternatives

Help is really appreciated!!

Swadhin Shahriar · Accepted Answer · 2017-03-22 05:54:36Z

1

To access org.apache.spark.sql.DataFrameNaFunctions such as replace you have to call .na. So your code should look something like this,

import com.google.common.collect.ImmutableMap

df.na.replace("Stringcolumn", Map(10 -> 12, 11 -> 17))

see here to get all the list of DataFrameNaFunctions and how to use them

answered Mar 22, 2017 at 5:54

Swadhin Shahriar

465 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to replace specific columns multiple value in Spark Dataframe?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related