I have sample data as below.
I would like populate another column exp based on country field value.
Something like below
df.withColumn("exp",col(s"exp_$country"))
So that respective country number can be placed there.
But above code errors out saying:
cannot resolve country
Output I need is
Any help appreciated.
You can chain multiple when expressions from the list of countries:
val countries = Seq("us", "uk", "ind")
val expCol = countries.foldLeft(lit(null)) { case (acc, country) =>
when(col("country")===country, col(s"exp_$country")).otherwise(acc)
}
val df1 = df.withColumn("exp", expCol)
Or if you prefer creating a map expression country -> exp from the columns exp_* than use the map to create exp column:
val mapCountries = map(
df.columns
.filter(_.startsWith("exp_"))
.flatMap(c => Seq(lit(c.split("_")(1)), col(c))): _*
)
val df1 = df.withColumn("exp", mapCountries(col("country")))
