Spark Scala Dataframe How to create new column with two or more existing columns

Question

Suppose I have a Spark Scala DataFrame object like:

+----------------+ |col1 |col2 | +----------------+ |1 |2 | |3 |4 | +----------------+

And I want a DataFrame like: +--------------------+ |col1 |col2 |col3 | +--------------------+ |1 |2 |3 | |3 |4 |7 | +--------------------+

Which adds col1 and col2 to col3, could anyone please tell me how to do that? WithColumn takes only one column as parameter whereas I need to take two columns.

Consider that withColumn is lazy and will be optimized at runtime, I'm fairly sure you can call it twice for the two columns without any problem. — stefanobaghino
– stefanobaghino, Commented Jun 15, 2018 at 6:17
Possible duplicate of Adding a column of rowsums across a list of columns in Spark Dataframe — Anahcolus
– Anahcolus, Commented Jun 15, 2018 at 9:58

koiralo · Accepted Answer · 2018-06-15 06:19:41Z

3

You can use withColumn or select as

val df = Seq(
  (1,2),
  (3,4)
).toDF("col1", "col2")

df.withColumn("col3", $"col1" + $"col2").show(false)

df.select($"col1", $"col2", ($"col1" + $"col2").as("col3")).show(false)

Output:

+----+----+----+
|col1|col2|col3|
+----+----+----+
|1   |2   |3   |
|3   |4   |7   |
+----+----+----+

answered Jun 15, 2018 at 6:19

koiralo

23.2k6 gold badges57 silver badges77 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Arnon Rotem-Gal-Oz · Accepted Answer · 2018-06-15 06:47:28Z

0

WithColumn takes two parameters a name and a function that should result in a type column - so a function or expression whose result is a column is valid hence you can do the below (or similar)

df.withColumn("col3", df("col1")+df("col2"))

answered Jun 15, 2018 at 6:47

Arnon Rotem-Gal-Oz

26k3 gold badges51 silver badges70 bronze badges

Collectives™ on Stack Overflow

Spark Scala Dataframe How to create new column with two or more existing columns

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related