add a new column to spark dataframe from array list

Question

I am trying to add a new column to spark data frame as below:

val abc :Array[String] = ("a","b","c","d")

I am trying to add this Array[String] as new column to dataframe and trying to do sha2 on that new column

var words=abc.mkString("||") source = source.withColumn("newcolumn", sha2(col(words), 256).cast(StringType))

It complied and the runtime error i am getting as

Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve 'a||b||c||d||e' given input columns:

The expected output should be a dataframe with newcolum as column name and the value as varchar64 with sha2 of concatenate of Array of string with ||.

Anyhelp is appreciated.

notNull · Accepted Answer · 2020-04-21 18:13:30Z

0

You need to use lit(words) and caluculate sha2(col,256)

Example

import org.apache.spark.sql.functions._

var words=abc.mkString("||")

val source = source.withColumn("newcolumn",sha2(lit(words),256))
source.show()
//+----------------------------------------------------------------+
//|newcolumn                                                       |
//+----------------------------------------------------------------+
//|b51ff0f0a6aa9f36848b8fdc10ece7242698a4061e141c761b1ac9a08634c067|
//+----------------------------------------------------------------+

answered Apr 21, 2020 at 18:13

notNull

31.8k4 gold badges41 silver badges58 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

abiratsis Over a year ago

@SparkUser personal thanks are always welcome :) although upvoting and accepting the answer that helped you is even better for the StackOverflow users since they spend time and effort to answer them

SparkUser Over a year ago

Actually it's my bad. The question itself is wrong. I have val abc : is set of columns.I want to add this set of column values with || separator and sha2 as a newcolumn in dataframe. I will do upvoting and accept the answer.

notNull Over a year ago

@SparkUser, Just posted answer for the new question.. stackoverflow.com/a/61376746/7632695

Collectives™ on Stack Overflow

add a new column to spark dataframe from array list

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related