Adding new column using existing one using Spark Scala

Question

Hi I want to add new column using existing column in each row of DataFrame , I am trying this in Spark Scala like this ... df is dataframe containing variable number of column , which can be decided at run time only.

// Added new column "docid"
val df_new = appContext.sparkSession.sqlContext.createDataFrame(df.rdd, df.schema.add("docid", DataTypes.StringType))

 df_new.map(x => {
        import appContext.sparkSession.implicits._
      val allVals = (0 to x.size).map(x.get(_)).toSeq
      val values = allVals ++ allVals.mkString("_") 
      Row.fromSeq(values)
    })

But this is giving error is eclipse itself

Unable to find encoder for type stored in a Dataset. Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._ Support for serializing other types will be added in future releases.
not enough arguments for method map: (implicit evidence$7: org.apache.spark.sql.Encoder[org.apache.spark.sql.Row])org.apache.spark.sql.Dataset[org.apache.spark.sql.Row]. Unspecified value parameter evidence$7.

Please help.

Can you give example of input data and expected output? This should be possible to solve in an more efficient way. — Shaido
– Shaido, Commented Oct 9, 2017 at 9:59
Any reason why you can't use df.withColumn() as suggested in werner's answer? That would be the most straightforward solution — sparker
– sparker, Commented Dec 16, 2020 at 11:23

werner · Accepted Answer · 2020-06-13 19:22:15Z

1

concat_ws from the functions object can help.

This code adds the docid field

df = df.withColumn("docid", concat_ws("_", df.columns.map(df.col(_)):_*))

assuming all columns of df are strings.

answered Jun 13, 2020 at 19:22

werner

15k6 gold badges36 silver badges56 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Adding new column using existing one using Spark Scala

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related