Create a dataframe by combining header and data dataframes

Question

I have 2 dataframes:

|data          |
|--------------|
|[1,Rob,12]    |
|[2,Jeremy,11] |
|[3,Bart,14]   |


scala> data.printSchema()

root
 |-- data: array (nullable = true)
 |    |-- element: string (containsNull = true)

and

|headers         |
|----------------|
|[id,name,deptid]|


scala> headers.printSchema()

root
|-- headers: array (nullable = true)
|    |-- element: string (containsNull = true)

Question: How do I create an output data-frame with the following format using the headers DF and the data DF?

| id | name  | deptid|
|----| ------|-------|
| 1  | Rob   | 12    |
| 2  | Jeremy| 11    |
| 3  | Bart  | 14    |

Leo C · Accepted Answer · 2018-09-24 04:56:43Z

0

You can create the list of the column names from the headers DataFrame and use select to map the data array elements to the corresponding headers array elements:

import org.apache.spark.sql.functions._

val dataDF = Seq(
  Seq("1", "Rob", "12"),
  Seq("2", "Jeremy", "11"),
  Seq("3", "Bart", "14")
).toDF("data")

val headersDF = Seq(
  Seq("id", "name", "deptid")
).toDF("headers")

val cols = headersDF.first.getSeq[String](0)
// cols: Seq[String] = WrappedArray(id, name, deptid)

val resultDF = dataDF.
  select( (0 until cols.size).map( i => $"data"(i).as(cols(i)) ): _* )

resultDF.show
// +---+------+------+
// | id|  name|deptid|
// +---+------+------+
// |  1|   Rob|    12|
// |  2|Jeremy|    11|
// |  3|  Bart|    14|
// +---+------+------+

answered Sep 24, 2018 at 4:56

Leo C

22.5k3 gold badges28 silver badges42 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Ramdev Sharma · Accepted Answer · 2018-09-24 13:37:10Z

-1

You can check following thread to get columns from array:

How to explode an array into multiple columns in Spark

you can rename column using withColumnRenamed function.

edited Sep 24, 2018 at 13:37

answered Sep 24, 2018 at 2:49

Ramdev Sharma

1,0141 gold badge14 silver badges18 bronze badges

Collectives™ on Stack Overflow

Create a dataframe by combining header and data dataframes

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related