How to create a DataFrame from List?

Question

I want to create DataFrame df that should look as simple as this:

+----------+----------+
| timestamp|      col2|
+----------+----------+
|2018-01-11|       123|
+----------+----------+

This is what I do:

val values = List(List("timestamp", "2018-01-11"),List("col2","123")).map(x =>(x(0), x(1)))    
val df = values.toDF    
df.show()

And this is what I get:

+---------+----------+
|       _1|        _2|
+---------+----------+
|timestamp|2018-01-11|
|     col2|       123|
+---------+----------+

What's wrong here?

Tzach Zohar · Accepted Answer · 2018-01-26 20:08:30Z

5

Use

val df = List(("2018-01-11", "123")).toDF("timestamp", "col2")

toDF expects the input list to contain one entry per resulting Row
Each such entry should be a case class or a tuple
It does not expect column "headers" in the data itself (to name columns - pass names as arguments of toDF)

answered Jan 26, 2018 at 20:08

Tzach Zohar

37.9k3 gold badges83 silver badges86 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Denis Iakunchikov · Accepted Answer · 2018-01-26 20:11:43Z

1

If you don't know the names of the columns statically you can use following syntax sugar

.toDF( columnNames: _*)

Where columnNames is the List with the names.

answered Jan 26, 2018 at 20:11

Denis Iakunchikov

1,3492 gold badges11 silver badges17 bronze badges

Comments

Alper t. Turker · Accepted Answer · 2018-01-26 20:31:30Z

1

EDIT (sorry, I missed that you had the headers glued to each column). Maybe something like this could work:

val values = List(
  List("timestamp", "2018-01-11"),
  List("col2","123")
)

val heads = values.map(_.head) // extracts headers of columns
val cols = values.map(_.tail) // extracts columns without headers
val rows = cols(0).zip(cols(1)) // zips two columns into list of rows
rows.toDF(heads: _*)

This would work if the "values" contained two longer lists, but it does not generalize to more lists.

edited Jan 26, 2018 at 20:31

Alper t. Turker

35.3k9 gold badges89 silver badges118 bronze badges

answered Jan 26, 2018 at 20:08

Andrey Tyukin

45.1k4 gold badges65 silver badges103 bronze badges

Collectives™ on Stack Overflow

How to create a DataFrame from List?

3 Answers 3

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related