Trying to create dataframe with two columns [Seq(), String] - Spark

Question

When I run the following on the spark-shell, I get a dataframe:

scala> val df = Seq(Array(1,2)).toDF("a")

scala> df.show(false)
+------+
|a     |
+------+
|[1, 2]|
+------+

But when I run the following to create a dataframe with two columns:

scala> val df1 = Seq(Seq(Array(1,2)),"jf").toDF("a","b")
<console>:23: error: value toDF is not a member of Seq[Object]
    val df1 = Seq(Seq(Array(1,2)),"jf").toDF("a","b")

I get the error:

Value toDF is not a member of Seq[Object].

How do I go about this? Is toDF only supported for sequences with primitive datatypes?

akuiper · Accepted Answer · 2018-04-12 22:19:30Z

8

You need a Seq of Tuple for the toDF method to work:

val df1 = Seq((Array(1,2),"jf")).toDF("a","b")
// df1: org.apache.spark.sql.DataFrame = [a: array<int>, b: string]

df1.show
+------+---+
|     a|  b|
+------+---+
|[1, 2]| jf|
+------+---+

Add more tuples for more rows:

val df1 = Seq((Array(1,2),"jf"), (Array(2), "ab")).toDF("a","b")
// df1: org.apache.spark.sql.DataFrame = [a: array<int>, b: string]

df1.show
+------+---+
|     a|  b|
+------+---+
|[1, 2]| jf|
|   [2]| ab|
+------+---+

answered Apr 12, 2018 at 22:19

akuiper

216k33 gold badges362 silver badges379 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Shibani Over a year ago

Thanks it helped! So I was wrong when I created a Sequence(Array[Int], String), a tuple supports different kinds, a sequence doesn't, if I'm correct.

BlueDolphin Over a year ago

I got the same error. "value toDF is not a member of Seq[Array[String]]" Any suggestion?

jhole89 Over a year ago

Late reply but import spark.implicits._

Collectives™ on Stack Overflow

Trying to create dataframe with two columns [Seq(), String] - Spark

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related