Dataframe creation in Scala

Question

wordsDF = sqlContext.createDataFrame([('cat',), ('elephant',), ('rat',), ('rat',), ('cat', )], ['word'])

This is a way of creating dataframe from a list of tuples in python. How can I do this in scala ? I'm new to Scala and I'm facing problem in figuring it out.

Any help will be appreciated!

Possible duplicate of Convert an RDD to a DataFrame in Spark using Scala — Alberto Bonsanto
– Alberto Bonsanto, Commented May 12, 2016 at 14:19

elm · Accepted Answer · 2016-05-12 10:57:28Z

6

One simple way,

val df = sc.parallelize(List( (1,"a"), (2,"b") )).toDF("key","value")

and so df.show

+---+-----+
|key|value|
+---+-----+
|  1|    a|
|  2|    b|
+---+-----+

Refer to the worked example in Programmatically Specifying the Schema for constructing a DataFrame with createDataFrame.

answered May 12, 2016 at 10:57

elm

20.5k15 gold badges72 silver badges116 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Pawan B · Accepted Answer · 2016-05-12 12:16:30Z

0

To create a dataframe , you need to create SQLContext .

val sc: SparkContext // An existing SparkContext.
val sqlContext = new org.apache.spark.sql.SQLContext(sc)

// this is used to implicitly convert an RDD to a DataFrame , after importing it you can use .toDF method
import sqlContext.implicits._

now you can create dataframes

val df1 = sc.makeRDD(1 to 5).map(i => (i, i * 2)).toDF("single", "double")

learn more about creating dataframes here

answered May 12, 2016 at 12:16

Pawan B

4,6332 gold badges27 silver badges39 bronze badges

Collectives™ on Stack Overflow

Dataframe creation in Scala

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related