How to convert a list of array to Spark dataframe

Question

Suppose I have a list:

x = [[1,10],[2,14],[3,17]]

I want to convert x to a Spark dataframe with two columns id (1,2,3) and value (10,14,17).

How could I do that?

Thanks

Poonam · Accepted Answer · 2017-08-24 10:29:58Z

7

x = [[1,10],[2,14],[3,17]]
df = sc.parallelize(x).toDF(['ID','VALUE'])
df.show()

answered Aug 24, 2017 at 10:29

Poonam

6794 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Rahul Sharma · Accepted Answer · 2021-10-26 23:23:27Z

2

Alternatively you can create it directly using SparkSession-

x = [[1,10],[2,14],[3,17]]
df = spark.createDataFrame(data=x, schema = ["id","value"])
df.printSchema()
df.show()

answered Oct 26, 2021 at 23:23

Rahul Sharma

5,86511 gold badges60 silver badges97 bronze badges