Spark: AttributeError: 'SQLContext' object has no attribute 'createDataFrame'

Question

How to createDataFrame from a dict? I use the following code and meet errors.

from pyspark import SparkContext, SQLContext
sc = SparkContext.getOrCreate()
spark = SQLContext(sc)

result_dict = {'a':3,'b':44}
data = list(map(list, result_dict.items()))
f_rdd = spark.createDataFrame(data, ["A", "B"]).repartition(1)

Error:

AttributeError                      Traceback (most recent call last)
<ipython-input-10-a25453caa1c3> in <module>
      5 result_dict = {'a':3,'b':44}
      6 data = list(map(list, result_dict.items()))
----> 7 f_rdd = spark.createDataFrame(data, ["A", "B"]).repartition(1)

AttributeError: 'SQLContext' object has no attribute 'createDataFrame'

@rosefun Your code works perfectly fine. I have written down examples of how to create different kinds of PySpark dataframes here in case you are interested. — Jacob Celestine
– Jacob Celestine, Commented Jul 4, 2020 at 19:48

ggorlen · Accepted Answer · 2023-05-06 01:45:41Z

1

You can try this way:

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName('so')\
    .getOrCreate()

sc = spark.sparkContext

map = {'a': 3, 'b': 44}
data = sc.parallelize([(k, v) for k, v in map.items()]).toDF(['A', 'B'])

data.show()

# +---+---+
# |  A|  B|
# +---+---+
# |  a|  3|
# |  b| 44|
# +---+---+

edited May 6, 2023 at 1:45

ggorlen

59.3k8 gold badges119 silver badges173 bronze badges

answered Jul 2, 2020 at 23:46

kites

1,40510 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Spark: AttributeError: 'SQLContext' object has no attribute 'createDataFrame'

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related