Getting Error while trying create a spark dataframe in pandas [duplicate]

Question

I am new to python and I am facing problem in creating a Dataframe using pandas:

import pandas as pd

df = spark.createDataFrame([(66, "a", "4"), 
                            (67, "a", "0"), 
                            (70, "b", "4"), 
                            (71, "d", "4")],
                            ("id", "code", "amt"))


dfa = pd.DataFrame(data=df)

This is the error I am getting ValueError: DataFrame constructor not properly called!

Do you have to start with spark?. You can pass that list to the dataframe constructor directly — Paul H
– Paul H, Commented Feb 24, 2021 at 3:39
You also need to be more specific than "i am facing a problem". What kind of problem? Are the data in the wrong order? Is an error raised? — Paul H
– Paul H, Commented Feb 24, 2021 at 3:41
OK, so the dataframe constructor was not properly called. Did you read the docs on how to call it? What wasn't clear about those docs? — Paul H
– Paul H, Commented Feb 24, 2021 at 3:41
But did you look at the official documentation? it has many example of creating dataframes — Paul H
– Paul H, Commented Feb 24, 2021 at 3:46
That's not a pandas dataframe, it's a PySpark one. And please add the missing 'import' for spark — smci
– smci, Commented Feb 24, 2021 at 3:49

Paul H · Accepted Answer · 2021-02-24 03:52:53Z

1

dfa = df.select("*").toPandas()

see https://spark.apache.org/docs/latest/sql-pyspark-pandas-with-arrow.html#enabling-for-conversion-tofrom-pandas

edited Feb 24, 2021 at 3:52

Paul H

68.7k23 gold badges165 silver badges139 bronze badges

answered Feb 24, 2021 at 3:47

Apollon16

363 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

delimiter · Accepted Answer · 2021-02-24 03:44:24Z

0

What is the error message? Have you imported spark or just pandas? Why are you using spark instead of pandas for creating the dataframe, just go as

pd.DataFrame([(66, "a", "4"), 
              (67, "a", "0"), 
              (70, "b", "4"), 
              (71, "d", "4")], columns=("id", "code", "amt"))

And it will make a dataframe for you.

    id      code    amt
0   66      a       4
1   67      a       0
2   70      b       4
3   71      d       4

answered Feb 24, 2021 at 3:44

delimiter

8156 silver badges13 bronze badges

Collectives™ on Stack Overflow

Getting Error while trying create a spark dataframe in pandas [duplicate]

2 Answers 2

Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Linked

Related