StructType can not accept object float in pyspark

Question

Why this is working just fine

from pyspark.sql.types import *
l=[("foo",83.33)]
schema = StructType([
   StructField("type", StringType(), True),
   StructField("value", DoubleType(), True)])
df= spark.createDataFrame(l,schema)

And this

 l=[(83.33)]
    schema = StructType([
    StructField("value", DoubleType(), True)])
    df= spark.createDataFrame(l,schema)

Gives me an error

StructType can not accept object 83.33 in type <class 'float'>

Bala · Accepted Answer · 2018-03-25 16:59:14Z

7

When using a one element tuple, a trailing comma is required. Check this TupleSyntax

>>> l=[(83.33,)]           //note the comma (,)
>>> schema = StructType([
...     StructField("value", DoubleType(), True)])
>>> df= spark.createDataFrame(l,schema)
>>> df.show()
+-----+
|value|
+-----+
|83.33|
+-----+

edited Mar 25, 2018 at 16:59

answered Mar 24, 2018 at 12:21

Bala

11.3k19 gold badges75 silver badges134 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Bala Over a year ago

@Yakov: Updated with explanation and a link

Yinthewind Over a year ago

python never fails surprising me

Collectives™ on Stack Overflow

StructType can not accept object float in pyspark

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related