unexpected type: <class 'pyspark.sql.types.DataTypeSingleton'> when casting to Int on a ApacheSpark Dataframe

Question

I'm having an error when trying to cast a StringType to a IntType on a pyspark dataframe:

joint = aggregates.join(df_data_3,aggregates.year==df_data_3.year)
joint2 = joint.filter(joint.CountyCode==999).filter(joint.CropName=='WOOL')\
    .select(aggregates.year,'Production')\
    .withColumn("ProductionTmp", df_data_3.Production.cast(IntegerType))\
    .drop("Production")\
    .withColumnRenamed("ProductionTmp", "Production")

I'm getting:

TypeErrorTraceback (most recent call last) in () 1 joint = aggregates.join(df_data_3,aggregates.year==df_data_3.year) ----> 2 joint2 = joint.filter(joint.CountyCode==999).filter(joint.CropName=='WOOL')
.select(aggregates.year,'Production') .withColumn("ProductionTmp", df_data_3.Production.cast(IntegerType)) .drop("Production")
.withColumnRenamed("ProductionTmp", "Production")

/usr/local/src/spark20master/spark/python/pyspark/sql/column.py in cast(self, dataType) 335 jc = self._jc.cast(jdt) 336 else: --> 337 raise TypeError("unexpected type: %s" % type(dataType)) 338 return Column(jc) 339

TypeError: unexpected type:

Bebeerna · Accepted Answer · 2021-11-11 08:59:16Z

49

PySpark SQL data types are no longer (it was the case before 1.3) singletons. You have to create an instance:

from pyspark.sql.types import IntegerType
from pyspark.sql.functions import col

col("foo").cast(IntegerType())

Column<b'CAST(foo AS INT)'>

In contrast to:

col("foo").cast(IntegerType)

TypeError  
   ...
TypeError: unexpected type: <class 'type'>

cast method can be also used with string descriptions:

col("foo").cast("integer")

Column<b'CAST(foo AS INT)'>

For an overview of the supported Data Types in Spark SQL and Dataframes, one can click this link.

edited Nov 11, 2021 at 8:59

Bebeerna

971 silver badge6 bronze badges

answered Nov 20, 2016 at 7:41

zero323

331k108 gold badges981 silver badges958 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

seth127 Over a year ago

Does anyone know where you can find a list of the valid strings to pass to the cast()? I've looked and I find things like this or this but none of them are explicitly answering the question.

Reza Keshavarz Over a year ago

@seth127 it's better to pass an instance of spark data types, listed here: spark.apache.org/docs/latest/sql-ref-datatypes.html.

Collectives™ on Stack Overflow

unexpected type: <class 'pyspark.sql.types.DataTypeSingleton'> when casting to Int on a ApacheSpark Dataframe

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related