PySpark Conversion to Array Types

Question

I'm currently dealing with the following error while trying to run pyspark.sql.functions.explode on a array column in a DataFrame in PySpark. I've tried creating a UDF to convert the column to a python list if it is not a list instance. However, this still throws the same error. In Pandas I'd typically pull out the row and determine what to do from there. I'm not sure how I can access this row to look at the data to understand what conditions I need to account for.

I'm more looking for debugging advice in general, but if you know the answer that's great too!

Py4JJavaError: An error occurred while calling o2850.withColumn. : org.apache.spark.sql.AnalysisException: cannot resolve 'explode(lot)' due to data type mismatch: input to function explode should be array or map type, not LongType;;

df.schema()

root
 |-- lists: array (nullable = true)
 |    |-- element: long (containsNull = true)
 |-- data: string (nullable=true)

df = df.withColumn("list",df.lists)
df = df.withColumn('list',sf.explode(df.list))

Original Code

from pyspark.sql import functions as sf 

# create duplicate column to use with explode 
# explode the array datetype into multiple rows per element 

df = spark.read("s3a://path/parquet/*")     
df = df.withColumn("list",df.lists)
df = df.withColumn('list',sf.explode(df.list))

can you please post the code snippet of the problem, that will be much helpful in debugging or finding out the probelem — Ankit Kumar Namdeo
– Ankit Kumar Namdeo, Commented Nov 21, 2017 at 9:23
please, do not post code snippets in the comments! Edit your post instead... — desertnaut
– desertnaut, Commented Nov 21, 2017 at 21:35

Bijan · Accepted Answer · 2019-09-03 17:53:07Z

1

There is no need to used withcolumn you can directly explode the array.

df = spark.read("s3a://path/parquet/*")     
df.select(df['data'],explode(df['lists']).alias('list'))

edited Sep 3, 2019 at 17:53

Bijan

8,81622 gold badges107 silver badges164 bronze badges

answered Nov 22, 2017 at 6:30

Sahil Desai

3,6964 gold badges22 silver badges45 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Sahil Desai Over a year ago

if you want original lists column with this then you can also select lists.

Collectives™ on Stack Overflow

PySpark Conversion to Array Types

Original Code

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

Original Code

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related