numpy to spark error: TypeError: Can not infer schema for type: <class 'numpy.float64'>

Question

While trying to convert a numpy array into a Spark DataFrame, I receive Can not infer schema for type: <class 'numpy.float64'> error. The same thing happens with numpy.int64 arrays.

Example:

df = spark.createDataFrame(numpy.arange(10.))

TypeError: Can not infer schema for type: <class 'numpy.float64'>

blackbishop · Accepted Answer · 2021-12-20 18:23:27Z

1

Or without using pandas:

df = spark.createDataFrame([(float(i),) for i in numpy.arange(10.)])

answered Dec 20, 2021 at 18:23

blackbishop

32.8k11 gold badges61 silver badges86 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Douglas M · Accepted Answer · 2021-12-20 17:02:03Z

0

A quick conversion to a pandas DataFrame works nicely:

import pandas
import numpy
df = spark.createDataFrame(pandas.DataFrame(numpy.arange(10.)))

answered Dec 20, 2021 at 17:02

Douglas M

1,1469 silver badges19 bronze badges

Collectives™ on Stack Overflow

numpy to spark error: TypeError: Can not infer schema for type: <class 'numpy.float64'>

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related