Error creating Spark DataSet with array of bytearrays

Question

I'm creating a Spark dataset in scala using a case class and the spark.sql({query}).as[MyCaseClass] syntax

All is fine until I try to create a dataset with one of my members defined as Array[Array[Byte]]

case class HbaseRow(
  ip: Array[Array[Byte]]
)

val hbaseDataSet = spark
   .sql("""select ip from test_data""")
   .as[HbaseRow]

Normally this works fine but with the array of byte arrays this fails.

java.lang.ClassCastException: 
org.apache.spark.sql.types.BinaryType$ cannot be cast to org.apache.spark.sql.types.ObjectType
at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$org$apache$spark$sql$catalyst$ScalaReflection$$arrayClassFor$1.apply(ScalaReflection.scala:106)
at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$org$apache$spark$sql$catalyst$ScalaReflection$$arrayClassFor$1.apply(ScalaReflection.scala:95)
at scala.reflect.internal.tpe.TypeConstraints$UndoLog.undo(TypeConstraints.scala:56)

the column is a spark array of IP addresses encoded as byte arrays themselves.

benlaird · Accepted Answer · 2018-11-26 13:24:14Z

1

Ok, I asked this while stuck but believe I've landed on the solution. Defining my case class type with an Option seems to have done the trick.

scala> case class HbaseRow(
 |     ip: Array[Option[Array[Byte]]]
 | )
defined class HbaseRow

scala> df.select($"ip").as[HbaseRow]
res13: org.apache.spark.sql.Dataset[HbaseRow] = [ip: array<binary>]

answered Nov 26, 2018 at 13:24

benlaird

8797 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Error creating Spark DataSet with array of bytearrays

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related