Getting B cannot be cast to java.lang.String when using Spark SQL

Question

My issue is when I'm using trying to read data from a sql.Row as a String. I'm using pyspark, but I've heard people have this issue with Scala API too.

The pyspark.sql.Row object is a pretty intransigent creature. The following exception is thrown:

java.lang.ClassCastException: [B cannot be cast to java.lang.String
 at org.apache.spark.sql.catalyst.expressions.GenericRow.getString(Row.scala 183)

So what we have is one of the fields is being represented as a byte array. The following python printing constructs do NOT work

repr(sqlRdd.take(2))

Also

import pprint
pprint.pprint(sqlRdd.take(2))

Both result in the ClassCastException.

So.. how do other folks do this? I started to roll my own (can not copy/paste here unfortunately..) But this is a bit re-inventing the wheel .. or so I suspect.

samthebest · Accepted Answer · 2015-09-18 09:58:49Z

4

Try

sqlContext.setConf("spark.sql.parquet.binaryAsString", "true")

I think since Spark 1.1.0 they broke it - reading binary as strings used to work, then they made it not work, but added this flag, but set it's default to false.

edited Sep 18, 2015 at 9:58

answered Sep 18, 2015 at 9:42

samthebest

31.7k25 gold badges106 silver badges153 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

WestCoastProjects Over a year ago

nice! thx for coming back (well later), getting out the backhoe to dig this up, and giving a good solution.

Collectives™ on Stack Overflow

Getting B cannot be cast to java.lang.String when using Spark SQL

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related