java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema cannot be cast t

Question

I am trying to convert a Row in a dataframe to a case class and getting following error

2019-08-19 20:13:08 Executor task launch worker for task 1 ERROR Executor:91 - Exception in task 0.0 in stage 1.0 (TID 1) java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema cannot be cast to Models.City

Sample Log = {"Id": "1","City": {"name": "A","state": "B"}}

Below is the code that is reading a text file having data in json format which is throwing above error

case class City(name: String, state: String)

val file = new File("src/test/resources/log.txt")
val logs = spark.
  read.
  text(file.getAbsolutePath).
  select(col("value").
    as("body"))
import spark.implicits._
var logDF: DataFrame = spark.read.json(logs.as[String])
logDF.map(row => row.getAs[City]("City").state).show()

Basically I can not perform any operation on the dataframe itself due to some restrictions. So given a row how can we cast it into a case class (i cannot use match pattern here as case class can have lot of fields and nested case classes)

Thanks in advance. Any help is greatly appreciated!!

@Robin.. thanks for the link, still i need to explicitly cast each filed to its type, is there a better way to cast the whole row into case class — Abhishek Gupta
– Abhishek Gupta, Commented Aug 26, 2019 at 6:51

Iraj Hedayati · Accepted Answer · 2022-03-28 13:03:09Z

1

I had the same issue (Spark SQL 3.1.3). The solution was to use Spark to convert Dataframe to Dataset and then access the fields.

import spark.implicits._
var logDF: DataFrame = spark.read.json(logs.as[String])
logDF.select("City").as[City].map(city => city.state).show()

answered Mar 28, 2022 at 13:03

Iraj Hedayati

1,7071 gold badge21 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema cannot be cast t

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related