Select case class based on String in Scala

Question

How can I select a case class based on a String value?

My code is

val spark = SparkSession.builder()...
val rddOfJsonStrings: RDD[String] = // some json strings as RDD
val classSelector: String = ??? // could be "Foo" or "Bar", or any other String value
case class Foo(foo: String)
case class Bar(bar: String)


if (classSelector == "Foo") {
  val df: DataFrame = spark.read.json(rddOfJsonStrings)
  df.as[Foo]
} else if (classSelector == "Bar") {
  val df: DataFrame = spark.read.json(rddOfJsonStrings)
  df.as[Bar]
} else {
  throw ClassUnknownException //custom Exception
}

The variable classSeletector is a simple String that should be used to point to the case class of the same name.

Imagine I don't only have Foo and Bar as case classes but more then those two. How is it possible to call the df.as[] statement based on the String (if possible at all)?

Or is there a completely different approach available in Scala?

the classSelector is a simple String that is given e.g. through the main(args) function. Is it possible to use the value of this string to point to the case class with the identical name? — Michael Heil
– Michael Heil, Commented Jun 9, 2020 at 13:31
The "completely different approach" would be to use a typeclass that encapsulates the specific behaviour for Foo and Bar. There are various tutorials about this online, so pick one that looks right for you. — Tim
– Tim, Commented Jun 9, 2020 at 13:42
@mike Note: I've edited the answer because it turns out it's much easier to do partially than I thought, writing so you'll get a notification. — Alexey Romanov
– Alexey Romanov, Commented Jun 10, 2020 at 9:21

s.polam · Accepted Answer · 2020-06-09 13:40:41Z

2

Check below code

classSeletector match {
    case c if Foo.getClass.getSimpleName.replace("$","").equalsIgnoreCase(c) =>  spark.read.json(rddOfJsonStrings).as[Foo]
    case c if Bar.getClass.getSimpleName.replace("$","").equalsIgnoreCase(c) =>  spark.read.json(rddOfJsonStrings).as[Bar]
    case _ => throw ClassUnknownException //custom Exception
}

answered Jun 9, 2020 at 13:40

s.polam

10.4k2 gold badges17 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Alexey Romanov · Accepted Answer · 2020-06-11 13:13:43Z

How is it possible to call the df.as[] statement based on the String (if possible at all)?

It isn't (or based on any runtime value). You may note that all answers still need to:

have a separate branch for Foo and Bar (and one more branch for each class you'll want to add);
repeat the class name twice in the branch.

You can avoid the second:

import scala.reflect.{classTag, ClassTag}

val df: DataFrame = spark.read.json(rddOfJsonStrings)
// local function defined where df and classSelector are visible
def dfAsOption[T : Encoder : ClassTag] =
  Option.when(classSelector == classTag[T].runtimeClass.simpleName)(df.as[T])

dfAsOption[Foo].dfAsOption(asOption[Bar]).getOrElse(throw ClassUnknownException)

But for the first you'd need a macro if it's possible at all. I would guess it isn't.

QuickSilver · Accepted Answer · 2020-06-09 13:16:25Z

1

Define a generic method and invoke it,


getDs[Foo](spark,rddOfJsonStrings)
getDs[Bar](spark,rddOfJsonStrings)

def getDs[T](spark : SparkSession, rddOfJsonStrings:String)  {
    spark.read.json(rddOfJsonStrings).as[T](Encoders.bean[T](classOf[T]))
  }

answered Jun 9, 2020 at 13:16

QuickSilver

4,0452 gold badges15 silver badges31 bronze badges

1 Comment

Michael Heil Over a year ago

Thank you for your reply @QuickSilver. If I understand correctly, this will reduce a bit of code, but how could I use the content of classSelector to avoid calling getDs[Foo] and getDs[Bar] individually. I was hoping to do something like getDs[classSelector.asClassName].

Som · Accepted Answer · 2020-06-09 14:56:21Z

Alternative-

highlights-

Use simpleName of the case class and not of the companion object
if classSelector is null, the solution won't fail

case class Foo(foo: String)
case class Bar(bar: String)

Testcase-

 val rddOfJsonStrings: RDD[String] = spark.sparkContext.parallelize(Seq("""{"foo":1}"""))
    val classSelector: String = "Foo" // could be "Foo" or "Bar", or any other String value

    val ds = classSelector match {
      case foo if classOf[Foo].getSimpleName == foo =>
        val df: DataFrame = spark.read.json(rddOfJsonStrings)
        df.as[Foo]
      case bar if classOf[Bar].getSimpleName == bar =>
        val df: DataFrame = spark.read.json(rddOfJsonStrings)
        df.as[Bar]
      case _ => throw new UnsupportedOperationException
    }

    ds.show(false)

    /**
      * +---+
      * |foo|
      * +---+
      * |1  |
      * +---+
      */

Dmytro Mitin · Accepted Answer · 2022-09-15 20:10:25Z

You can use reflective toolbox

import org.apache.spark.sql.{Dataset, SparkSession}
import scala.reflect.runtime
import scala.tools.reflect.ToolBox

object Main extends App {
  val spark = SparkSession.builder
    .master("local")
    .appName("Spark SQL basic example")
    .getOrCreate()
  import spark.implicits._

  val rddOfJsonStrings: Dataset[String] = spark.createDataset(Seq("""{"foo":"aaa"}"""))
  // val rddOfJsonStrings: Dataset[String] = spark.createDataset(Seq("""{"bar":"bbb"}"""))

  val classSelector: String = "Foo"
  // val classSelector: String = "Bar"

  case class Foo(foo: String)
  case class Bar(bar: String)

  val runtimeMirror = runtime.currentMirror
  val toolbox = runtimeMirror.mkToolBox()

  val res = toolbox.eval(toolbox.parse(s"""
    import org.apache.spark.sql.DataFrame
    import Main._
    import spark.implicits._
    val df: DataFrame = spark.read.json(rddOfJsonStrings)
    df.as[$classSelector]
  """)).asInstanceOf[Dataset[_]]

  println(res) // [foo: string]
}

Notice that statically you will have a Dataset[_], not Dataset[Foo] or Dataset[Bar].

Collectives™ on Stack Overflow

Select case class based on String in Scala

5 Answers 5

Comments

Comments

1 Comment

Alternative-

Testcase-

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

Comments

Comments

1 Comment

Alternative-

Testcase-

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related