I am trying to run some sample code from this link: https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/5537430417240233/312903576646278/3506802399907740/latest.html
I am running it in a databricks notebook on a cluster with runtime 6.3 (includes Apache Spark 2.4.4, Scala 2.11) I initially create a dataframe using
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder.getOrCreate
import spark.implicits._
val df = Seq(
("one", 2.0),
("two", 1.5),
("three", 8.0)
).toDF("id", "val")
and then I try to get a list of strings by running
df.select("id").map(_.getString(0)).collect.toList
and i get the error below
java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)[Ljava/lang/Object;
t line3700fe51392b4abe9744f6b3a059dbfa46.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.(command-1275538363433250:2) at line3700fe51392b4abe9744f6b3a059dbfa46.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw.(command-1275538363433250:53) at line3700fe51392b4abe9744f6b3a059dbfa46.$read$$iw$$iw$$iw$$iw$$iw$$iw.(command-1275538363433250:55) at line3700fe51392b4abe9744f6b3a059dbfa46.$read$$iw$$iw$$iw$$iw$$iw.(command-1275538363433250:57) at line3700fe51392b4abe9744f6b3a059dbfa46.$read$$iw$$iw$$iw$$iw.(command-1275538363433250:59) at line3700fe51392b4abe9744f6b3a059dbfa46.$read$$iw$$iw$$iw.(command-1275538363433250:61) at line3700fe51392b4abe9744f6b3a059dbfa46.$read$$iw$$iw.(command-1275538363433250:63) at line3700fe51392b4abe9744f6b3a059dbfa46.$read$$iw.(command-1275538363433250:65) at line3700fe51392b4abe9744f6b3a059dbfa46.$read.(command-1275538363433250:67) at line3700fe51392b4abe9744f6b3a059dbfa46.$read$.(command-1275538363433250:71) at line3700fe51392b4abe9744f6b3a059dbfa46.$read$.(command-1275538363433250) at line3700fe51392b4abe9744f6b3a059dbfa46.$eval$.$print$lzycompute(:7) at line3700fe51392b4abe9744f6b3a059dbfa46.$eval$.$print(:6) at line3700fe51392b4abe9744f6b3a059dbfa46.$eval.$print() at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:793) at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1054) at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:645) at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:644) at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31) at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19) at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:644) at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:576) at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:572) at com.databricks.backend.daemon.driver.DriverILoop.execute(DriverILoop.scala:215) at com.databricks.backend.daemon.driver.ScalaDriverLocal$$anonfun$repl$1.apply$mcV$sp(ScalaDriverLocal.scala:202) at com.databricks.backend.daemon.driver.ScalaDriverLocal$$anonfun$repl$1.apply(ScalaDriverLocal.scala:202) at com.databricks.backend.daemon.driver.ScalaDriverLocal$$anonfun$repl$1.apply(ScalaDriverLocal.scala:202) at com.databricks.backend.daemon.driver.DriverLocal$TrapExitInternal$.trapExit(DriverLocal.scala:699) at com.databricks.backend.daemon.driver.DriverLocal$TrapExit$.apply(DriverLocal.scala:652) at com.databricks.backend.daemon.driver.ScalaDriverLocal.repl(ScalaDriverLocal.scala:202) at com.databricks.backend.daemon.driver.DriverLocal$$anonfun$execute$9.apply(DriverLocal.scala:385) at com.databricks.backend.daemon.driver.DriverLocal$$anonfun$execute$9.apply(DriverLocal.scala:362) at com.databricks.logging.UsageLogging$$anonfun$withAttributionContext$1.apply(UsageLogging.scala:251) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) at com.databricks.logging.UsageLogging$class.withAttributionContext(UsageLogging.scala:246) at com.databricks.backend.daemon.driver.DriverLocal.withAttributionContext(DriverLocal.scala:49) at com.databricks.logging.UsageLogging$class.withAttributionTags(UsageLogging.scala:288) at com.databricks.backend.daemon.driver.DriverLocal.withAttributionTags(DriverLocal.scala:49) at com.databricks.backend.daemon.driver.DriverLocal.execute(DriverLocal.scala:362) at com.databricks.backend.daemon.driver.DriverWrapper$$anonfun$tryExecutingCommand$2.apply(DriverWrapper.scala:644) at com.databricks.backend.daemon.driver.DriverWrapper$$anonfun$tryExecutingCommand$2.apply(DriverWrapper.scala:644) at scala.util.Try$.apply(Try.scala:192) at com.databricks.backend.daemon.driver.DriverWrapper.tryExecutingCommand(DriverWrapper.scala:639) at com.databricks.backend.daemon.driver.DriverWrapper.getCommandOutputAndError(DriverWrapper.scala:485) at com.databricks.backend.daemon.driver.DriverWrapper.executeCommand(DriverWrapper.scala:597) at com.databricks.backend.daemon.driver.DriverWrapper.runInnerLoop(DriverWrapper.scala:390) at com.databricks.backend.daemon.driver.DriverWrapper.runInner(DriverWrapper.scala:337) at com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.scala:219) at java.lang.Thread.run(Thread.java:748)
I get the same error when running
df.select("id").collect().map(_(0)).toList
but not when running
df.select("id").rdd.map(_(0)).collect.toList
The command above that runs successfully returns a List[Any], but I need a List[String]
Can someone please advise? I suspect it's a sparkscala version mismatch, but I can't figure out what is wrong.
