NoSuchMethodError: org.apache.spark.sql.SQLContext.applySchema

Question

I am trying to query on a file stored in hdfs using sqlcontext provided in Apache Spark using the below code but i am getting a NoSuchMethodError

package SQL

import org.apache.spark.SparkContext 
import org.apache.spark.sql._

object SparSQLCSV {   def main(args: Array[String]) {

    val sc = new SparkContext("local[*]","home")
    val sqlContext = new org.apache.spark.sql.SQLContext(sc)
    val people = sc.textFile("/home/devan/Documents/dataset/peoplesTest.csv")
    val delimiter = ","
    val schemaString = "a,b".split(delimiter)//csv header
    //Automated Schema creation
    val schema =   StructType(schemaString.map(fieldName => StructField(fieldName, StringType, true)))
    val peopleLines = people.flatMap(x=> x.split("\n"))
    val rowRDD = peopleLines.map(p=>{
      Row.fromSeq(p.split(delimiter))
    })
    val peopleSchemaRDD = sqlContext.applySchema(rowRDD, schema)
    peopleSchemaRDD.registerTempTable("people")
    sqlContext.sql("SELECT b FROM people").foreach(println)

  } }

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.sql.SQLContext.applySchema(Lorg/apache/spark/rdd/RDD;Lorg/apache/spark/sql/types/StructType;)Lorg/apache/spark/sql/DataFrame; at scalding.Main_Obj$.main(Main_Obj.scala:34) at scalding.Main_Obj.main(Main_Obj.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

I have tried the same using command line provided in spark, and it works but when i create a scala project and try to run it i get the above error. what am i doing wrong ?

Try changing ` val rowRDD = peopleLines.map(p=>{ Row.fromSeq(p.split(delimiter)) })` to ` val rowRDD = peopleLines.map(p=>{ Row.fromSeq(Seq(p.split(delimiter))) })` — Anahcolus
– Anahcolus, Commented Jun 1, 2017 at 12:46

Michael Armbrust · Accepted Answer · 2015-04-18 03:34:12Z

7

NoSuchMethodError usually means that you have incompatibles between libraries. In this particular case it looks like you might be using a version of spark-csv that requires Spark 1.3 with an older version of Spark.

answered Apr 18, 2015 at 3:34

Michael Armbrust

1,56511 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

NoSuchMethodError: org.apache.spark.sql.SQLContext.applySchema

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related