Spark- Exception in thread java.lang.NoSuchMethodError

Question

I am trying to score a model from pmml file using pmml4s library. Every time I submit the job in Spark I get the following error:

20/05/13 23:30:10 ERROR SparkSubmit: org.apache.spark.sql.types.StructType.names(). 
[Ljava/lang/String;
   java.lang.NoSuchMethodError: org.apache.spark.sql.types.StructType.names(). 
    [Ljava/lang/String;
    at org.pmml4s.spark.ScoreModel.transform(ScoreModel.scala:56)
    at com.aexp.JavaPMML.main(JavaPMML.java:24)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)

Following is my code sample:

ScoreModel model = ScoreModel.fromFile(args[0]);
SparkConf conf = new SparkConf();
SparkSession spark = SparkSession.builder().config(conf).getOrCreate();
Dataset<?> df = spark.read().format("csv")
                     .option("header", "true")
                     .option("inferSchema", "true")
                     .load(args[1]);

Dataset<?> scoreDf = model.transform(df);

Following is the pom file that I am using:

<dependencies>
    <dependency>
        <groupId>org.pmml4s</groupId>
        <artifactId>pmml4s-spark_2.11</artifactId>
        <version>0.9.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.11</artifactId>
        <version>2.3.2</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
          <artifactId>spark-core_2.11</artifactId>
          <version>2.3.2</version>
        </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-mllib_2.11</artifactId>
        <version>2.3.2</version>
    </dependency>
</dependencies>

I have edited my pom file and made the spark version similar still I face the same issue. When I am using Scala, I am facing the same problem. Is there any dependency that I am missing?

I have made the spark versions similar, still I face the same issue. — Kunal
– Kunal, Commented May 14, 2020 at 11:02

s.polam · Accepted Answer · 2020-05-14 05:12:53Z

3

Try to use same version of spark libraries. If spark versions are not matching we will be getting NoSuchMethodError issue in many places as those methods might have modified or removed in latest versions.

answered May 14, 2020 at 5:12

s.polam

10.4k2 gold badges17 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

PredictFuture · Accepted Answer · 2020-06-09 09:36:47Z

1

The error is caused by the PMML4S-Spark used the method names of StructType, which is introduced since Spark 2.4. Now it has been fixed in the latest PMML4S-Spark 0.9.5. Please, update your pom file to use the new version:

    <dependency>
        <groupId>org.pmml4s</groupId>
        <artifactId>pmml4s-spark_2.11</artifactId>
        <version>0.9.5</version>
    </dependency>

edited Jun 9, 2020 at 9:36

answered Jun 9, 2020 at 2:33

PredictFuture

2362 silver badges6 bronze badges

Collectives™ on Stack Overflow

Spark- Exception in thread java.lang.NoSuchMethodError

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related