Error during parameters parse when Spark application is submitting programically

Question

I would like to submit Apache Spark application to YARN programmatically (in Java, not in Scala). When I try do it (my code):

package application.RestApplication;

import org.apache.hadoop.conf.Configuration;
import org.apache.spark.SparkConf;
import org.apache.spark.deploy.yarn.Client;
import org.apache.spark.deploy.yarn.ClientArguments;

public class App {
    public static void main(String[] args1) {
        String[] args = new String[] {
                "--class", "org.apache.spark.examples.JavaWordCount",
                // "--deploy-mode", "cluster",
                // "--master", "yarn",
                // "--driver-memory", "3g",
                // "--executor-memory", "3g",
                "--jar", "/opt/spark/examples/jars/spark-examples_2.11-2.0.0.jar",
                "--arg", "hdfs://hadoop-master:9000/input/file.txt"
        };
        Configuration config = new Configuration();
        System.setProperty("SPARK_YARN_MODE", "true");
        SparkConf sparkConf = new SparkConf();
        ClientArguments cArgs = new ClientArguments(args);
        Client client = new Client(cArgs, config, sparkConf);
        client.run();
    }
}

I get an error in line: ClientArguments cArgs = new ClientArguments(args);:

Exception in thread "main" java.lang.NoSuchMethodError: scala.collection.immutable.$colon$colon.hd$1()Ljava/lang/Object;
    at org.apache.spark.deploy.yarn.ClientArguments.parseArgs(ClientArguments.scala:38)
    at org.apache.spark.deploy.yarn.ClientArguments.<init>(ClientArguments.scala:31)
    at application.RestApplication.App.main(App.java:37)

This is problem with parse String[] args - when array is empty program starts (but no parameters = no work). When I type the correct parameters (as above) or incorrect (eg. "--foo", "foo") I have same error. How can I fix it?

What are your dependencies? How do you assemble the app? What's the CLASSPATH? — Jacek Laskowski
– Jacek Laskowski, Commented Aug 26, 2016 at 21:26
This usually means there is a mismatch between the version of Scala on the server & the version your job was compiled with. Make sure they match (2.11.x for both, or 2.10.x for both) — Dan Osipov
– Dan Osipov, Commented Aug 27, 2016 at 0:13
Thank you. I changed version from 2.10 to 2.11 and it works. — user5729996
– user5729996, Commented Aug 29, 2016 at 7:51

Community · Accepted Answer · 2017-05-23 10:33:45Z

0

Similar question, probably you've got wrong Scala version in your machine.

By the way, you can check Livy, should be good for your job :)

edited May 23, 2017 at 10:33

CommunityBot

11 silver badge

answered Aug 28, 2016 at 10:50

T. Gawęda

16.1k5 gold badges51 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Error during parameters parse when Spark application is submitting programically

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related