2

I ran a mapreduce job on hadoop-2.7.0 but this mapreduce job doesnot completed and I faced with this error:

Job job_1491881070758_0003 failed with state FAILED due to: Application application_1491881070758_0003 failed 2 times due to AM Container for appattempt_1491881070758_0003_000002 exited with  exitCode: 1
For more detailed output, check application tracking page:http://erfan:8088/cluster/app/application_1491881070758_0003Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1491881070758_0003_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1: 
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
    at org.apache.hadoop.util.Shell.run(Shell.java:456)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
    at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 1
Failing this attempt. Failing the application.
17/04/11 08:07:15 INFO mapreduce.Job: Counters: 0`

When I traced the reason of this error in containerlogs I confront with this message:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/exceptions/YarnRuntimeException
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2625)
    at java.lang.Class.getMethod0(Class.java:2866)
    at java.lang.Class.getMethod(Class.java:1676)
    at sun.launcher.LauncherHelper.getMainMethod(LauncherHelper.java:494)
    at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:486)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.yarn.exceptions.YarnRuntimeException
    at java.net.URLClassLoader$1.run(URLClassLoader.java:359)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:348)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:347)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    ... 6 more

My POM.xml is:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>giraph.algorithms</groupId>
  <artifactId>minprop</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <packaging>jar</packaging>

  <name>minprop</name>
  <url>http://maven.apache.org</url>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>

  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.giraph</groupId>
        <artifactId>giraph-core</artifactId>
        <version>1.1.0-hadoop2</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-common</artifactId>
        <version>2.7.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-hdfs</artifactId>
        <version>2.7.0</version>
    </dependency>
    <dependency>
        <groupId>commons-io</groupId>
        <artifactId>commons-io</artifactId>
        <version>2.4</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-mapreduce-client-app</artifactId>
        <version>2.7.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-mapreduce-client-common</artifactId>
        <version>2.7.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-mapreduce-client-core</artifactId>
        <version>2.7.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-mapreduce-client-hs</artifactId>
        <version>2.7.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-mapreduce-client-hs-plugins</artifactId>
        <version>2.7.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
        <version>2.7.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-mapreduce-client-shuffle</artifactId>
        <version>2.7.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-yarn-api</artifactId>
        <version>2.7.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-yarn-common</artifactId>
        <version>2.7.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>
            hadoop-yarn-applications-distributedshell
        </artifactId>
        <version>2.7.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-yarn-client</artifactId>
        <version>2.7.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-yarn-registry</artifactId>
        <version>2.7.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>
            hadoop-yarn-server-applicationhistoryservice
        </artifactId>
        <version>2.7.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-yarn-server-common</artifactId>
        <version>2.7.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-yarn-server-nodemanager</artifactId>
        <version>2.7.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-yarn-server-resourcemanager</artifactId>
        <version>2.7.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-yarn-server-sharedcachemanager</artifactId>
        <version>2.7.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-yarn-server-tests</artifactId>
        <version>2.7.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-yarn-server-web-proxy</artifactId>
        <version>2.7.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>
            hadoop-yarn-applications-unmanaged-am-launcher
        </artifactId>
        <version>2.7.0</version>
    </dependency>
  </dependencies>
  <build>
  <pluginManagement>
  <plugins>
                <plugin>
                    <groupId>org.eclipse.m2e</groupId>
                    <artifactId>lifecycle-mapping</artifactId>
                    <version>1.0.0</version>
                    <configuration>
                        <lifecycleMappingMetadata>
                            <pluginExecutions>
                                <pluginExecution>
                                    <pluginExecutionFilter>
                                        <groupId>
                                            org.apache.maven.plugins
                                        </groupId>
                                        <artifactId>
                                            maven-dependency-plugin
                                        </artifactId>
                                        <versionRange>
                                            [2.0,)
                                        </versionRange>
                                        <goals>
                                            <goal>unpack</goal>
                                        </goals>
                                    </pluginExecutionFilter>
                                    <action>
                                        <ignore></ignore>
                                    </action>
                                </pluginExecution>
                            </pluginExecutions>
                        </lifecycleMappingMetadata>
                    </configuration>
                </plugin>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-jar-plugin</artifactId>
                    <version>2.4</version>
                    <configuration>
                        <archive>
                            <manifest>
                                <addClasspath>true</addClasspath>
                                <mainClass>giraph.algorithms.minprop.App</mainClass>
                                <classpathPrefix>dependency-jars/</classpathPrefix>
                            </manifest>
                        </archive>
                    </configuration>
                </plugin>
               <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.6.1</version>
            </plugin>
            </plugins>

        </pluginManagement>
          </build>
</project>

any help appreciated.

Thanks `

Improve this question

2 Answers 2

Reset to default
1

Your servers' JDK version is older than compiler's JDK. You should check your servers' JDK version,

java -version

Or, I think that you don't have to give hadoop -jar.

hadoop jar <jar> [mainClass] args
Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

Thanks. How to know compiler's jdk version?
but my server's jdk version is 1.8 and i think it's new. I will be so grateful if you explain more
1

NoClassDefFoundError is almost always caused by dependency collisions. In addition to being the same version of the jar, you sometimes also need to make sure you're using the same provider (e.g. apache, cloudera, etc) in your build as the server.

It could also be a collision in your project itself. If you are running a maven project, you can do this to find where they are. Example:

mvn dependency:tree -Dverbose -Dincludes=hadoop-yarn-api-2.7.0

Full explanation on that here here.

Then exclude from in the pom.xml(s) where appropriate.

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-yarn-api</artifactId>
    <version>2.7.0</version>
    <exclusions>
        <exclusion>
            <groupId>...</groupId>
            <artifactId>...</artifactId>
        </exclusion>
    </exclusions>
</dependency>
Improve this answer

4 Comments

Thanks. I also append my POM.xml to my question. is there any wrong in my POM? also my system jdk is 1.8.0_121 and my Compiler compliance level is 1.5 in java-compiler in eclipse.
I would bump your Eclipse Java to 1.8 for the sake of consistency and easier debugging for this project, it can't hurt. Re: your pom, you have a ton of yarn artifacts there. Almost definitely some dependency collisions. Do you actually need all of these to compile? Lastly, are you running this against a Hadoop cluster using Apache Hadoop, Cloudera, Hortonworks, or another? Ideally you'll use the version artifact versions in your project as on the server side.
I use apache hadoop (single node). my eclipse doesn't has java compliance level 1.8. only has 1.7. another question is do I need to run this command (mvn dependency:tree -Dverbose -Dincludes=hadoop-yarn-api-2.7.0) for all of the maven dependency after or before adding them to POM.xml?
1.7 will probably be adequate: way better than 1.5! You need to run the dependency:tree after adding it (or its parent) to the pom; otherwise maven won't return any results.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.