java.lang.NoClassDefFoundError: org/apache/spark/internal/Logging

Question

My Spark Streaming program received the following error: Exception in thread "main" java.lang.NoClassDefFoundError:org/apache/spark/internal/Logging My version of spark is 2.1, which is the same as the version running in the cluster.

The information I found on the Internet prompted me that the old version of org.apache.spark.Logging became org.apache.spark.internal.Logging in the new version, which prevented the jar package from being found. But the dependency introduced in my pom is a new version. Why can't I find the jar package?

    <properties>
            <spark.version>2.1.0</spark.version>
            <scala.version>2.11</scala.version>
    </properties>

    <dependencies>
            <dependency>
                    <groupId>org.apache.spark</groupId>
                    <artifactId>spark-core_${scala.version}</artifactId>
                    <version>${spark.version}</version>
            </dependency>
            <dependency>
                    <groupId>org.apache.spark</groupId>
                    <artifactId>spark-streaming_${scala.version}</artifactId>
                    <version>${spark.version}</version>
            </dependency>
            <dependency>
                    <groupId>org.apache.spark</groupId>
                    <artifactId>spark-sql_${scala.version}</artifactId>
                    <version>${spark.version}</version>
            </dependency>
            <dependency>
                    <groupId>org.apache.spark</groupId>
                    <artifactId>spark-hive_${scala.version}</artifactId>
                    <version>${spark.version}</version>
            </dependency>
            <dependency>
                    <groupId>org.apache.spark</groupId>
                    <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
                    <version>2.1.0</version>
            </dependency>
            <dependency>
                    <groupId>org.apache.hadoop</groupId>
                    <artifactId>hadoop-client</artifactId>
                    <version>2.6.0</version>
            </dependency>
            <dependency>
                    <groupId>org.scala-tools</groupId>
                    <artifactId>maven-scala-plugin</artifactId>
                    <version>2.15.2</version>
            </dependency>
            <dependency>
                    <groupId>log4j</groupId>
                    <artifactId>log4j</artifactId>
                    <version>1.2.17</version>
            </dependency>
    </dependencies>
    <build>
            <plugins>
                    <plugin>
                            <groupId>org.scala-tools</groupId>
                            <artifactId>maven-scala-plugin</artifactId>
                            <version>2.15.2</version>
                            <executions>
                                    <execution>
                                            <goals>
                                                    <goal>compile</goal>
                                                    <goal>testCompile</goal>
                                            </goals>
                                    </execution>
                            </executions>
                    </plugin>

                    <plugin>
                            <artifactId>maven-compiler-plugin</artifactId>
                            <version>3.6.0</version>
                            <configuration>
                                    <source>1.8</source>
                                    <target>1.8</target>
                            </configuration>
                    </plugin>
                    <plugin>
                            <groupId>org.apache.maven.plugins</groupId>
                            <artifactId>maven-surefire-plugin</artifactId>
                            <version>2.19</version>
                            <configuration>
                                    <skip>true</skip>
                            </configuration>
                    </plugin>
            </plugins>
    </build>

may be it'll be helpful for u java.lang.NoClassDefFoundError: org/apache/spark/Logging — Raifur-rahim
– Raifur-rahim, Commented Jan 7, 2019 at 6:26

Divya Darshan DD · Accepted Answer · 2019-01-07 08:25:41Z

I have encountered java.lang.NoClassDefFoundError several times.I have Spark-2.3.1 installed so I think it should work in your case too.

In my case, java.lang.NoClassDefFoundError:org/apache/spark/internal/Logging should come from spark-core_2.11-2.3.1.jar and in your case, it should be coming from spark-core_2.11-2.1.0.jar based on your spark and scala version mentioned in the question

When you look at the source code, it is using org.slf4j._ classes. So my recommendation would be to add this dependency in your pom and then try. For NoClassDefFoundError, it is always better to find the jar which is generating this error and then try to backtrack it.

Below is how you can find out which jar is causing the NoClassDefFound error, assuming you have all the dependency jars in ~/spark/jars location.

for i in `ls ~/spark/jars/*.jar`; 
  do 
  jar tvf $i | grep  org/apache/spark/internal/Logging; 
  echo "************** $i ************** "; 
done

Please let me know if this solved your issue.

Collectives™ on Stack Overflow

java.lang.NoClassDefFoundError: org/apache/spark/internal/Logging

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related