java.lang.NoClassDefFoundError: org/apache/spark/Logging

Question

I'm always getting the following error.Can somebody help me please?

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/Logging
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at com.datastax.spark.connector.japi.DStreamJavaFunctions.<init>(DStreamJavaFunctions.java:24)
    at com.datastax.spark.connector.japi.CassandraStreamingJavaUtil.javaFunctions(CassandraStreamingJavaUtil.java:55)
    at SparkStream.main(SparkStream.java:51)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 20 more

When I compile the following code. I've searched the web but didn't find a solution. I've got the error when I added the saveToCassandra.

import com.datastax.spark.connector.japi.CassandraStreamingJavaUtil;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.streaming.Duration;
import org.apache.spark.streaming.api.java.JavaDStream;
import org.apache.spark.streaming.api.java.JavaPairInputDStream;
import org.apache.spark.streaming.api.java.JavaStreamingContext;
import org.apache.spark.streaming.kafka.KafkaUtils;

import java.io.Serializable;
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
import java.util.Set;

import static com.datastax.spark.connector.japi.CassandraJavaUtil.mapToRow;

/**
 * Created by jonas on 10/10/16.
 */
public class SparkStream implements Serializable{
        public static void main(String[] args) throws Exception{
        SparkConf conf = new SparkConf(true)
                .setAppName("TwitterToCassandra")
                .setMaster("local[*]")
                .set("spark.cassandra.connection.host", "127.0.0.1")
                .set("spark.cassandra.connection.port", "9042");
;
        JavaSparkContext sc = new JavaSparkContext(conf);
        JavaStreamingContext ssc = new JavaStreamingContext(sc, new Duration(5000));

        Map<String, String> kafkaParams = new HashMap<>();
        kafkaParams.put("bootstrap.servers", "localhost:9092");
        Set<String> topics = Collections.singleton("Test");

        JavaPairInputDStream<String, String> directKafkaStream = KafkaUtils.createDirectStream(
                ssc,
                String.class,
                String.class,
                kafka.serializer.StringDecoder.class,
                kafka.serializer.StringDecoder.class,
                kafkaParams,
                topics
        );

        JavaDStream<Tweet> createTweet = directKafkaStream.map(s -> createTweet(s._2));


        CassandraStreamingJavaUtil.javaFunctions(createTweet)
                .writerBuilder("mykeyspace", "rawtweet", mapToRow(Tweet.class))
                .saveToCassandra();

        ssc.start();
        ssc.awaitTermination();

    }


    public static Tweet createTweet(String rawKafka){
        String[] splitted = rawKafka.split("\\|");
        Tweet t = new Tweet(splitted[0], splitted[1], splitted[2], splitted[3]);
        return t;
    }
}

My pom is the following.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.company</groupId>
    <artifactId>Sentiment</artifactId>
    <version>1.0-SNAPSHOT</version>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>
        </plugins>
    </build>

    <repositories>
        <repository>
            <id>twitter4j.org</id>
            <name>twitter4j.org Repository</name>
            <url>http://twitter4j.org/maven2</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
    </repositories>

    <dependencies>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_2.11</artifactId>
            <version>2.0.1</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.10</artifactId>
            <version>2.0.0</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.10</artifactId>
            <version>2.0.0</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming-kafka-0-8_2.11</artifactId>
            <version>2.0.1</version>
        </dependency>


        <!-- https://mvnrepository.com/artifact/org.scala-lang/scala-library -->
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>2.11.8</version>
        </dependency>



        <!-- https://mvnrepository.com/artifact/com.datastax.spark/spark-cassandra-connector_2.10 -->
        <dependency>
            <groupId>com.datastax.spark</groupId>
            <artifactId>spark-cassandra-connector_2.10</artifactId>
            <version>1.6.2</version>
        </dependency>

        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka_2.10</artifactId>
            <version>0.9.0.0</version>
        </dependency>

        <dependency>
            <groupId>org.twitter4j</groupId>
            <artifactId>twitter4j-core</artifactId>
            <version>[4.0,)</version>
        </dependency>

        <dependency>
            <groupId>org.twitter4j</groupId>
            <artifactId>twitter4j-stream</artifactId>
            <version>4.0.4</version>
        </dependency>

        <dependency>
            <groupId>org.twitter4j</groupId>
            <artifactId>twitter4j-async</artifactId>
            <version>4.0.4</version>
        </dependency>
    </dependencies>


</project>

Chandana Kumara · Accepted Answer · 2016-10-27 14:47:50Z

14

org.apache.spark.Logging is available in Spark version 1.5.2 or lower version. It is not in the 2.0.0. Pls change versions as follows

    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming_2.11</artifactId>
        <version>1.5.2</version>
    </dependency>

    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.10</artifactId>
        <version>1.5.2</version>
    </dependency>

    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.10</artifactId>
        <version>1.5.2</version>
    </dependency>

    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming-kafka-0-8_2.11</artifactId>
        <version>1.6.2</version>
    </dependency>

edited Oct 27, 2016 at 14:47

answered Oct 27, 2016 at 14:41

Chandana Kumara

2,6651 gold badge25 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Jonas Cristens Over a year ago

Thank you very much

jack AKA karthik Over a year ago

then what is the use of using higher version of spark? How to do it 2.x version spark

Stepan Yakovenko Over a year ago

This fails to build: The POM for org.apache.spark:spark-streaming-kafka-0-8_2.11:jar:1.6.2 is missing, no dependency information available

Chandana Kumara Over a year ago

Try to download from web and copy to your location. This may help to download the jar. jar-download.com/…

RussS · Accepted Answer · 2017-11-09 18:28:18Z

10

The error is because you are using Spark 2.0 libraries with the connector from Spark 1.6 (which looks for the Spark 1.6 logging class. Use the 2.0.5 version of the connector.

edited Nov 9, 2017 at 18:28

answered Oct 27, 2016 at 17:21

RussS

16.6k1 gold badge36 silver badges62 bronze badges

3 Comments

GreenThumb Over a year ago

What if you are using pyspark?

RussS Over a year ago

The answer is the same

Simon30 Over a year ago

Thank you @RussS seems like the best solution, I can't find how to update the connector on python, does anyone know how to?

djzhu · Accepted Answer · 2018-01-23 08:13:08Z

6

It's because of the missing of org.apache.spark.Logging class since 1.5.2, just like everybody say. (Only org.apache.spark.internal.Logging exists in later version... )

But it seems none of the solution against maven can solve this dependency, so I just try to add this class to lib manually. Here is my way to fixed the problem:

Package the scala org.apache.spark.internal.Logging into a public jar. Or download it from https://raw.githubusercontent.com/swordsmanliu/SparkStreamingHbase/master/lib/spark-core_2.11-1.5.2.logging.jar (Thanks to this host.)
Move the jar into your spark cluster's jars directory.
Submit your project again, wish it will help u.

answered Jan 23, 2018 at 8:13

djzhu

1,05412 silver badges18 bronze badges

Comments

Dharman · Accepted Answer · 2020-11-11 16:06:09Z

3

if you're using IntelliJ, just check the 'include dependencies with provided scope' box and that will fix the issue for you without mucking around with the pom or manually downloading the files.

edited Nov 11, 2020 at 16:06

Dharman♦

33.9k27 gold badges106 silver badges157 bronze badges

answered Nov 11, 2020 at 16:00

shivangi kulshrestha

1751 silver badge12 bronze badges

1 Comment

Vasanth Subramanian Over a year ago

I was using IntelliJ and this solution helped me. Thank you! :)

PatPanda · Accepted Answer · 2025-03-20 06:50:34Z

3

I got the solution by changing above mention jar.

Initially, I was having degraded jar for spark-kafka-streaming:

  <dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming-kafka-0-8_2.11</artifactId>
    <version>2.1.1</version></dependency>

I also removed multilple sl4j-log4j.jars and log4j.jars which I had added externally from spark and kafka jar library.

edited Mar 20 at 6:50

PatPanda

5,41833 gold badges143 silver badges281 bronze badges

answered Nov 10, 2017 at 5:50

Narendra Maru

8278 silver badges8 bronze badges

Comments

Priyanshu Singh · Accepted Answer · 2017-10-29 22:07:52Z

1

Download spark-core_2.11-1.5.2.logging.jar and use as --jar option

spark-submit --class com.SentimentTwiteer --packages "org.apache.spark:spark-streaming-twitter_2.11:1.6.3" --jars /root/Desktop/spark-core_2.11-1.5.2.logging.jar /root/Desktop/SentimentTwiteer.jar consumerKey consumerSecret accessToken accessTokenSecret yoursearchTag

https://github.com/sinhavicky4/SentimentTwiteer

answered Oct 29, 2017 at 22:07

Priyanshu Singh

7248 silver badges12 bronze badges

Comments

Priyanshu Singh · Accepted Answer · 2017-10-29 05:30:21Z

0

Downloaded the jar and use --jars in spark-submit ,worked for me spark-submit --class com.SentimentTwiteer --packages "org.apache.spark:spark-streaming-twitter_2.11:1.6.3" --jars /root/Desktop/spark-core_2.11-1.5.2.logging.jar /root/Desktop/SentimentTwiteer.jar XX XX XX XX

answered Oct 29, 2017 at 5:30

Priyanshu Singh

7248 silver badges12 bronze badges

Comments

Sampat Kumar · Accepted Answer · 2018-09-23 04:33:08Z

0

Download the below Jar ,and it into you library ,and it will work as expected.

https://raw.githubusercontent.com/swordsmanliu/SparkStreamingHbase/master/lib/spark-core_2.11-1.5.2.logging.jar

answered Sep 23, 2018 at 4:33

Sampat Kumar

5021 gold badge6 silver badges14 bronze badges

Comments

PatPanda · Accepted Answer · 2025-03-20 06:51:07Z

0

this pom.xml is resolve my issue:

    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-core_2.10</artifactId>
      <version>1.6.1</version>
    </dependency>

    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-streaming_2.10</artifactId>
      <version>1.6.1</version>
    </dependency>

edited Mar 20 at 6:51

PatPanda

5,41833 gold badges143 silver badges281 bronze badges

answered Aug 17, 2017 at 7:11

merlin.chan

111 bronze badge

Comments

PatPanda · Accepted Answer · 2025-03-20 06:52:40Z

One reason that may cause this problem is lib and class conflict. I faced this problem and solved it using some maven exclusions:

       <dependency>
           <groupId>org.apache.spark</groupId>
           <artifactId>spark-core_2.11</artifactId>
           <version>2.0.0</version>
           <scope>provided</scope>
           <exclusions>
               <exclusion>
                   <groupId>log4j</groupId>
                   <artifactId>log4j</artifactId>
               </exclusion>
           </exclusions>
       </dependency>

       <dependency>
           <groupId>org.apache.spark</groupId>
           <artifactId>spark-streaming_2.11</artifactId>
           <version>2.0.0</version>
           <scope>provided</scope>
       </dependency>

       <dependency>
           <groupId>org.apache.spark</groupId>
           <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
           <version>2.0.0</version>
           <exclusions>
               <exclusion>
                   <groupId>org.slf4j</groupId>
                   <artifactId>slf4j-log4j12</artifactId>
               </exclusion>
               <exclusion>
                   <groupId>log4j</groupId>
                   <artifactId>log4j</artifactId>
               </exclusion>
           </exclusions>
       </dependency>

Narendra Reddy · Accepted Answer · 2017-08-02 09:59:53Z

-2

Missing logging Jar in your dependency jars list. Try to download "spark-core_2.11-1.5.2.logging" jar from mvn repository, then add it as external jar to your spark project, you don't get "java.lang.NoClassDefFoundError: org/apache/spark/Logging" error. Based on scala version you can download the jar{2.10,2.11,etc}.

answered Aug 2, 2017 at 9:59

Narendra Reddy

1

4 Comments

Jens Over a year ago

Why downloading and add as external jar. Why not add it as dependency directly in the pom?

Narendra Reddy Over a year ago

Yes I agree with you comment, think if you have installed higher than spark 1.5.2 (ex 2.x.x)in your system what is the use to downloading of all jars related to 1.5.2 version, does it make sense ?, it will through an error like version compatibility.

Priyanshu Singh Over a year ago

I added jar to dependencies .still no go.. Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/Logging at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at

Priyanshu Singh Over a year ago

Downloaded the jar and use --jars in spark-submit spark-submit --class com.SentimentTwiteer --packages "org.apache.spark:spark-streaming-twitter_2.11:1.6.3" --jars /root/Desktop/spark-core_2.11-1.5.2.logging.jar /root/Desktop/SentimentTwiteer.jar XX XX XX XX

PatPanda · Accepted Answer · 2025-03-20 06:52:12Z

-2

it is a version issue try with the latest version

<dependency>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-sql_2.11</artifactId>
  <version>2.1.0</version>
  <scope>provided</scope>
</dependency>
<dependency>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-streaming-kafka-0-8_2.11</artifactId>
  <version>2.1.1</version>
</dependency>
<dependency>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-streaming_2.11</artifactId>
  <version>2.1.0</version>
</dependency>
<dependency>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-core_2.11</artifactId>
  <version>2.1.0</version>
</dependency>

edited Mar 20 at 6:52

PatPanda

5,41833 gold badges143 silver badges281 bronze badges

answered Jun 15, 2017 at 10:57

Manish

1171 silver badge8 bronze badges

Collectives™ on Stack Overflow

java.lang.NoClassDefFoundError: org/apache/spark/Logging

12 Answers 12

4 Comments

3 Comments

Comments

1 Comment

Comments

Comments

Comments

Comments

Comments

Comments

4 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

12 Answers 12

4 Comments

3 Comments

Comments

1 Comment

Comments

Comments

Comments

Comments

Comments

Comments

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related