Spark Twitter Streaming exception : (org.apache.spark.Logging) classnotfound

Question

I am trying Spark Twitter Streaming example with Scala using Maven but I am getting below error when I run it:

Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging

Below are my dependencies:

<dependencies>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.10</artifactId>
    <version>2.0.0</version>
</dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming_2.10</artifactId>
    <version>2.0.0</version>
</dependency> 
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming-twitter_2.11</artifactId>
    <version>1.6.2</version> 
</dependency>

I know that the Logging has been moved to org.apache.spark.internal.Logging but I don't know if it is the reason , I already tried to change the version of dependencies to the latest one but with no luck.

have you resolved the issue?

Hasson
– Hasson

2016-08-25 01:03:41 +00:00
Commented Aug 25, 2016 at 1:03 — Hasson
– Hasson, Commented Aug 25, 2016 at 1:03

Ajeet Shah · Accepted Answer · 2016-08-30 05:59:49Z

17

TLDR;

Class org.apache.spark.Logging is available in Spark version 1.5.2 or lower (though I didn't test on all lower versions) but is not available in versions higher than the same.

It all comes down to using incompatible version of Apache Spark:

1. Let's try to `import org.apache.spark.Logging` on `Spark 2.0.0`:

user@ubuntu:~$ /opt/spark/bin/spark-shell
Welcome to
  ____              __
 / __/__  ___ _____/ /__
_\ \/ _ \/ _ `/ __/  '_/
/___/ .__/\_,_/_/ /_/\_\   version 2.0.0
   /_/      
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_101)
scala> import org.apache.spark.Logging
<console>:23: error: object Logging is not a member of package org.apache.spark
import org.apache.spark.Logging
          ^

Class `org.apache.spark.Logging` is not found.

2. Let's try to `import org.apache.spark.Logging` on `Spark 1.6.2`:

(same as above i.e. Class org.apache.spark.Logging is not found.)

3. Let's try to `import org.apache.spark.Logging` on `Spark 1.5.2`:

user@ubuntu:~$ /opt/spark-1.5.2-bin-hadoop2.6/bin/spark-shell
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.5.2
      /_/
Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_101)
scala> import org.apache.spark.Logging
import org.apache.spark.Logging

YES! It is available and successfully imported

As you can see that org.apache.spark.Logging which is required by the Spark-Streaming-Twitter, is available in Spark version 1.5.2 or lower, so I would recommend you to use 1.5.2 or a lower version of spark.

Hence, you should replace your maven dependencies with followings: (Assuming that you are using Scala 2.11.x)

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.11</artifactId>
    <version>1.5.2</version>
</dependency>

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming_2.11</artifactId>
    <version>1.5.2</version>
</dependency>

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming-twitter_2.11</artifactId>
    <version>1.6.2</version>
</dependency>

Note that the artifactId: 2.11 refers to scala version and version: 1.5.2 or 1.6.2 refers to the library (spark-core or spark-streaming-twitter) version.

edited Aug 30, 2016 at 5:59

answered Aug 28, 2016 at 19:05

Ajeet Shah

19.9k10 gold badges66 silver badges107 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

ypriverol Over a year ago

No way to do this without downgrading Apache Spark version.

ypriverol Over a year ago

I changed to <groupId>org.apache.bahir</groupId> and it works, perfectly works. I don't think downgrading spark is the way forward.

Ajeet Shah Over a year ago

@ypriverol yes. What you did is the right thing. You are welcome to edit my answer or I will do it soon.

Mike Slinn · Accepted Answer · 2016-11-04 00:20:29Z

10

Good news! There is another dependency that solves this problem, and is compatible with Spark 2.x.

For SBT, use this dependency:

"org.apache.bahir" %% "spark-streaming-twitter" % "2.0.0"

answered Nov 4, 2016 at 0:20

Mike Slinn

8,4887 gold badges57 silver badges92 bronze badges

1 Comment

Priyanshu Singh Over a year ago

I am using below dependencies but result is same. libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.2.0" // mvnrepository.com/artifact/org.apache.spark/… //libraryDependencies += "org.apache.spark" % "spark-streaming-twitter_2.11" % "1.6.3" % "provided" libraryDependencies+="org.apache.bahir" %% "spark-streaming-twitter" % "2.0.0" //--packages "org.apache.spark:spark-streaming-twitter_2.11:1.6.3" libraryDependencies += "org.apache.spark" % "spark-streaming_2.11" % "2.2.0"

Ahmed Ashour · Accepted Answer · 2017-02-21 10:11:43Z

5

I encountered this exception using spark-streaming-kafka_2.11-1.6.3.jar, and solved it by using spark-streaming-kafka-0-8_2.11-2.1.0.jar instead

edited Feb 21, 2017 at 10:11

Ahmed Ashour

5,66310 gold badges41 silver badges63 bronze badges

answered Feb 21, 2017 at 9:45

hill

711 silver badge2 bronze badges

1 Comment

Omal Perera Over a year ago

Worked for me !

Vikkurthi Manikanta · Accepted Answer · 2017-02-03 11:18:17Z

2

Without changing version, we can solve the issue.

Download spark-core_2.11-1.5.2.logging.jar from https://raw.githubusercontent.com/swordsmanliu/SparkStreamingHbase/master/lib/spark-core_2.11-1.5.2.logging.jar and paste it in spark\jars folder. Include this jar path in spark-submit command . It solved my issue.

answered Feb 3, 2017 at 11:18

Vikkurthi Manikanta

316 bronze badges

Comments

Joseph Kim · Accepted Answer · 2018-05-25 20:54:48Z

0

As @ypriverol suggested, change dependency to org.apache.bahir. Apache Bahir provides extensions to number of Apache analytical platforms including Spark.

<dependency>
    <groupId>org.apache.bahir</groupId>
    <artifactId>spark-streaming-twitter_2.11</artifactId>
    <version>2.2.0</version>
</dependency>

answered May 25, 2018 at 20:54

Joseph Kim

358 bronze badges

Comments

cresclux · Accepted Answer · 2020-04-19 19:57:40Z

0

Try adding https://mvnrepository.com/artifact/commons-logging/commons-logging as a dependency and build the jar again..

answered Apr 19, 2020 at 19:57

cresclux

763 bronze badges

Collectives™ on Stack Overflow

Spark Twitter Streaming exception : (org.apache.spark.Logging) classnotfound

6 Answers 6

1. Let's try to `import org.apache.spark.Logging` on `Spark 2.0.0`:

Class `org.apache.spark.Logging` is not found.

2. Let's try to `import org.apache.spark.Logging` on `Spark 1.6.2`:

3. Let's try to `import org.apache.spark.Logging` on `Spark 1.5.2`:

YES! It is available and successfully imported

3 Comments

1 Comment

1 Comment

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

6 Answers 6

1. Let's try to import org.apache.spark.Logging on Spark 2.0.0:

Class org.apache.spark.Logging is not found.

2. Let's try to import org.apache.spark.Logging on Spark 1.6.2:

3. Let's try to import org.apache.spark.Logging on Spark 1.5.2:

YES! It is available and successfully imported

3 Comments

1 Comment

1 Comment

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related

1. Let's try to `import org.apache.spark.Logging` on `Spark 2.0.0`:

Class `org.apache.spark.Logging` is not found.

2. Let's try to `import org.apache.spark.Logging` on `Spark 1.6.2`:

3. Let's try to `import org.apache.spark.Logging` on `Spark 1.5.2`: