Exception in thread "streaming-start" java.lang.NoSuchMethodError: org.apache.kafka.clients.consumer.KafkaConsumer.subscribe(Ljava/util/Collection;)V

Question

When I submit the spark application, getting the below error:

Exception in thread "streaming-start" java.lang.NoSuchMethodError:

org.apache.kafka.clients.consumer.KafkaConsumer.subscribe(Ljava/util/Collection;)V

Went through the below URL: http://apache-spark-developers-list.1001551.n3.nabble.com/testing-the-kafka-0-10-connector-td17956.html

Set the below environment variable: SPARK_KAFKA_VERSION=0.10

Added the below dependency:

<dependency>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
  <version>2.1.0.cloudera1</version>
</dependency>

Added the below jar: spark-streaming-kafka-0-10-assembly_2.11.jar

Versions I'm using:

Spark-2.0.2
Scala-2.11
Kafka-0.10

A piece of code:

val CLASS_NAME = this.getClass
val log = org.apache.log4j.LogManager.getLogger(CLASS_NAME)

Your help is highly appreciated.

dbustosp · Accepted Answer · 2018-02-08 12:52:26Z

3

Try seting this:

export SPARK_KAFKA_VERSION=0.10

answered Feb 8, 2018 at 12:52

dbustosp

4,48629 silver badges50 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

rams Over a year ago

THANK YOU! This helped me get past the annoying method not found error. You saved my day.

ozw1z5rd Over a year ago

Why is it required ?

Ladislav Zitka Over a year ago

Yeah, why is this required?

Collectives™ on Stack Overflow

Exception in thread "streaming-start" java.lang.NoSuchMethodError: org.apache.kafka.clients.consumer.KafkaConsumer.subscribe(Ljava/util/Collection;)V

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related