Am getting error in scala programming while integrating spark streaming with kafka

Question

Am trying to add some imports , but when I add import org.apache.spark.streaming.kafka.kafkautils its showing the below error

object kafka is not a member of the object org.apache.spark.streaming.kafka.kafkautils

working on eclipse with scala ide 4.7 version 2.11.11, spark-2.3.0-bin-hadoop2.7 jar files, kafka 2.11 jars, spark-streaming-kafka-0-10_2.11-2.3.0 jar

koiralo · Accepted Answer · 2018-03-29 15:46:05Z

1

If you are using spark-streaming-kafka-0-10_2.11-2.3.0 jar then the KafkaUtils is available in org.apache.spark.streaming.kafka010 this package. So import

import org.apache.spark.streaming.kafka010.KafkaUtils

and not

import org.apache.spark.streaming.kafka.kafkautils

Hope this hepls!

answered Mar 29, 2018 at 15:46

koiralo

23.2k6 gold badges57 silver badges77 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Anonymous Over a year ago

Thank you Shankar, that solved my problem , but am getting error at val kafkastream = KafkaUtils.createDirectStream(ssc,"localhost:2181","spark-streaming-consumer-group",Map("mytopic"->5))

Anonymous Over a year ago

near KafkaUtils.createDirectStream its showing error

koiralo Over a year ago

Here is simple example try this one github.com/spirom/spark-streaming-with-kafka/blob/master/src/…

Anonymous Over a year ago

I tried as per the example. it worked. Thank you so much.

Gagan Over a year ago

Did not work in Pyspark. Even i used from pyspark.streaming.kafka010 import KafkaUtils

Collectives™ on Stack Overflow

Am getting error in scala programming while integrating spark streaming with kafka

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related