Spark Streaming Kafka java.lang.ClassNotFoundException: org.apache.kafka.common.serialization.StringDeserializer

Question

I am using spark streaming with the Kafka integration, When i run the streaming application from my IDE in Local mode, everything works as a charm. However as soon as i submit it to the cluster i keep having the following error:

java.lang.ClassNotFoundException: org.apache.kafka.common.serialization.StringDeserializer

I am using sbt assembly to build my project.

my sbt is as such:

libraryDependencies ++= Seq(
  "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.2.0" % Provided,
  "org.apache.spark" % "spark-core_2.11" % "2.2.0" % Provided,
  "org.apache.spark" % "spark-streaming_2.11" % "2.2.0" % Provided,
  "org.marc4j" % "marc4j" % "2.8.2",
  "net.sf.saxon" % "Saxon-HE" % "9.7.0-20"
)


run in Compile := Defaults.runTask(fullClasspath in Compile, mainClass in (Compile, run), runner in (Compile, run)).evaluated


mainClass in assembly := Some("EstimatorStreamingApp")

I also tried to use the --package option

attempt 1

--packages org.apache.spark:spark-streaming-kafka-0-10_2.11:2.2.0

attempt 2

--packages org.apache.spark:spark-streaming-kafka-0-10-assembly_2.11:2.2.0

All with no success. Does anyone has anything to suggest

Yuval Itzchakov · Accepted Answer · 2017-08-22 12:03:09Z

4

You need to remove the "provided" flag from the Kafka dependency, as it is a dependency not provided OOTB with Spark:

libraryDependencies ++= Seq(
  "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.2.0",
  "org.apache.spark" % "spark-core_2.11" % "2.2.0" % Provided,
  "org.apache.spark" % "spark-streaming_2.11" % "2.2.0" % Provided,
  "org.marc4j" % "marc4j" % "2.8.2",
  "net.sf.saxon" % "Saxon-HE" % "9.7.0-20"
)

answered Aug 22, 2017 at 12:03

Yuval Itzchakov

150k32 gold badges275 silver badges333 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Spark Streaming Kafka java.lang.ClassNotFoundException: org.apache.kafka.common.serialization.StringDeserializer

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related