Spark : Exception in thread "main" java.lang.ClassNotFoundException: com.mysql.jdbc.Driver

Question

I wrote simple program in spark to write a dataframe to table in mySql.

The program is as follows:

import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.SQLContext
import org.apache.spark.sql.hive.HiveContext

import org.apache.spark.SparkContext._
import org.apache.spark.rdd._
//import org.apache.spark.rdd.RDD

import org.apache.spark.sql.types._
import org.apache.spark.sql.Row;

import java.util.Properties

import java.sql.{ Connection, DriverManager, SQLException }

object MySQLTrial {
  def main(args: Array[String]) {
    val sparkConf = new SparkConf().setAppName("AnalyseBusStreaming")
    val sc = new SparkContext(sparkConf)
    val df = sc.parallelize(Array((1, 234), (2, 1233)))
    val sqlContext = new org.apache.spark.sql.SQLContext(sc)
    import sqlContext.implicits._
    val prop = new Properties()
    prop.put("user", "admin")
    prop.put("password", "admin")

    val driver = "com.mysql.jdbc.Driver"
    Class.forName(driver)
    val dfWriter = df.toDF().write.mode("append")
    dfWriter.jdbc("jdbc:mysql://127.0.0.1:3306/IOT_DB", "table1", prop)
  }
}

The POM file for my project is as follows

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>ggi.bigdata</groupId>
    <artifactId>internet_of_things</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <dependencies>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.10</artifactId>
            <version>1.6.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-hive_2.10</artifactId>
            <version>1.6.0</version>
        </dependency>
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>5.1.38</version>
        </dependency>
    </dependencies>
</project>

I'm running this program using spark submit (tried on local and yarn mode). I haven't included any jar files explicitly to run this code. I keep getting the error :

Exception in thread "main" java.lang.ClassNotFoundException: com.mysql.jdbc.Driver

What should be done about this ?

eliasah · Accepted Answer · 2016-04-05 20:32:22Z

2

This is because your driver isn't present in the uber-jar that you are submitting to the cluster whether it's a standalone cluster or yarn or mesos, etc.

Solution 1 : Since you are using maven, you can use the assembly plugin to build your uber-jar with all the needed dependencies. More information about maven assembly plugin here.

Solution 2 : Provide these dependency libraries on runtime when you submit your application using the --jars option. I advice your to read ore information about advanced dependencies management and submitting applications in the official documentation.

e.g it can look like this :

./bin/spark-submit \
  --class <main-class>
  --master <master-url> \
  --jars /path/to/mysql-connector-java*.jar

I hope this helps !

answered Apr 5, 2016 at 20:32

eliasah

40.5k12 gold badges128 silver badges159 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

vineet sinha Over a year ago

I have been using maven plugin on eclipse and haven't had this issues before. Does this means that a simple "maven install" to create a fat jar not enough ? I don't want to go to the --jars route because all the executors (or some other URI) would have to have that jar

eliasah Over a year ago

No, the install option isn't enough. Unless you'd want to install all your dependencies on your slaves one by one.

eliasah Over a year ago

Concerning maven plugin, I don't actually understand what you are talking about in the comment. I mentioned the assembly plugin (as one of mavens many) to be used to build uber-jar and I shared you the link to it too :-)

vineet sinha Over a year ago

Exactly, the maven plugin for eclipse (M2Eclipse) does create that uber/fat jar so I don't know why it's not including this dependency. There might be a problem with my build configurations. I'll try out your solutions and let you know if anything works. Thanks ! :)

eliasah Over a year ago

I can't help you with eclipse plugins, sorry ! I do it the old fashioned way, using mvn command line and it works perfectly for me. :)

vineet sinha · Accepted Answer · 2016-04-07 18:55:13Z

0

Eliasah was right. M2Eclipse does create a jar file but it's not a fat/uber jar. If I explicitly install the "maven assembly" plugin on the eclipse, I am able to create a fat jar with the dependency jars included and hence the program runs.

answered Apr 7, 2016 at 18:55

vineet sinha

4171 gold badge5 silver badges12 bronze badges

Collectives™ on Stack Overflow

Spark : Exception in thread "main" java.lang.ClassNotFoundException: com.mysql.jdbc.Driver

2 Answers 2

5 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related