How to use Spark DataFrame with MySQL

Question

OK, I have known that I could use jdbc connector to create DataFrame with this command:

val jdbcDF = sqlContext.load("jdbc", 
Map("url" -> "jdbc:mysql://localhost:3306/video_rcmd?user=root&password=123456",
"dbtable" -> "video"))

But I got this error: java.sql.SQLException: No suitable driver found for ...

And I have tried to add jdbc jar to spark_path with both commands but failed:

spark-shell --jars mysql-connector-java-5.0.8-bin.jar
SPARK_CLASSPATH=mysql-connector-java-5.0.8-bin.jar spark-shell

My Spark version is 1.3.0 while Class.forName("com.mysql.jdbc.Driver").newInstance is worked.

Prem Singh Bist · Accepted Answer · 2015-04-22 14:30:47Z

1

It is caused because the data frame does the find the the Mysql Connector Jar in the class path. This can be resolved by adding the jar to the spark class path as below:

Edit /spark/bin/compute-classpath.sh as

CLASSPATH="$CLASSPATH:$ASSEMBLY_JAR:yourPathToJar/mysql-connector-java-5.0.8-bin.jar"

Save the file and Restart the spark.

answered Apr 22, 2015 at 14:30

Prem Singh Bist

1,3136 gold badges22 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

ofirski · Accepted Answer · 2015-09-01 11:35:22Z

0

You might want to try mysql-connector-java-5.1.29-bin.jar

edited Sep 1, 2015 at 11:35

ofirski

7417 silver badges18 bronze badges

answered Mar 22, 2015 at 16:46

hadoop bigdata

16

3 Comments

Ganesh Sittampalam Over a year ago

Is there some specific reason you expect this to help? If so perhaps you could edit your post to include that.

2shou Over a year ago

I have updated to mysql-connector-java-5.1.34-bin.jar and it worked.

Zhiyong Over a year ago

mysql-connector-java-5.1.38-bin.jar works too, don't know what's wrong with 5.0.8.

Collectives™ on Stack Overflow

How to use Spark DataFrame with MySQL

2 Answers 2

Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related