java.lang.ClassNotFoundException: org.apache.spark.sql.Dataset

Question

When running a Scala file that uses the Spark Dataset type I get the following stack trace:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/Dataset
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
    at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
    at java.lang.Class.getMethod0(Class.java:3018)
    at java.lang.Class.getMethod(Class.java:1784)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:125)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.Dataset
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 6 more

I find this strange because I have the following import:

import org.apache.spark.sql._

Also, in my build.sbt I have the following added to libraryDependencies:

  "org.apache.spark" %% "spark-core" % "1.6.2" % "provided",
  "org.apache.spark" %% "spark-sql" % "1.6.2" % "provided",

How are you running this? If you are submitting to a cluster, is it possible that the spark version is not correct there? — Justin Pihony
– Justin Pihony, Commented Jul 8, 2016 at 14:23
where are you running this? You are excluding the Spark core and sql libraries from your package in your build file. — GameOfThrows
– GameOfThrows, Commented Jul 8, 2016 at 14:27

Simon · Accepted Answer · 2016-07-08 14:39:08Z

46

If you are executing this standalone you can try removing provided from your dependencies. Provided means that you expect the dependencies to already be on the classpath when you run this application. So the Spark dependencies won't be included in your jar if you use provided.

edited Jul 8, 2016 at 14:39

answered Jul 8, 2016 at 14:24

Simon

6,3832 gold badges32 silver badges36 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Gakuo Over a year ago

I have a similar problem running on Intellij but the error will not go away even after removing "provided" . The error includes Error: A JNI error has occurred, please check your installation and try again Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/Dataset – Gakuo just now edit

Galuoises · Accepted Answer · 2021-02-18 15:47:24Z

10

In IntelliJ 2020.3.2 community edition, go to menu run then edit configurations. Finally, in Modify options select 'Include dependencies with "Provided" scope'.

answered Feb 18, 2021 at 15:47

Galuoises

3,35333 silver badges47 bronze badges

Comments

Simson · Accepted Answer · 2019-11-07 13:39:42Z

9

Select checkbox 'Include dependencies with "Provided" scope' in Run/Debug Configurations.

edited Nov 7, 2019 at 13:39

Simson

3,5732 gold badges29 silver badges50 bronze badges

answered Nov 7, 2019 at 12:51

KP87

911 silver badge2 bronze badges

1 Comment

kgui Over a year ago

This worked for me but, curious what it did and why it worked.

soaptree · Accepted Answer · 2017-11-02 21:36:33Z

2

Your build.sbt file specified that the spark dependencies are provided to the application's classpath, but it wasn't able to locate them. If you're not running on a cluster, then you can try removing the "provided" from your build.sbt, or put the Spark dependencies on your classpath.

answered Nov 2, 2017 at 21:36

soaptree

4314 silver badges6 bronze badges

1 Comment

Gakuo Over a year ago

I have a similar problem on Intellij but the error will not go away even after removing "provided" . The error includes Error: A JNI error has occurred, please check your installation and try again Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/Dataset

Collectives™ on Stack Overflow

java.lang.ClassNotFoundException: org.apache.spark.sql.Dataset

4 Answers 4

1 Comment

Comments

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

1 Comment

Comments

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related