Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/ml/feature/VectorAssembler in IntelliJ

Question

I am trying to build a model in Spark ML using Linear Regression in IntellijIDEA.

Before fitting the model, i am supposed to create a VectorAssembler having feature column.

import org.apache.spark.ml.feature.VectorAssembler
import org.apache.spark.ml.linalg.Vectors

//creating features column
val assembler = new VectorAssembler()
  .setInputCols(Array("col4","col5","col6","col7"))
  .setOutputCol("features")

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/ml/feature/VectorAssembler at energydata$.main(energydata.scala:35) at energydata.main(energydata.scala) Caused by: java.lang.ClassNotFoundException: org.apache.spark.ml.feature.VectorAssembler at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 2 more

But this gives error in Intellij. When i tried the same in spark-shell, it works.

can anybody suggest where I might be going wrong here??

name := "hello"

version := "1.0"

scalaVersion := "2.11.8"

libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.1.0"
libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector" % "2.0.6"
libraryDependencies += "org.apache.spark" %% "spark-mllib" % "2.1.0" % "provided"

well, remove the "provided" keyword from the mllib dependency, apparently these classes are missing at runtime — Raphael Roth
– Raphael Roth, Commented Dec 6, 2017 at 12:23

Mith Coriel · Accepted Answer · 2021-11-19 15:37:50Z

2

I had a similar problem with other classes that it couldn't find. And it was because, in the maven dependency, it said "<scope>provided</scope>". I removed that, pressed reload, and now it works.

answered Nov 19, 2021 at 15:37

Mith Coriel

294 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Super Kai - Kazuya Ito · Accepted Answer · 2022-02-10 09:06:48Z

0

I have also face same error while running spark application. i have just remove (%%"provide") scope of dependency spark sql

edited Feb 10, 2022 at 9:06

Super Kai - Kazuya Ito

42.8k23 gold badges258 silver badges257 bronze badges

answered Feb 9, 2022 at 11:49

Vishal rana

197 bronze badges

Collectives™ on Stack Overflow

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/ml/feature/VectorAssembler in IntelliJ

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related