Spark Scala Error: java.lang.NoSuchMethodError: scala.collection.mutable.Buffer$.empty()Lscala/collection/GenTraversable

Question

I am facing following issue while read CSV file using Intellij Scala.

Error Message:

Exception in thread "main" java.lang.NoSuchMethodError: scala.collection.mutable.Buffer$.empty()Lscala/collection/GenTraversable;
    at org.apache.spark.sql.SparkSessionExtensions.<init>(SparkSessionExtensions.scala:72)
    at org.apache.spark.sql.SparkSession$Builder.<init>(SparkSession.scala:780)
    at org.apache.spark.sql.SparkSession$.builder

Source Code:

import org.apache.spark.sql.SparkSession
object broadcastright {

   def main(args : Array[String]): Unit = {
     val spark = SparkSession
       .builder()
       .master("local")
       .appName("Read CSV File")
       .getOrCreate()

     val df = spark.read
       .option("header", "true")
       .option("delimiter", ",")
       .option("inferSchema", "false")
       .load("src\\main\\resources\\people.csv")

     df.show()
   }
  }

Pom.xml

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.12</artifactId>
    <version>2.4.5</version>
</dependency>

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql_2.12</artifactId>
    <version>2.4.0</version>
</dependency>

I am not sure what is the cause for this issue.

from pom.xml your spark uses scala version 2.12 & you have to use same scala version in ide also. — s.polam
– s.polam, Commented May 23, 2020 at 16:59

s.polam · Accepted Answer · 2020-05-23 12:10:12Z

3

If any version difference in spark libraries we will be getting lot of issue as some of methods might not be available in newer version or some new methods might not available in lower version of spark.

Please add same version of spark libraries in maven file. Also scala version in pom.xml and classpath scala version should match.

edited May 23, 2020 at 12:10

answered May 23, 2020 at 12:02

s.polam

10.4k2 gold badges17 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Nasruddin · Accepted Answer · 2021-08-19 12:24:50Z

You run into these type of issues if your Intellij is using incompatible Scala compiler version. This is a very common issue which happens within Intellij workspace if you are building Spark pipeline using Scala.

If you install Scala plugin in your Intellij, it installs a default version of SDK with it. You just have to make sure the installed version of Scala compiler/SDK is compatible with you Spark version.

For example you are using using Spark 3.x.x, be assured that you have Scala 2.12.x compiler Installed.

Process to fix the issue:

File -> Project Structure -> Platform Settings -> Global Libraries -> +

Click on + icon

Click on version on Scala you want to download. In my case Maven 2.12.2

Select Scala version from Dropdown and click ok. In my case 2.12.2

Tag your project against it.

This is how your configuration should look like.

Sam · Accepted Answer · 2021-05-08 18:16:56Z

0

In my case, Scala 2.13 was installed automatically on IntellijIDEA but the spark was running on Scala 2.12.

I removed Scala 2.13 from IntellijIDEA and installed Scala 2.12 using maven.

answered May 8, 2021 at 18:16

Sam

4668 silver badges18 bronze badges

Collectives™ on Stack Overflow

Spark Scala Error: java.lang.NoSuchMethodError: scala.collection.mutable.Buffer$.empty()Lscala/collection/GenTraversable

3 Answers 3

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related