Getting error while running Scala Spark code to list blobs in storage

Question

I am getting below error while trying to list blobs using google-cloud-storage library:

Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V

I have tried to change version of google-cloud-storage library in build.sbt but getting the same error again and again.

import com.google.auth.oauth2.GoogleCredentials
import com.google.cloud.storage._
import com.google.cloud.storage.Storage.BlobListOption

val credentials: GoogleCredentials = GoogleCredentials.getApplicationDefault()
val storage: Storage = StorageOptions.newBuilder().setCredentials(credentials).setProjectId(projectId).build().getService()

val blobs =storage.list(bucketName, BlobListOption.currentDirectory(), BlobListOption.prefix(path))

My build.sbt looks like this:

version := "0.1"

scalaVersion := "2.11.8"

logBuffered in Test := false

libraryDependencies ++=
  Seq(
    "org.apache.spark" %% "spark-core" % "2.2.0" % "provided",
    "org.apache.spark" %% "spark-sql" % "2.2.0" % "provided",
    "org.scalatest" %% "scalatest" % "3.0.0" % Test,
    "com.typesafe" % "config" % "1.3.1",
    "org.scalaj" %% "scalaj-http" % "2.4.0",
    "com.google.cloud" % "google-cloud-storage" % "1.78.0"
  )

Please, help me.

Igor Dvorzhak · Accepted Answer · 2019-07-02 03:43:23Z

2

This is happening because Spark uses older version of Guava library than google-cloud-storage library that doesn't have Preconditions.checkArgument method. This leads to java.lang.NoSuchMethodError exception.

You can find more detailed answer and instructions on how to fix this issue here.

edited Jul 2, 2019 at 3:43

answered Jul 1, 2019 at 16:48

Igor Dvorzhak

4,4653 gold badges19 silver badges36 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Getting error while running Scala Spark code to list blobs in storage

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related