0

While running test cases, I am encountering the following error:

pyspark.errors.exceptions.captured.IllegalArgumentException: Cannot initialize FileIO implementation org.apache.iceberg.gcp.gcs.GCSFileIO: Cannot find constructor for interface org.apache.iceberg.io.FileIO Missing org.apache.iceberg.gcp.gcs.GCSFileIO [java.lang.NoClassDefFoundError: com/google/auth/Credentials]

I am trying to use gcs file IO instead of the hadoop file io following the iceberg catalog documentation. Could you please advise on how to resolve this?

Improve this question

1 Answer 1

Reset to default
0

The error message you encountered is likely due to missing dependencies in your PySpark environment. According to this related github issue about missing no-arg constructor, the issue was resolved by adding iceberg-gcp to the dependency. Also ensure that you are using the version of iceberg-gcp that corresponds to the version of Iceberg that you are using.

Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

Did this work? Any suggestions or confirmation to my solution would be appreciated.
The issue is iceberg-gcs jar doesn't have com/google/auth/Credentials class file. How can I resolve the issue?
pyspark.errors.exceptions.captured.IllegalArgumentException: Cannot initialize FileIO implementation org.apache.iceberg.gcp.gcs.GCSFileIO: Cannot find constructor for interface org.apache.iceberg.io.FileIO Missing org.apache.iceberg.gcp.gcs.GCSFileIO [java.lang.NoClassDefFoundError: com/google/auth/Credentials]
I checked the iceberg-gcs jar and it is missing Credentials class

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.