spark2.0 error Multiple sources found for json when read json file

Question

when i use spark2.0 read json file like:

Dataset<Row> logDF = spark.read().json(path);
logDF.show();

but it failed :

 16/08/04 15:35:05 ERROR yarn.ApplicationMaster: User class threw exception: java.lang.RuntimeException: Multiple sources found for json (org.apache.spark.sql.execution.datasources.json.JsonFileFormat, org.apache.spark.sql.execution.datasources.json.DefaultSource), please specify the fully qualified class name.
 java.lang.RuntimeException: Multiple sources found for json (org.apache.spark.sql.execution.datasources.json.JsonFileFormat, org.apache.spark.sql.execution.datasources.json.DefaultSource), please specify the fully qualified class name.
  at scala.sys.package$.error(package.scala:27)
at org.apache.spark.sql.execution.datasources.DataSource.lookupDataSource(DataSource.scala:167)
at org.apache.spark.sql.execution.datasources.DataSource.providingClass$lzycompute(DataSource.scala:78)
at org.apache.spark.sql.execution.datasources.DataSource.providingClass(DataSource.scala:78)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:310)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:149)
at org.apache.spark.sql.DataFrameReader.json(DataFrameReader.scala:287)
at org.apache.spark.sql.DataFrameReader.json(DataFrameReader.scala:249)

when I use spark 1.6 it was run correct. the error tell specify the fully qualified class name , but i cant find which class conflict.

thank you very much!

I am not sure how to fix this but no one would be able to do that unless you should add your dependencies file here. — Aseem Bansal
– Aseem Bansal, Commented Aug 4, 2016 at 8:57
Try spark.read().format("json").json(path);. I was facing something similar for csv and found github.com/databricks/spark-csv/issues/367 — Aseem Bansal
– Aseem Bansal, Commented Aug 4, 2016 at 11:51

Shahab Niaz · Accepted Answer · 2020-12-09 11:19:19Z

1

I came across this and found below to work for me.

df = spark.read.format("org.apache.spark.sql.execution.datasources.json.JsonFileFormat").load(path)

More details can be found here https://github.com/AbsaOSS/ABRiS/issues/147

answered Dec 9, 2020 at 11:19

Shahab Niaz

16610 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

spark2.0 error Multiple sources found for json when read json file

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related