-Spark Scala Mongodb- MongoTypeConversionException Cannot cast STRING into a StructType(...)

Question

Any help would be very much appreciated.

I am trying to build a dataframe using data from mongodb.

val spark = SparkSession.builder()
      .master("local")
      .appName("app")
      .config("spark.mongodb.input.uri", uri)
      .config("spark.mongodb.input.collection", "collectionName")
      .config("spark.mongodb.input.readPreference.name", "secondary")
      .getOrCreate()

val df = MongoSpark.load(spark).limit(1)

and from there i'm trying to read elements row by row, and the schema of the dataframe looks something like this:

root
 |-- A: struct (nullable = true)
 |    |-- oid: string (nullable = true)
 |-- B: boolean (nullable = true)
 |-- C: string (nullable = true)
 |-- D: string (nullable = true)
 |-- E: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- a: string (nullable = true)
 |    |    |-- b: string (nullable = true)
 |    |    |-- c: string (nullable = true)
 |    |    |-- d: string (nullable = true)

if the dataframe does not include E, dataframe.show() would print out just fine.

However, if the dataframe does inlcude E, then dataframe.show() would give me

Cannot cast STRING into a StructType(StructField(a,StringType,true), StructField(b,StringType,true), StructField(c,StringType,true), StructField(d,StringType,true)) (value: BsonString{value='http://...some url...'})

I tried pretty much every solution related to this problem listed on stackoverflow, but I'm still having no luck passing this error.

How should I approach this problem? Thank you!

Can you add a example of mongodb document that contains E ? It seems that E is actually an array of string instead of an array of struct of strings. — Vincent Doba
– Vincent Doba, Commented Jul 27, 2021 at 17:19
@VincentDoba I posted the screenshot of the example of mongodb document below. — Jay Kwon
– Jay Kwon, Commented Jul 28, 2021 at 1:36

sai saran · Accepted Answer · 2022-11-08 09:05:58Z

0

Issue occurring becz of data read and data conversion issues with Mongosource type connector.

Me also faced the same situation but you can avoid it by below method :

Infer schema =True
save the schema in another object
again read the data with "saved schema", infer schema= False
Load data and you can do whatever analysis with updated one.

answered Nov 8, 2022 at 9:05

sai saran

76710 silver badges33 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Jay Kwon · Accepted Answer · 2021-07-28 01:35:51Z

0

E is actually an array of objects that contains multiple strings.

example of mongodb document

answered Jul 28, 2021 at 1:35

Jay Kwon

133 bronze badges

Collectives™ on Stack Overflow

-Spark Scala Mongodb- MongoTypeConversionException Cannot cast STRING into a StructType(...)

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related