java.lang.String is not valid external type for schema of string

Question

Given this, seem have done this in the past ok, but...:

val arrayStructData2 = Seq(
      Row("James", 2),
      Row("Alex", 3)
    )

 val arrayStructSchema2 = new StructType()
                            .add("names",new StructType()
                                 .add("name", StringType)
                                 .add("extraField", IntegerType)
                                )

val df = spark.createDataFrame(spark.sparkContext.parallelize(arrayStructData2),arrayStructSchema2)
df.printSchema()
df.show()

I get this:

...
Caused by: RuntimeException: java.lang.String is not a valid external type for schema of struct<name:string,extraField:int>

Can't see it immediately.

Ged · Accepted Answer · 2021-10-10 15:44:55Z

1

For others, as a reminder, needed Row(Row... as in:

val arrayStructData2 = Seq(
      Row(Row("James", 2)),
      Row(Row("Alex", 3))
    )

Not so obvious error imho.

edited Oct 10, 2021 at 15:44

answered Oct 10, 2021 at 14:49

Ged

18.5k8 gold badges53 silver badges108 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Jarrod Baker · Accepted Answer · 2021-10-11 07:32:49Z

1

When you create the DataFrame with createDataFrame you register the schema, but nothing is actually evaluated which is why df.printSchema works as expected. When you execute df.show the DataFrame is evaluated and Spark tries to load the first value you have given it (in this case a String) into a struct which results in the runtimeException you're seeing. Here is the scaladoc for Spark 3.1.1:

Creates a DataFrame from a java.util.List containing Rows using the given schema. It is important to make sure that the structure of every Row of the provided List matches the provided schema. Otherwise, there will be runtime exception.

It's telling you that you are trying to force a string into a struct.

answered Oct 11, 2021 at 7:32

Jarrod Baker

1,22010 silver badges13 bronze badges

1 Comment

Ged Over a year ago

See my answer..

Collectives™ on Stack Overflow

java.lang.String is not valid external type for schema of string

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related