How to convert ArrayList into Scala Array in Spark

Question

I want to create a StructType dynamically out of a json file. I iterate my fields and I want to find out how (if it is even possible) can I create some list with my iteration, and then to create a StructType from it.

The code I've tried:

List<StructField> structFieldList = new ArrayList<>();
for (String field : fields.values()) {
  StructField sf = Datatypes.createStructField(field, DataTypes.StringType, true);
  structFieldList.add(sf);
}
StructType structType = new StructType(structFieldList.toArray());

But this one is pretty impossible. Is there any way to figure this out?

Why can't you just add fields to struct one by one, using StructType.add(StructField)? spark.apache.org/docs/latest/api/java/org/apache/spark/sql/… Or maybe I misunderstood the question. — mazaneicha
– mazaneicha, Commented Oct 11, 2021 at 18:07

Vincent Doba · Accepted Answer · 2021-10-11 18:07:01Z

1

Here you don't need to convert ArrayList to Scala Array, as StructType constructor takes java StructField[] array as argument.

Your code can be changed by setting type when calling .toArray() method in your last line of code snippet so it returns a StructField[] array instead of an Object[] array:

List<StructField> structFieldList = new ArrayList<>();
for (String field : fields.values()) {
  StructField sf = DataTypes.createStructField(field, DataTypes.StringType, true);
  structFieldList.add(sf);
}
StructType structType = new StructType(structFieldList.toArray(new StructField[0]));

answered Oct 11, 2021 at 18:07

Vincent Doba

5,1683 gold badges28 silver badges49 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to convert ArrayList into Scala Array in Spark

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related