Apache Spark Row of multiple String fields to single Row with String array converting Exception

Question

I try to implement such code:

    StructType dataStruct = new StructType()
            .add("items", DataTypes.createArrayType(DataTypes.StringType, false), false);
    ExpressionEncoder<Row> encoder = RowEncoder.apply(dataStruct);

    Dataset<Row> arrayItems = transactions.map((MapFunction<Row, Row>) row -> {
        List<String> items = new LinkedList<>();
        for (int i = 1; i <= 12; i++) {
            if (row.getString(i) != null)
                items.add(row.getString(i));
        }
        System.out.println(items);
        return RowFactory.create(items.toArray());
    }, encoder);

to convert dataset with such schema:

|user<String>|item1<String>|item2<String>|item3<String>|...|item12<String>|

to dataset with such schema:

|item<String[]>|

but i take following exception: java.lang.RuntimeException: java.lang.String is not a valid external type for schema of array

I don't understand why RowFactory takes as argument String, not String[]? Can somebody help me, what I should do in this situation?

Data example:

user|item1|item2|item3|item4|item5|item6|item7|item8|item9|item10|item11|item12
Bob|01W|01J|01W|01J|01W|01J|01W|01J|01W|01J|null|null
John|03T|018T|003H|A44I|03T|null|003H|A44I|03T|018T|003H|null
Bill|CMZI|UDAG|01W|null|null|01J|018T|003H|A44I|018T|003H|A44I

please share data, otherwise your code is not going to be reproducible. — mtoto
– mtoto, Commented Jan 25, 2018 at 10:45
Which version of Spark do you use and on which line exception is? — Yehor Krivokon
– Yehor Krivokon, Commented Jan 25, 2018 at 10:51
I use Spark 2.2.1. Exception is on line "return RowFactory.create(items.toArray());" — Jack Loki
– Jack Loki, Commented Jan 25, 2018 at 11:06

Alper t. Turker · Accepted Answer · 2018-01-25 11:34:32Z

2

This happens because varargs in Java are just syntactic sugar and

Object ... values

is equivalent to

Object[] values

so

return RowFactory.create(items.toArray());

will expand the array. You'll need a nested structure:

Object[] rowItems =  {items.toArray()};
RowFactory.create(rowItems));

Further reading Can I pass an array as arguments to a method with variable arguments in Java?

edited Jan 25, 2018 at 11:34

answered Jan 25, 2018 at 11:28

Alper t. Turker

35.3k9 gold badges89 silver badges118 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Alper t. Turker Over a year ago

@JackLoki I am glad I could help. This is a nice question. Would you consider upvote? :)

Collectives™ on Stack Overflow

Apache Spark Row of multiple String fields to single Row with String array converting Exception

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related