How to add string to string array column in spark dataset

Question

I have a "Dataset(Row)" as below

+-----+--------------+
|val  |  history     |
+-----+--------------+
|500  |[a=456, a=500]|
|800  |[a=456, a=500]|
|784  |[a=456, a=500]|
+-----+--------------+

Here val is "String" and history is an "string array". I'm trying to add the content in val column to the history column, so that my dataset looks like :

+-----+---------------------+
|val  |  history            |
+-----+---------------------+
|500  |[a=456, b=500, c=500]|
|800  |[a=456, b=500, c=800]|
|784  |[a=456, b=500, c=784]|
+-----+---------------------+

A similar question is discussed here https://stackoverflow.com/a/49685271/2316771 , but I don't know scala and couldn't create a similar java solution.

Please help me to achieve this in java

Are you sure that is a string array? Why does it have keys? — gmds
– gmds, Commented Jun 7, 2019 at 1:18

Oli · Accepted Answer · 2019-06-07 08:14:37Z

4

In Spark 2.4 (not before), you can use the concat function to concat two arrays. In your case, you could do something like:

df.withColumn("val2", concat(lit("c="), col("val")))
  .select(concat(col("history"), array(col("val2")));

NB: the first time I use concat is to concat strings, the second time, to concat arrays. array(col("val2")) creates an array of one element.

answered Jun 7, 2019 at 8:14

Oli

10.5k5 gold badges31 silver badges51 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

DxG · Accepted Answer · 2019-06-07 06:13:56Z

0

I coded a solution but I'm not sure if it can be further optimized

    dataset.map(row -> {
        Seq<String> seq = row.getAs("history");
        ArrayList<String> list = new ArrayList<>(JavaConversions.seqAsJavaList(seq));
        list.add("c="+row.getAs("val"));

        return RowFactory.create(row.getAs("val"),list.toArray(new String[0]));},schema);

answered Jun 7, 2019 at 6:13

DxG

2219 silver badges26 bronze badges

Collectives™ on Stack Overflow

How to add string to string array column in spark dataset

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related