How to convert WrappedArray to String using Spark / JAVA

Question

I have the following dataframe :

+--------------------+
|    column          |
+--------------------+
| [99896, 10, ]      |     
|[50, 30, 40, ]      |
+--------------------+

Shema of column is :

 |-- column: array (nullable = true)
    |-- element: string (containsNull = true)

When I execute the following code :

for (Iterator<Row> iter = dataframee.toLocalIterator(); iter.hasNext();){
        String item = (iter.next()).get(0).toString();
        System.out.println(item);
    }

I get the following output :

WrappedArray(99896, 10, )
WrappedArray(50, 30, 40, )

How can I convert this output to String like :

[99896, 10,50,30,40 ]

I need your help .

Thank you

Som · Accepted Answer · 2020-08-15 05:40:21Z

Try this-

Load the test data provided

  Dataset<Row> df = spark.sql("select column from values array(99896, 10, null), array(50, 30, 40, null) T(column)");
        df.show(false);
        df.printSchema();
        /**
         * +-------------+
         * |column       |
         * +-------------+
         * |[99896, 10,] |
         * |[50, 30, 40,]|
         * +-------------+
         *
         * root
         *  |-- column: array (nullable = false)
         *  |    |-- element: integer (containsNull = true)
         */

Option-1


      StringBuilder sb = new StringBuilder();
        sb.append("[");
        for (java.util.Iterator<Row> iter = df.toLocalIterator(); iter.hasNext();){
            String item = (iter.next()).getList(0).stream()
                    .filter(Objects::nonNull)
                    .map(String::valueOf)
                    .collect(Collectors.joining(","));
            sb.append(item).append(",");
        }
        int i = sb.lastIndexOf(",");
        sb.replace(i, i+1, "]");
        System.out.println(sb);
        /**
         * [99896,10,50,30,40]
         */

option-2


         Dataset<Row> p = df.withColumn("column",
                expr("concat('[', concat_ws(',', collect_list(concat_ws(',', column))), ']')"));
        for (java.util.Iterator<Row> iter = p.toLocalIterator(); iter.hasNext();){
            String item = (iter.next()).get(0).toString();
            System.out.println(item);
        }
        /**
         * [99896,10,50,30,40]
         */

William Banks · Accepted Answer · 2020-08-14 20:17:32Z

2

So basically, what you're doing is looping through each row, getting the WrappedArray for that row and using WrappedArray's toString() method. What you need to do instead of calling toString() is to loop over that WrappedArray and print each value in it

answered Aug 14, 2020 at 20:17

William Banks

891 silver badge8 bronze badges

Collectives™ on Stack Overflow

How to convert WrappedArray to String using Spark / JAVA

2 Answers 2

Load the test data provided

Option-1

option-2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Load the test data provided

Option-1

option-2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related