How to convert rows into string values in Apache Spark

Question

I have a spark dataframe like this:

fruit | name
--------------
fruit | apple
fruit | orange
fruit | mango

I want to convert it into this:

fruit | string
----------------------------
fruit | apple, orange, mango

How can I achieve this in Apache Spark?

Possible duplicate of SPARK SQL replacement for mysql GROUP_CONCAT aggregate function — mtoto
– mtoto, Commented Nov 30, 2016 at 12:30

Community · Accepted Answer · 2017-05-23 12:24:15Z

2

Look at collect_list

df.groupBy("fruit").agg(collect_list("name"))

it will group values and create array of them as a new column.

If you want to have string, please see this question (thanks @mtoto)

CommunityBot

11 silver badge

answered Nov 30, 2016 at 10:13

T. Gawęda

16.1k5 gold badges51 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

1 Answer 1