How to Concat 2 column of ArrayType on axis = 1 in Pyspark dataframe?

Question

I have a the following dataframe:

I would like to concatenate the lat and lon into a list. Where mmsi is similar to an ID (This is unique)

+---------+--------------------+--------------------+
|     mmsi|                 lat|                 lon|
+---------+--------------------+--------------------+
|255801480|[47.1018366666666...|[-5.3017783333333...|
|304182000|[44.6343033333333...|[-63.564803333333...|
|304682000|[41.1936, 41.1715...|[-8.7716, -8.7514...|
|305930000|[49.5221333333333...|[-3.6310166666666...|
|306216000|[42.8185133333333...|[-29.853155, -29....|
|477514400|[47.17205, 47.165...|[-58.6317, -58.60...|

Therefore, I would like to concatenate the lat and lon array but on axis = 1, that is, I would like to have at the end a list of lists, in a separate column, like:

[[47.1018366666666, -5.3017783333333], ... ]

How is that could be possible in pyspark dataframe? I have tried concat, but that will return:

[47.1018366666666, 44.6343033333333, ..., -5.3017783333333, -63.564803333333, ...]

Any help is much appreciated!

Vamsi Prabhala · Accepted Answer · 2019-12-09 19:01:50Z

1

Starting Spark version 2.4, you can use the inbuilt function arrays_zip.

from pyspark.sql.functions import arrays_zip
df.withColumn('zipped_lat_lon',arrays_zip(df.lat,df.lon)).show()

answered Dec 9, 2019 at 19:01

Vamsi Prabhala

49.4k4 gold badges41 silver badges64 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

I. A Over a year ago

Thank you @Vamsi for your answer. I just figured out that I am getting back: [{"lat":"47.101836666666664","lon":"-5.301778333333333"},{.... how can I remove the "lat" and "lon"?

Vamsi Prabhala Over a year ago

the return type of the function is a named struct. if you post what actually resulted in that output, i can take a look

Collectives™ on Stack Overflow

How to Concat 2 column of ArrayType on axis = 1 in Pyspark dataframe?

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related