Creating Separate Spark dataframe from existing arraytype column

Question

I have a spark dataframe as

with schema

StructType(structField("a",IntegerType,False),structField("b",IntegerType,False),structField("c",ArrayType(structType(structField("d",IntegerType,False),structField("e",IntegerType,False)))

I want to create a separate dataframe from column "c" which is of array type.

Desired output format is

@GamingFelix Thanks for your help. Explode is not dynamically creating separate columns. Can you please help me with exact syntax you are referring to, where you can have n number of columns in array. — Ankush Monga
– Ankush Monga, Commented Aug 25, 2020 at 15:53
Hi! does this help stackoverflow.com/questions/51609740/… ? — GamingFelix
– GamingFelix, Commented Aug 25, 2020 at 19:04

Som · Accepted Answer · 2020-08-25 16:50:21Z

1

Try this-

df.selectExpr("a", "b", "inline_outer(c)").show()

answered Aug 25, 2020 at 16:50

Som

6,3681 gold badge13 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Daeho Ro Over a year ago

Wow. this is really useful.

Collectives™ on Stack Overflow

Creating Separate Spark dataframe from existing arraytype column

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related