How to explode each row that is an Array into columns in Spark (Scala)?

Question

I have a Spark DataFrame with the a single column 'value', whereby each row is an Array of equal length. How can I explode this single 'value' column into multiple columns, which follow a schema like this?

Single-column DataFrame

val bronzeDfSchema = new StructType()
  .add("DATE", IntegerType)
  .add("NUMARTS", IntegerType)
  .add("COUNTS", StringType)
  .add("THEMES", StringType)
  .add("LOCATIONS", StringType)
  .add("PERSONS", StringType)
  .add("ORGANIZATIONS", StringType)
  .add("TONE", StringType)
  .add("CAMEOEVENTIDS", StringType)
  .add("SOURCES", StringType)
  .add("SOURCEURLS", StringType)

Thank you!

Rushabh Gujarathi · Accepted Answer · 2021-07-24 20:00:42Z

2

This should work just fine

val schema=Seq(("DATE",0),("NUMARTS",1),("COUNTS",2),("THEMES",3),("LOCATIONS",4),("PERSONS",5),("ORGANIZATIONS",6),("TONE",7),("CAMEOEVENTIDS",8),("SOURCES",9),("SOURCEURLS",10))

val df2=schema.foldLeft(df)((df,x)=>df.withColumn(x._1,col("value").getItem(x._2)))

After you do this just cast the column into the data type you want.

answered Jul 24, 2021 at 20:00

Rushabh Gujarathi

1461 silver badge9 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

viethungha0610 Over a year ago

Thanks, this works! I'm fairly new to Scala so what does the x._1 mean? How is it different from e.g. x(0)?

Rushabh Gujarathi Over a year ago

Welcome, it is the first element of the tuple.

Collectives™ on Stack Overflow

How to explode each row that is an Array into columns in Spark (Scala)?

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related