Convert PySpark DataFrame struct column to string of key-value pairs

Question

I have a struct with a large number of key-value pairs:

|-- struct_col: struct (nullable = false)
|    |-- key1: string (nullable = false)
|    |-- key2: string (nullable = false)
|    |-- key3: string (nullable = false)
|    |-- key4: string (nullable = false)
|    |-- key5: string (nullable = false)
|    |-- (... and so on ...)

I want to turn this into a long string of key-value pairs concatenated together like so:

key1=var1&key2=var2&key3=var3&key4=var4&...

So far I have tried this:

fn.concat_ws("&", *[f"struct_col.{col}" for col in df.select(fn.col("struct_col.*")).columns])

However this only concatenates the values. I know to_json exists using a workflow like this one here, however I would like to use different separators for the key-value pairs and the concatenated struct fields. I would also like to do this dynamically as there is a possibility the struct fields change.

What's the best way to do this?

mck · Accepted Answer · 2020-12-04 12:21:02Z

2

Add one more concat_ws inside the list comprehension:

F.concat_ws("&", *[F.concat_ws("=", F.lit(col), F.col(f"struct_col.{col}")) for col in df.select(F.col("struct_col.*")).columns])

answered Dec 4, 2020 at 12:21

mck

42.7k13 gold badges44 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Convert PySpark DataFrame struct column to string of key-value pairs

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related