Pyspark create multiple columns from dictionary column

Question

@udf(returnType=MapType(StringType(), FloatType()))
def postprocess(data):
    ret = dict()
    ....
    # insert key and values to dictionary from data
    ...

    return ret

ret = postprocess(col('data'))
print(ret) # Column<'postprocess(data)'>

I would like to create multiple columns from dictionary column.

If ret has {"key1": 0.1, "key2": 0.3}, the result should be

| key1 | key2 |

| 0.1 | 0.3 |

How can I create it?

Jonathan · Accepted Answer · 2022-12-20 06:26:04Z

1

To achieve your goal, you can use .explode() to create multiple columns from a dictionary column. Details: https://spark.apache.org/docs/3.1.3/api/python/reference/api/pyspark.sql.functions.explode.html

However, in the performance perspective, not sure how complicated your UDF is, I think you should use the spark sql function to create the columns instead of using the Python UDF function if it's possible. You can check this post: https://stackoverflow.com/a/38297050/10445333

answered Dec 20, 2022 at 6:26

Jonathan

2,3332 gold badges12 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Pyspark create multiple columns from dictionary column

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related