PySpark - Format String using Column Values

Question

I'm trying to create a new column that creates a string using 4 columns.

df.withColumn("input",
                    F.lit("http://address.com/process?field1={}&field2={}&field3={}&field4={}".format(F.col('field1'),F.col('field2'),F.col('field3'),F.col('field4'))).show()

However, where I'm trying to insert the column values into the string, it's showing up as field1=Column<'field1'> instead of the actual value. I tried wraping it in F.format_string as well but I'm not getting the actual values.

So what it should be returning is something like this, assuming that each column has a string = VALUE.

http://address.com/process?field1=VALUE&field2=VALUE&field3=VALUE&field4=VALUE

blackbishop · Accepted Answer · 2021-11-09 23:11:44Z

4

You can use format_string function like this:

import pyspark.sql.functions as F

df = df.withColumn(
    "input",
    F.format_string(
        "http://address.com/process?field1=%s&field2=%s&field3=%s&field4=%s",
        F.col('field1'), F.col('field2'), F.col('field3'), F.col('field4')
    )
)

df.show(truncate=False)

#+------+------+------+------+--------------------------------------------------------------+
#|field1|field2|field3|field4|input                                                         |
#+------+------+------+------+--------------------------------------------------------------+
#|a     |b     |c     |d     |http://address.com/process?field1=a&field2=b&field3=c&field4=d|
#+------+------+------+------+--------------------------------------------------------------+

answered Nov 9, 2021 at 23:11

blackbishop

32.8k11 gold badges61 silver badges86 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

dcrowley01 Over a year ago

This is clean, thanks!

pltc · Accepted Answer · 2021-11-09 21:23:08Z

0

You should use concat, not lit, so something like F.concat(F.lit('http://example.com/'), F.col('field1'), F.lit('/'), F.col('field2'))

answered Nov 9, 2021 at 21:23

pltc

6,0371 gold badge16 silver badges32 bronze badges

Collectives™ on Stack Overflow

PySpark - Format String using Column Values

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related