Pyspark: Pass parameter to String Column in Dataframe

Question

I'm quite new to PySpark and coming from SAS I still don't get how to handle parameters (or Macro Variables in SAS terminology).

I have a date parameter like "202105" and want to add it as a String Column to a Dataframe. Something like this:

date = 202105
df = df.withColumn("DATE", lit('{date}'))

I think it's quite trivial but so far, I didn't find an exact answer to my problem, maybe it's just too trivial...

Hope you guys can help me out. Best regards

notNull · Accepted Answer · 2021-08-28 12:02:36Z

3

You can use string interpolations i.e. {}.format() (or) f'{}'.

Example:

df.withColumn("DATE", lit("{0}".format(date)))
df.withColumn("DATE", lit("{}".format(date)))

#or

df.withColumn('DATE', lit(f'{date}'))

answered Aug 28, 2021 at 12:02

notNull

31.8k4 gold badges41 silver badges58 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Thanks, the first two worked for me, the third one did not ... maybe it has to do something with the python version or functions? Anyway, solved!