Pass date string into withColumn

Question

I'm using PySpark and want to add a yyyy_mm_dd string to my DataFrame as a column, I have tried doing it like this:

end_date = '2020-01-20'
final = (
    df1
    .join(df, on = ['id', 'product'], how = 'left_outer')
    .where((sf.col('id').isNotNull()))
    .withColumn('status', when(sf.col('count') >= 10, 3)
    .when((sf.col('count') <= 9) & (sf.col('count') >= 1), 2)
    .when(sf.col('count').isNull(), 1))
    .withColumn('yyyy_mm_dd', end_date)
)
final.fillna(0, subset=['count']).orderBy('id', 'product').show(500,False)

This works without the last .withColumn, but I run into the below error when I include it:

AssertionError: col should be Column

From the docs, it seems I should be passing in a col as the second parameter to withColumn. Though, I'm unsure how to convert my date string to type col. I saw this solution from another post but I don't want to use current_date() since my end_date var will be read in from a coordinator script.

mck · Accepted Answer · 2020-12-22 10:42:07Z

1

Use lit:

.withColumn('yyyy_mm_dd', sf.lit(end_date))

If you want a date type, you can cast accordingly:

.withColumn('yyyy_mm_dd', sf.lit(end_date).cast("date"))

answered Dec 22, 2020 at 10:42

mck

42.7k13 gold badges44 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Zohar Stiro · Accepted Answer · 2020-12-22 10:45:49Z

1

Please check with_column documentation. It get the column name as first arg and a type of col as a second arg. You can use lit() to turn you string to a col With a const value.

pyspark.sql.functions.lit(col) Creates a Column of literal value.

df.select(lit(5).alias('height')).withColumn('spark_user', lit(True)).take(1) [Row(height=5, spark_user=True)]

answered Dec 22, 2020 at 10:45

Zohar Stiro

2412 silver badges5 bronze badges

Collectives™ on Stack Overflow

Pass date string into withColumn

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related