Python using list parameter in pyspark.sql like a macro in sas

Question

I have a list and want to use in pyspark.sql statement.

VLIST=['afhjh', 'aikn5','hsa76']
INC=pyspark.sql("select * from table1 where VIG=$VLIST")

I tried to use like a sas statement(using $ instead of &) which failed. How can I use it correctly.

Steven · Accepted Answer · 2020-09-28 08:25:48Z

1

You can try something a bit different :

VLIST = ('afhjh', 'aikn5','hsa76')
INC = pyspark.sql(f"select * from table1 where VIG in {VLIST}")

or another way :

from pyspark.sql import functions as F

INC = pyspark.table("table1")
INC = INC.where(F.col("VIG").isin(*(F.lit(val) for val in VLIST)))

answered Sep 28, 2020 at 8:14

Steven

15.4k7 gold badges49 silver badges80 bronze badges

Sign up to request clarification or add additional context in comments.

1 Answer 1