How to pass variable arguments to a Spark Dataframe using PySpark?

Question

I am using the Crealytics Spark library to read an Excel Workbook into a Spark Dataframe using a Databricks Python notebook.

Hardcoded like this works fine:

df = spark.read.format("com.crealytics.spark.excel")
     .option("useHeader","true")
     .option("dataAddress","'Sheet1'!")
     .load("/FileStore/tables/Test.xlsx")

I would like to read a dynamic list of options from a table into a PySpark structure (such as list or dict) and pass these to the DataFrame as varargs.

However, it fails even when trying to pass in just one option:

test = {"useHeader":"True"}

df = spark.read.format("com.crealytics.spark.excel")
     .option(*test)
     .option("dataAddress","'Sheet'!")
     .load("/FileStore/tables/Test.xlsx")

TypeError: option() takes exactly 3 arguments (2 given)

user11042628 · Accepted Answer · 2019-02-10 22:13:46Z

2

Use options not option

options(**options)

Adds input options for the underlying data source.

As you can see from the signature, it takes keyword arguments, hence dictionary unpacking will be a valid way to provide these.

answered Feb 10, 2019 at 22:13

user11042628

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to pass variable arguments to a Spark Dataframe using PySpark?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related