Pass String from SCALA to Python in Databricks

Question

I have the following question. I want to pass the value of a variable (string) from scala to python in databricks. I know I can transfer dataframe information between the two languages using this command:

%scala
scalaDF.registerTempTable("some_table")

%python
spark.table("some_table")

But I can't transfer a string this way, any ideas?

You can't really. Either save it to a new dataframe/rdd or if operating on master node to a file. Yuck. — toni057
– toni057, Commented Jan 26, 2021 at 12:08

code.gsoni · Accepted Answer · 2021-01-26 12:42:01Z

2

Create one broadcast variable in scala spark context and get it's value in pyspark context.

answered Jan 26, 2021 at 12:42

code.gsoni

6953 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

mck · Accepted Answer · 2021-01-26 13:13:36Z

2

You can run a Spark SQL command to save a variable:

%scala

spark.sql("set var1 = 'abc'")

And in Python you can get it back using

%python

spark.sql("select ${var1}").head()[0]

answered Jan 26, 2021 at 13:13

mck

42.7k13 gold badges44 silver badges62 bronze badges

Collectives™ on Stack Overflow

Pass String from SCALA to Python in Databricks

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related