I'm using Databricks and trying to pass a dataframe from Scala to Python, within the same Scala notebook. I passed a dataframe from Python to Spark using:
%python
python_df.registerTempTable("temp_table")
val scalaDF = table("temp_table")
How do I do that same thing in reverse? Thank you so much!!
The reverse will pretty much the same. In Scala:
scalaDF.registerTempTable("some_table")
In Python:
spark.table("some_table")
If you use recent Spark version you should use createOrReplaceTempView in place of registerTempTable.
You can make use of the .createOrReplaceTempView() method or sql().
Here is an example to pass a dataframe through from scala, python, onto sql with a modification along the way ...and back to scala.
%scala
var df = spark.range(0,10).selectExpr("*","'scala' language_origin")
df.createOrReplaceTempView("tableName")
display(df)
%python
df = sql("select * from tableName")
df2 = df.selectExpr("*","'python' language_added")
df2.createOrReplaceTempView("tableName")
display(df2)
%sql
create or replace temp view tableName as
select *, 'sql' language_added from tableName;
select * from tableName
%scala
df = sql("select * from tableName")
display(df)
pipe to send data to python and back? Is this more efficient?