Using Python variable in scala cell

Question

I have the below command in databricks notebook which is in python.

batdf = spark.sql(f"""select cast((from_unixtime((timestamp/1000), 'yyyy-MM-dd HH:mm:ss')) as date) as event_date,(from_unixtime((timestamp/1000), 'yyyy-MM-dd HH:mm:ss')) as event_datetime, * from testable  """)
    
srcRecCount = batdf.count()

I have one more cell in the same notebook which is in scala as below.

%scala
import java.time._
var srcRecCount: Long = 99999
val endPart = LocalDateTime.now()
val endPartDelta =  endPart.toString.substring(0,19)
dbutils.notebook.exit(s"""{'date':'$endPartDelta', 'srcRecCount':'$srcRecCount'}""")

I want to access the variable srcRecCount from python cell into scala cell in databricks notebook. Could you please let me know if this is possible.

Does this answer your question? Pass variables from Scala to Python in Databricks — Szymon Kocot
– Szymon Kocot, Commented Oct 4, 2021 at 20:23

Alex Ott · Accepted Answer · 2021-10-23 10:03:23Z

3

For example, you can pass data via Spark configuration using spark.conf.set & spark.conf.get, like this:

# Python part
srcRecCount = batdf.count()
spark.conf.set("mydata.srcRecCount", str(srcRecCount))

and

// Scala part
val srcRecCount = spark.conf.get("mydata.srcRecCount")
dbutils.notebook.exit(
  s"""{'date':'$endPartDelta', 'srcRecCount':'$srcRecCount'}""")

P.S. But really, do you really need that Scala piece? Why not to do everything in Python?

edited Oct 23, 2021 at 10:03

answered Oct 20, 2021 at 12:12

Alex Ott

88.1k10 gold badges110 silver badges157 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

SanjanaSanju Over a year ago

Thanks for your reply.Could you please let me know how we do dbutils.notebook.exit in python.

Alex Ott Over a year ago

The same way as in Scala - most of dbutils functions are available in Python under the same names

Karthikeyan Rasipalay Durairaj · Accepted Answer · 2021-10-20 15:15:36Z

0

I don't think this is possible . Way Databricks has been configured,When you invoke a language magic command in cell , the command is dispatched to the REPL in the execution context for the notebook.Variables defined in that cell are not available in the REPL of another language/ another cell. REPLs can share state only through external resources such as files in DBFS or objects in object storage. In your case , You are trying to use magic command in both cells . This is expected behavioral . Hope this helps you to understand. Ref : https://docs.databricks.com/notebooks/notebooks-use.html#mix-languages. But still this is possible as workaround , you can write value in temp DBFS location and read it from there.

answered Oct 20, 2021 at 15:15

Karthikeyan Rasipalay Durairaj

2,33922 silver badges42 bronze badges

Collectives™ on Stack Overflow

Using Python variable in scala cell

2 Answers 2

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related