Error debugging PySpark after upgrading cluster's Databricks Runtime

Question

I have updated an Azure Databricks cluster from runtime 5.5LTS to 7.3LTS. Now I'm getting an error when I debug in VSCode. I have updated my Anaconda connection like this:

> conda create --name dbconnect python=3.7
> conda activate dbconnect
> pip uninstall pyspark
> pip install -U databricks-connect==7.3.*
> databricks-connect configure
> databricks-connect test

So far so good, but now I'm trying to debug the following

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
setting = spark.conf.get("spark.master")

if "local" in setting:
    from pyspark.dbutils import DBUtils
    dbutils = DBUtils(spark.sparkContext)

On the dbutils = DBUtils(spark.sparkContext), it throws an exception

Exception has occurred: AttributeError 'SparkContext' object has no attribute 'conf'

I have tried creating the conf

from pyspark.dbutils import DBUtils
import pyspark
conf = pyspark.SparkConf()
pyspark.SparkContext.getOrCreate(conf=conf)
dbutils = DBUtils(spark.sparkContext)

but I still get the same error. Can someone tell me what I'm doing wrong please?

blackbishop · Accepted Answer · 2021-03-18 10:59:22Z

5

From the docs Access DBUtils, you need to pass the SparkSession spark not the SparkContext :

from pyspark.sql import SparkSession
from pyspark.dbutils import DBUtils

spark = SparkSession.builder.getOrCreate()

dbutils = DBUtils(spark)

answered Mar 18, 2021 at 10:59

blackbishop

32.8k11 gold badges61 silver badges86 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Error debugging PySpark after upgrading cluster's Databricks Runtime

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related