How can i change my Spark session in Databricks Community Edition?

Question

I want to change my spark session from 'pyspark.sql.connect.dataframe.DataFrame' to 'pyspark.sql.dataframe.DataFrame' so that I can run StringIndexer and VectorAssembler.

If I run it in pyspark.sql.connect.dataframe.DataFrame, I'm getting an Assertion error with:

File /databricks/python/lib/python3.12/site-packages/pyspark/ml/wrapper.py:87, in JavaWrapper._new_java_obj(java_class, *args) 84 from pyspark.core.context import SparkContext 86 sc = SparkContext._active_spark_context ---> 87 assert sc is not None

Thank you!

Nalini Panwar · Accepted Answer · 2025-09-30 07:32:43Z

2

Answering my own question just in case somebody googles it:

It's not possible in Databricks Community Edition, since they only give a SQL warehouse compute and .connect is the default session. In the paid version, change the session from SQL warehouse to a different compute.

P.S. ML functions are not available in .connect Databricks Spark. You can use ML only through scikit-learn.

edited Sep 30 at 7:32

answered Sep 30 at 7:31

Nalini Panwar

278 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How can i change my Spark session in Databricks Community Edition?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related