Spark Error when running python script on databricks

Question

I have the following basic script that works fine using pycharm on my machine.

from pyspark.sql import SparkSession

print("START")

spark = SparkSession \
    .Builder() \
    .appName("myapp") \
    .master('local[*, 4]') \
    .getOrCreate()

print(spark)

data = [('James', '', 'Smith', '1991-04-01', 'M', 3000),
        ('Michael', 'Rose', '', '2000-05-19', 'M', 4000),
        ('Robert', '', 'Williams', '1978-09-05', 'M', 4000),
        ('Maria', 'Anne', 'Jones', '1967-12-01', 'F', 4000),
        ('Jen', 'Mary', 'Brown', '1980-02-17', 'F', -1)
        ]

columns = ["firstname", "middlename", "lastname", "dob", "gender", "salary"]
df = spark.createDataFrame(data=data, schema=columns)
print(df)

However when trying to run on a databricks cluster, directly through python script it gives an error.

START Traceback (most recent call last): File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/Workspace/Repos/***********/sdk_test/tests/snippets/spark_tests.py", line 13, in class SparkTests: File "/Workspace/Repos/*******/sdk_test/tests/snippets/spark_tests.py", line 16, in SparkTests sc = SparkContext.getOrCreate() File "/databricks/spark/python/pyspark/context.py", line 400, in getOrCreate SparkContext(conf=conf or SparkConf()) File "/databricks/spark/python/pyspark/context.py", line 147, in init self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer, File "/databricks/spark/python/pyspark/context.py", line 192, in _do_init raise RuntimeError("A master URL must be set in your configuration") RuntimeError: A master URL must be set in your configuration CalledProcessError: Command 'b'cd ../\n\n/databricks/python3/bin/python -m tests.snippets.spark_tests\n# python -m tests.runner --env=qa --runtime_env=databricks --upload=True --package=sdk\n'' returned non-zero exit status 1.

What am I missing?

Alex Ott · Accepted Answer · 2022-07-07 11:38:59Z

0

Remove .master('local[*, 4]') \ from your code. Master is set automatically on Databricks.

answered Jul 7, 2022 at 11:38

Alex Ott

88.1k10 gold badges110 silver badges157 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

dzoku Over a year ago

still gives the same error :\ I"m running through python file not directly from notebook

Alex Ott Over a year ago

Show how do you trigger the job

dzoku Over a year ago

%%sh cd ../ /databricks/python3/bin/python tests/snippets/spark_tests.py

Alex Ott Over a year ago

it's not the best way of running tests, etc. Look onto dbx instead: docs.databricks.com/dev-tools/dbx.html

Chris · Accepted Answer · 2022-07-07 17:00:59Z

0

Your code works absolutely fine in Databricks:

I will suggest you to restart your databricks cluster and run it again. And check command line arguments for running script.

edited Jul 7, 2022 at 17:00

Chris

139k139 gold badges316 silver badges293 bronze badges

answered Jul 7, 2022 at 11:43

Soumyadeep Debnath

11 bronze badge

1 Comment

dzoku Over a year ago

Thanks for reply, through a notebook it works fine, but if I run it through the python shell (or trigger the script through a job) it throws this error

Collectives™ on Stack Overflow

Spark Error when running python script on databricks

2 Answers 2

4 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related