ValueError: Cannot run multiple SparkContexts at once in spark with pyspark

Question

i am new in using spark , i try to run this code on pyspark

from pyspark import SparkConf, SparkContext
import collections

conf = SparkConf().setMaster("local").setAppName("RatingsHistogram")
sc = SparkContext(conf = conf)

but he till me this erore message

Using Python version 3.5.2 (default, Jul  5 2016 11:41:13)
SparkSession available as 'spark'.
>>> from pyspark import SparkConf, SparkContext
>>> import collections
>>> conf = SparkConf().setMaster("local").setAppName("RatingsHistogram")
>>> sc = SparkContext(conf = conf)



   Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "C:\spark\python\pyspark\context.py", line 115, in __init__
        SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
      File "C:\spark\python\pyspark\context.py", line 275, in _ensure_initialized
        callsite.function, callsite.file, callsite.linenum))
    ValueError: Cannot run multiple SparkContexts at once; existing SparkContext(app=PySparkShell, master=local[*]) created by getOrCreate at C:\spark\bin\..\python\pyspark\shell.py:43
    >>>

i have version spark 2.1.1 and python 3.5.2 , i search and found it is problem in sc ,he could not read it but no when till why , any one have help here

You are using your code inside of pyspark2, which creates a SparkSession for you already. Don't use pyspark shell since you are creating your own SparkContext. Save your code into a Python file and submit it via spark-submit — arun
– arun, Commented Sep 21, 2017 at 19:57
it run in win shell, but not run in jupyter notebook or canopy , at all i will find the problem , thank you a lot @arun — ibrahim
– ibrahim, Commented Sep 21, 2017 at 20:19

Siraj Alam · Accepted Answer · 2017-12-10 08:21:52Z

80

You can try out this

sc = SparkContext.getOrCreate();

edited Dec 10, 2017 at 8:21

Siraj Alam

10.2k10 gold badges61 silver badges76 bronze badges

answered Dec 9, 2017 at 17:52

Maicol Demetrio Lastra Bazán

8168 silver badges3 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

scuberula · Accepted Answer · 2019-08-11 15:40:08Z

28

You can try:

sc = SparkContext.getOrCreate(conf=conf)

edited Aug 11, 2019 at 15:40

scuberula

351 silver badge10 bronze badges

answered Aug 11, 2019 at 13:34

lvjiujin

5814 silver badges6 bronze badges

Comments

Bridget Huang · Accepted Answer · 2021-04-28 18:42:31Z

8

Your previous session is still on. You can run

sc.stop()

answered Apr 28, 2021 at 18:42

Bridget Huang

1031 silver badge8 bronze badges

Comments

kunal · Accepted Answer · 2022-09-22 12:05:21Z

0

it can run through Jupyter lab also. but you have to use as your previous session is still running and local can not run two sessions at a time sc = SparkContext.getOrCreate( conf =conf)

answered Sep 22, 2022 at 12:05

kunal

1

Collectives™ on Stack Overflow

ValueError: Cannot run multiple SparkContexts at once in spark with pyspark

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related