How to access SparkContext in pyspark script

Question

The following SOF question How to run script in Pyspark and drop into IPython shell when done? tells how to launch a pyspark script:

 %run -d myscript.py

But how do we access the existin spark context?

Just creating a new one does not work:

 ---->  sc = SparkContext("local", 1)

 ValueError: Cannot run multiple SparkContexts at once; existing 
 SparkContext(app=PySparkShell, master=local) created by <module> at 
 /Library/Python/2.7/site-packages/IPython/utils/py3compat.py:204

But trying to use an existing one .. well what existing one?

In [50]: for s in filter(lambda x: 'SparkContext' in repr(x[1]) and len(repr(x[1])) < 150, locals().iteritems()):
    print s
('SparkContext', <class 'pyspark.context.SparkContext'>)

i.e. there is no variable for a SparkContext instance

What happens when you run this first: from pyspark import SparkContext? — dnlbrky
– dnlbrky, Commented May 4, 2015 at 13:59
With Spark 2.0.0 onwards, the sparkSession which you can create without a clash has a sparkContext property to access the original context. — Akhil Nair
– Akhil Nair, Commented Jan 25, 2017 at 12:20

Jacques Uber · Accepted Answer · 2021-06-02 04:14:15Z

80

Include the following:

from pyspark.context import SparkContext

and then invoke a static method on SparkContext as:

sc = SparkContext.getOrCreate()

edited Jun 2, 2021 at 4:14

Jacques Uber

52 bronze badges

answered Jan 1, 2017 at 7:32

TechnoIndifferent

1,1241 gold badge10 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

ρяσѕρєя K Over a year ago

Add some explanation with answer for how this answer help OP in fixing current issue

TechnoIndifferent Over a year ago

sc is the existing SparkContext OP is looking for. Earlier there was no way to obtain an existing SparkContext, but the static method getOrCreate() was added to get and existing context or create a new one if one does not exist.

Statham Over a year ago

It works for me! 3x! but can you explain that please?

Rene B. · Accepted Answer · 2020-12-04 07:31:52Z

10

If you created a already a SparkSession:

spark = SparkSession \
    .builder \
    .appName("StreamKafka_Test") \
    .getOrCreate()

Then you can access the "existing" SparkContext like this:

sc = spark.sparkContext

answered Dec 4, 2020 at 7:31

Rene B.

7,5548 gold badges54 silver badges81 bronze badges

Comments

vijay kumar · Accepted Answer · 2015-06-22 03:43:31Z

Standalone python script for wordcount : write a reusable spark context by using contextmanager

"""SimpleApp.py"""
from contextlib import contextmanager
from pyspark import SparkContext
from pyspark import SparkConf


SPARK_MASTER='local'
SPARK_APP_NAME='Word Count'
SPARK_EXECUTOR_MEMORY='200m'

@contextmanager
def spark_manager():
    conf = SparkConf().setMaster(SPARK_MASTER) \
                      .setAppName(SPARK_APP_NAME) \
                      .set("spark.executor.memory", SPARK_EXECUTOR_MEMORY)
    spark_context = SparkContext(conf=conf)

    try:
        yield spark_context
    finally:
        spark_context.stop()

with spark_manager() as context:
    File = "/home/ramisetty/sparkex/README.md"  # Should be some file on your system
    textFileRDD = context.textFile(File)
    wordCounts = textFileRDD.flatMap(lambda line: line.split()).map(lambda word: (word, 1)).reduceByKey(lambda a, b: a+b)
    wordCounts.saveAsTextFile("output")

print "WordCount - Done"

to launch:

/bin/spark-submit SimpleApp.py

mnm · Accepted Answer · 2015-06-22 02:47:29Z

1

When you type pyspark at the terminal, python automatically creates the spark context sc.

answered Jun 22, 2015 at 2:47

mnm

2,0224 gold badges22 silver badges49 bronze badges

2 Comments

WestCoastProjects Over a year ago

That's the bin/pyspark program not a standalone pyspark script.

Anahcolus Over a year ago

And sc variable is not created automatically, SparkContext instance is created automatically.

Collectives™ on Stack Overflow

How to access SparkContext in pyspark script

4 Answers 4

3 Comments

Comments

Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

3 Comments

Comments

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related