How can i solve TypeError: 'SparkContext' object is not callable error?

Question

my code is:

import pyspark
from pyspark.sql import SparkSession
from pyspark.conf import SparkConf
from pyspark import SparkContext

spark = SparkSession.builder \
    .master("local") \
    .appName("pyspark_uygulama") \
    .getOrCreate()

sc = spark.sparkContext()
sc

and I get this error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-2-2fb21ed9bfcc> in <module>
      9     .getOrCreate()
     10 
---> 11 sc = spark.sparkContext()
     12 sc

TypeError: 'SparkContext' object is not callable

First, I changed the spark.SparkContext() to spark.sparkContext() it didn't work(it solved another problem). I update my java development kit to the newest. It didn't solve the error either. Any advice?

Colin · Accepted Answer · 2020-10-21 08:16:55Z

2

the error message is right, spark.sparkContext is surely not callable, check the documentation here(not familiar with pyspark, I will explain using spark with scala)

val sparkContext: SparkContext
The Spark context associated with this Spark session.\

according to the doc, spark.sparkContext does not accept any parameters.

there actually are two sparkContexts: one is in org.apache.spark and another is in org.apache.spark.sql.SparkSession.

for user's conveniences, spark.sparkContext provides one way to access SparkContext without having to instantiate it, but its functionality is limited. People do have to instantiate org.apache.spark.SparkContext themselves for full functionality.

if you do need a sparkContext object of full functionality, I would suggest instantiating SparkContext yourself, if you want to access the methods of SparkContext, like parallelize, textFile and etc, you can do it without calling sparkContext,

sc = spark.sparkContext
sc.parallelize(...) 
sc.textFile(...)

parameters of constructors of the SparkContext inside org.apache.spark (in pyspark, it should be pyspark.SparkContext I guess) can be set through SparkSession.builder().master().appName(...).config(...), so you actually do not need to call spark.sparkContext.

hope this helps and forgive my expression.

edited Oct 21, 2020 at 8:16

answered Oct 21, 2020 at 8:09

Colin

999 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

nosdaet Over a year ago

So, in my case, simply I can delete '()' in the spark.sparkContext because it doesn't accept any parameters. I tried it and worked! Thanks for the expression. I got you

Collectives™ on Stack Overflow

How can i solve TypeError: 'SparkContext' object is not callable error?

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related