Pyspark - Error related to SparkContext - no attribute _jsc

Question

Unsure of what the issue is with this. I've seen similar issues regarding this problem, but nothing that solves my issue. Full Error,

Traceback (most recent call last):
  File "C:/Users/computer/PycharmProjects/spark_test/spark_test/test.py", line 4, in <module>
    sqlcontext = SQLContext(sc)
  File "C:\Users\computer\AppData\Local\Programs\Python\Python36\lib\site-packages\pyspark\sql\context.py", line 74, in __init__
    self._jsc = self._sc._jsc
AttributeError: type object 'SparkContext' has no attribute '_jsc'

Here is the simple code I am trying to run:

from pyspark import SQLContext
from pyspark.context import SparkContext as sc

sqlcontext = SQLContext(sc)

df = sqlcontext.read.json('random.json')

yardstick17 · Accepted Answer · 2020-09-29 09:27:39Z

6

If you are using Spark Shell, you will notice that SparkContext is already created.

Otherwise, you can create the SparkContext by importing, initializing and providing the configuration settings. In your case you only passed the SparkContext to SQLContext

import pyspark

conf = pyspark.SparkConf()
# conf.set('spark.app.name', app_name) # Optional configurations

# init & return
sc = pyspark.SparkContext.getOrCreate(conf=conf)
sqlcontext = SQLContext(sc)

df = sqlcontext.read.json('random.json')

edited Sep 29, 2020 at 9:27

answered May 30, 2019 at 7:53

yardstick17

4,6321 gold badge29 silver badges36 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Pyspark - Error related to SparkContext - no attribute _jsc

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related