Error when running pyspark

Question

I tried to run pyspark via terminal. From my terminal, I runs snotebook and it will automatically load jupiter. After that, when I select python3, the error comes out from the terminal.

[IPKernelApp] WARNING | Unknown error in handling PYTHONSTARTUP file /Users/simon/spark-1.6.0-bin-hadoop2.6/python/pyspark/shell.py

Here's my .bash_profile setting:

export PATH="/Users/simon/anaconda/bin:$PATH"

export SPARK_HOME=~/spark-1.6.0-bin-hadoop2.6

export PATH=$PATH:$SPARK_HOME/bin

export PYSPARK_DRIVER_PYTHON=jupyter

export PYSPARK_DRIVER_PYTHON_OPTS='notebook'

export PYSPARK_PYTHON=python3

alias snotebook='$SPARK_HOME/bin/pyspark'

Please let me know if you have any ideas, thanks.

SB07 · Accepted Answer · 2017-11-30 02:50:27Z

1

You need to add below line in your code

PYSPARK_DRIVER_PYTHON=ipython

or

PYSPARK_DRIVER_PYTHON=ipython3

Hope it will help.

answered Nov 30, 2017 at 2:50

SB07

767 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

martinarroyo · Accepted Answer · 2018-08-29 13:19:27Z

1

In my case, I was using a virtual environment and forgot to install Jupyter, so it was using some version that it found in the $PATH. Installing it inside the environment fixed this issue.

answered Aug 29, 2018 at 13:19

martinarroyo

9,7713 gold badges42 silver badges79 bronze badges

Comments

Community · Accepted Answer · 2020-06-20 09:12:55Z

0

Spark now includes PySpark as part of the install, so remove the PySpark library unless you really need it.

Remove the old Spark, install latest version. Install (pip) findspark library.

In Jupiter, import and use findspark:

import findspark
findspark.init()

Quick PySpark / Python 3 Check

import findspark

findspark.init()

from pyspark import SparkContext 

sc = SparkContext()

print(sc)

sc.stop()

edited Jun 20, 2020 at 9:12

CommunityBot

11 silver badge

answered Mar 31, 2020 at 10:47

Martin Lewis

1

Collectives™ on Stack Overflow

Error when running pyspark

3 Answers 3

Comments

Comments

Quick PySpark / Python 3 Check

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Quick PySpark / Python 3 Check

Comments

Your Answer

Sign up or log in

Post as a guest

Related