Submitting Python Application with Apache Spark Submit

Question

I am trying to follow the examples on the Apache Spark documentation site: https://spark.apache.org/docs/2.0.0-preview/submitting-applications.html

I started a Spark standalone cluster and want to run the example Python application. I am in my spark-2.0.0-bin-hadoop2.7 directory and ran the following command

./bin/spark-submit \
--master spark://207.184.161.138:7077 \
examples/src/main/python/pi.py \
1000

However, I get the error

jupyter: '/Users/MyName/spark-2.0.0-bin- \
hadoop2.7/examples/src/main/python/pi.py' is not a Jupyter command

This is what my bash_profile looks like

#setting path for Spark
export SPARK_PATH=~/spark-2.0.0-bin-hadoop2.7
export PYSPARK_DRIVER_PYTHON="jupyter"
export PYSPARK_DRIVER_PYTHON_OPTS="notebook"
alias snotebook='$SPARK_PATH/bin/pyspark --master local[2]'

What am I doing wrong?

Unset PYSPARK_DRIVER_PYTHON and PYSPARK_DRIVER_PYTHON_OPTS before submitting. — zero323
– zero323, Commented Sep 3, 2016 at 11:37

Community · Accepted Answer · 2017-05-23 11:46:31Z

1

The PYSPARK_DRIVER_PYTHON and PYSPARK_DRIVER_PYTHON_OPTS are meant for running the ipython/jupyter shell when opening the pyspark shell ( More info at How to load IPython shell with PySpark ).

You can set this up like:

alias snotebook='PYSPARK_DRIVER_PYTHON=jupyter PYSPARK_DRIVER_PYTHON_OPTS=notebook $SPARK_PATH/bin/pyspark --master local[2]'

So that it doesn't interfere with pyspark when submitting

edited May 23, 2017 at 11:46

CommunityBot

11 silver badge

answered Sep 6, 2016 at 10:51

AbdealiLoKo

3,3572 gold badges23 silver badges39 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

divyanshm · Accepted Answer · 2017-02-14 18:53:53Z

1

Add PYSPARK_DRIVER_PYTHON=ipython before the spark-submit command.

Example:

PYSPARK_DRIVER_PYTHON=ipython ./bin/spark-submit \ 
/home/SimpleApp.py

edited Feb 14, 2017 at 18:53

divyanshm

6,8407 gold badges46 silver badges72 bronze badges

answered Feb 14, 2017 at 18:28

Lei Feng

112 bronze badges

1 Comment

Pithikos Over a year ago

Nice. Only problem is if I want to pass arguments to the python script. For some reason Ipython is interferring, thinking they are for it.

Collectives™ on Stack Overflow

Submitting Python Application with Apache Spark Submit

2 Answers 2

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related