How to run Python Script in Pyspark

Question

I'm trying to run Python Script in Pyspark on cloudera VM

First I run pyspark by

$ which pyspark
$ pyspark

After launching the spark, i tried:

$ spark-submit /home/cloudera/test.py

Gives me "name 'spark' is not defined"

$ ./bin/spark-submit /home/cloudera/test.py

Gives me "SyntaxError: invalid syntax"

I know there are many similiar questions online but I still can't figure it out. Can someone please help?

starball · Accepted Answer · 2023-02-12 23:00:53Z

1

You will have to run the spark-submit shell from the cluster itself. You do not have to pyspark into it.

If you want to run the code interactively (type line by line or copy/paste)then you would use pyspark.

edited Feb 12, 2023 at 23:00

starball♦

59.4k52 gold badges312 silver badges1k bronze badges

answered Jun 7, 2017 at 20:44

Hgottipati

214 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

vikrame · Accepted Answer · 2017-02-23 22:17:41Z

0

Check spark is installed as expected by invoking spark-shell. Also try PySpark Shell and try to test whats in your test.py file. Once you are successful try spark-submit

answered Feb 23, 2017 at 22:17

vikrame

4852 silver badges12 bronze badges

Collectives™ on Stack Overflow

How to run Python Script in Pyspark

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related