how to run a spark scala program in a linux terminal?

Question

I wrote a spark program in scala. now I want to run the script I wrote, in terminal. in pyspark I use spark-submit for a python file. now I want to do the same for my scala program. I do not want to use Intellij or write my program in spark-shell. I just want to write my code in an editor and run it by using a command in terminal. is that possible? Thank you in advance

You'll need to create a JAR. See Submitting Application

philantrovert
– philantrovert

2018-08-17 09:01:31 +00:00
Commented Aug 17, 2018 at 9:01 — philantrovert
– philantrovert, Commented Aug 17, 2018 at 9:01

Praveen L · Accepted Answer · 2018-08-17 09:58:42Z

2

Create a JAR file for your code (jar name is HelloWorld.jar) say . You can use HDFS or local path like below examples.

You can add lot options in below commands which you can found in the url given by philantrovert in comments.

Run in Local mode.

spark-submit --class path.to.YourMainClass  --master local[*] /path/to/your/jar/HelloWorld.jar

Run in cluster mode.

spark-submit --deploy-mode cluster --class path.to.YourMainClass  --master yarn hdfs://nameservice1/hdfsPath/to/your/jar/HelloWorld.jar

answered Aug 17, 2018 at 9:58

Praveen L

9878 silver badges16 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Ali AzG Over a year ago

Thank you. and what "path.to.YourMainClass" is exactly? what should I palce instead of it?

Chandan Ray Over a year ago

It should be your complete package name of the class with your class name

Praveen L Over a year ago

If your package in which you created your program is org.myprograms and your main class is MainClass.... give org.myprograms.MainClass

Collectives™ on Stack Overflow

how to run a spark scala program in a linux terminal?

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related