Configure Scala Script in IntelliJ IDE to run a spark standalone script through spark-submit

Question

I want to run a standalone Spark script that I've already compiled with sbt package command. How could I set the right configuration of Scala Script to run my script in IntelliJ IDE? Currently I'm using the command line with the following command to run it(but I want to run in IntelliJ to further debugging, for example):

~/spark-1.2.0/bin/spark-submit --class "CoinPipe" target/scala-2.10/coinpipe_2.10-1.0.jar /training/data/dir 7 12

Bellow is a snapshot of what I'm trying to do: The figure shows how I'm trying to configure my script to run in IntelliJ

Here is a 2 step solution where you run spark-submit and then run the debugger in Intellij: bigendiandata.com/… — ecoe
– ecoe, Commented May 14, 2018 at 16:51

Óscar Andreu · Accepted Answer · 2019-05-13 20:06:20Z

5

I realize that this post is old, but I ran into the same problem and found a solution so figured I'd post it here.

Create a java application run configuration with main class:
```
org.apache.spark.deploy.SparkSubmit
```
VM options should include the classpath of spark conf and jars, at a minimum:
```
-cp "c:\spark\conf\;c:\spark\jars\*"
```
Program arguments should contain your jar file as the first argument, followed by the actual program arguments that you wish to pass to your program:
```
yourapp.jar arg1 arg2
```
Use classpath of module should be set to your module.

edited May 13, 2019 at 20:06

Óscar Andreu

1,70014 silver badges32 bronze badges

answered Jul 19, 2018 at 3:40

sudomksandwich

511 silver badge3 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

You're awesome Over a year ago

I correctly chose my module but My IDE showed "org.apache.spark.deploy.SparkSubmit not found in module ". How to fix it?

jwhitlark · Accepted Answer · 2016-05-06 00:43:32Z

0

You can setup spark-submit as an External Tool. It won't be in the run menu, though.

answered May 6, 2016 at 0:43

jwhitlark

5053 silver badges16 bronze badges

Comments

Hrishikesh · Accepted Answer · 2016-06-13 12:18:37Z

0

Try these steps:

Install sbt plugins: Goto File -> Settings -> Plugins -> Install IntelliJ Plugins -> Search for sbt and install it.
After sbt plugin install, restart intellij and Start New Scala sbt project (File -> New Project -> Scala -> SBT).
Now open up the build.sbt file and add the all the dependencies.
Now Create a new Scala class in src -> main -> scala and type in your code.
Right click and hit Run.

NOTE : You might want to restart intelliJ sometime to get the dependencies pulled from the build.sbt file.

edited Jun 13, 2016 at 12:18

answered Mar 5, 2015 at 9:44

Hrishikesh

471 silver badge13 bronze badges

Collectives™ on Stack Overflow

Configure Scala Script in IntelliJ IDE to run a spark standalone script through spark-submit

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related