I have a java program which runs separate small spark programs so How could I make my java program to run small spark modules/programs on cluster?
for example: I have a program name executor(java program).
and some spark programs --add two numbers --subtract two numbers
So how could I run those spark programs on Cluster from my java program i.e. executor in this case.
Thanks!!!
Check spark job server Project, let you create a shared contex and execute jobs from a rest interface: https://github.com/spark-jobserver/spark-jobserver Hope this Will be usefull
Possible solution could be:
Write a bash script and sequentially execute Spark program.
Write all the operations in a single program, and call each operation in a single program and print respective results.
Write a single program but use principle of parallel programming. It means, execute such operation in parallel. Well, it depend on what type of data you have and what you would like to achieve, hard to comment.
You could use SparkLauncher this way:
import org.apache.spark.launcher.SparkLauncher
import scala.collection.JavaConverters._
val env = Map(
"HADOOP_CONF_DIR" -> hadoopConfDir,
"YARN_CONF_DIR" -> yarnConfDir
)
println(env.asJava)
val process = new SparkLauncher(env.asJava)
.setSparkHome(sparkHome)
.setAppResource(jarPath)
.setAppName(appName)
.setMainClass(className) //Main class in your uber spark JAR
.setMaster(master)
//.setConf("spark.driver.memory", "2g") example additional conf prop
.setVerbose(true)
.launch()