Can i run a shell script through the spark-scala program?

Question

i'm writing a spark-scala program in intelligi, my code is basically to bring the table from oracle and store them in hdfs as text files insert_df.rdd.saveAsTextFile("hdfs://path"). I have tried this way, but it didn't work val script_sh = "///samplepath/file_creation_script.sh".!

But i have some conversions to make to the text file that I generated, I wrote a shell script to that. I dont want to run the spark jar file and the .sh file separately.

Please let me know if there is any way that i can call shell script through the program.

can you try df.write.text("/hdfs/path")

Shankar
– Shankar

2016-10-27 01:44:33 +00:00
Commented Oct 27, 2016 at 1:44 — Shankar
– Shankar, Commented Oct 27, 2016 at 1:44

Yehor Krivokon · Accepted Answer · 2017-08-10 10:42:33Z

2

If you want to save the output of your command to a variable you need use:

import sys.process._
val result = "/path/to/your/script.sh".!! ("!" just execute the command)

And this can work like workaround:

import java.io.{BufferedReader, InputStreamReader}

  val p = new ProcessBuilder("/bin/bash","/path/to/your/script")
  val p2 = p.start()
  val br = new BufferedReader(new InputStreamReader(p2.getInputStream()))

  var line:String = ""
  while ({line = br.readLine();  line!= null}) {
    println(line)
  }

answered Aug 10, 2017 at 10:42

Yehor Krivokon

8775 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Tejwinder · Accepted Answer · 2017-08-10 11:21:44Z

0

I would say try using spark APIs only.

Still if you want to trigger a shell script from spark (1) and (2) worked for me

In client mode:

Just run the shell script from spark code

val cmd = "home/some_script.sh"

cmd!

In cluster mode:

I usually use Oozie , keep the .sh file in /lib folder of the workflow , this will copy the script to the containers.

from code invoke

val cmd = "./some_script.sh"

cmd!

For python "." wasnt required

val cmd = "python some_script.py"

cmd!

If using spark-submit Use --files to copy the script to containers .

Haven't tried 3 , one may try and share.

edited Aug 10, 2017 at 11:21

answered Aug 10, 2017 at 7:19

Tejwinder

259 bronze badges

Collectives™ on Stack Overflow

Can i run a shell script through the spark-scala program?

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related