How to launch in Ruby commands in parallel

Question

I am writing a terminal program using ruby as a launcher for a set of program written in C++ and Java which should be executed in a distributed system.

I would like to translate this instruction in ruby:

for i in {1..40}; do
  ssh node$i program & #note & so that that process is detached
done

This is my ruby code:

class Launcher
   # Other method that we can ignore
   def order(command)
     @nodelist.each {#Do something here}
   end 
end

I though about creating a pool of thread and each thread execute that command. Is it the appropriate way? As I studied threads can not execute "exec" since threads share the same memory address.

Do you want your Ruby process to still be alive after it launches everything? Does the Ruby process have to do anything with the output of these commands or wait for them to finish? — Andrew Marshall
– Andrew Marshall, Commented Feb 22, 2015 at 15:58
Each process has at least one thread. As far as I know, all threads of a process are able to do exec. Though exec() replaces one process by another. That means that all the threads that belong to the old process are gone. — hagello
– hagello, Commented Mar 1, 2015 at 21:15

intale · Accepted Answer · 2015-02-23 11:36:45Z

6

+50

Here is the solution:

class Launcher

  def initialize(commands_list)
    #Expected to be an array of command string
    @commands_list = commands_list
  end

  def execute
    p_threads = @commands_list.map do |command|
      Process.detach(Process.spawn(command))
    end
    p_threads.each(&:join)
   end
end

answered Feb 23, 2015 at 11:36

intale

8316 silver badges8 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

hagello · Accepted Answer · 2015-03-01 21:28:39Z

1

Do you know GNU parallel? It's made for running jobs in parallel (surprise!). It is developped by experienced people, tested and tried. You might just use it instead of reimplementing most of it. Or you might have a look in its manual and/or source code, and maybe learn from it.

answered Mar 1, 2015 at 21:28

hagello

3,3052 gold badges31 silver badges39 bronze badges

3 Comments

Vicente Bolea Over a year ago

Does it have a ruby library? Or are you refering about launching gnu parallel using the backquotes?

Vicente Bolea Over a year ago

Can you explain how to extend it to a cluster?

hagello Over a year ago

@Vicente After looking around a bit, I do not think that you can use GNU parallel as a library. You can call it using backquotes, system or Popen3. I do not have any experience in using it on clusters, but you will want to have a look at stackoverflow.com/questions/22236337/…

Collectives™ on Stack Overflow

How to launch in Ruby commands in parallel

2 Answers 2

Comments

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related