Ruby Parallel each loop

Question

I have a the following code:

FTP ... do |ftp| 
  files.each do |file| 
  ...
  ftp.put(file)
  sleep 1
  end 
end

I'd like to run the each file in a separate thread or some parallel way. What's the correct way to do this? Would this be right?

Here's my try on the parallel gem

FTP ... do |ftp| 
  Parallel.map(files) do |file| 
  ...
  ftp.put(file)
  sleep 1
  end 
end

The issue with parallel is puts/outputs can occur at the same time like so:

as = [1,2,3,4,5,6,7,8]
results = Parallel.map(as) do |a|
  puts a
end

How can I force puts to occur like they normally would line separated.

Usually I use threach, which uses SizedQueues hence allows you to execute a maximum amount of parallel jobs at time (peach does it too) — mdesantis
– mdesantis, Commented Apr 1, 2014 at 22:12
I just tried parallel looks but not sure if it works correctly yet. — Sten Kin
– Sten Kin, Commented Apr 1, 2014 at 22:16

César Izurieta · Accepted Answer · 2016-06-08 06:56:14Z

5

The whole point of parallelization is to run at the same time. But if there's some part of the process that you'd like to run some of the code sequentially you could use a mutex like:

semaphore = Mutex.new
as = [1,2,3,4,5,6,7,8]
results = Parallel.map(as, in_threads: 3) do |a|
  # Parallel stuff
  sleep rand
  semaphore.synchronize {
    # Sequential stuff
    puts a
  }
  # Parallel stuff
  sleep rand
end

You'll see that it prints stuff correctly but not necesarily in the same order. I used in_threads instead of in_processes (default) because Mutex doesn't work with processes. See below for an alternative if you do need processes.

References:

answered Jun 8, 2016 at 6:56

César Izurieta

4456 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Kache · Accepted Answer · 2014-04-15 21:56:07Z

0

In the interest of keeping it simple, here's what I'd do with built-in Thread:

results = files.map do |file|
  result = Thread.new do
    ftp.put(file)
  end
end

Note that this code assumes that ftp.put(file) returns safely. If that isn't guaranteed, you'll have to do that yourself by wrapping calls in a timeout block and have each thread return an exception if one is thrown and then at the very end of the loop have a blocking check to see that results does not contain any exceptions.

edited Apr 15, 2014 at 21:56

answered Apr 15, 2014 at 21:50

Kache

17k13 gold badges59 silver badges87 bronze badges

Collectives™ on Stack Overflow

Ruby Parallel each loop

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related