Ruby arrays and Threads

Question

I am populating an array with the responses from API requests. In order to speed up the process I am using threads. I know they are not (really) parallel, but they do make the overall process faster.

To maintain the order I first define an empty array and then I populate the specific slots. This is the simplified code

instances = [array of instances]
number_of_instances = instances.size
case_run_status["machine_statuses"] = Array.new(number_of_instances){{}}

threads = []
instances.each_with_index do |instance, i|

  threads << Thread.new do 
    machine_status = {}

    machine_status["ip"] = instance.public_ip_address



    uri = "request...." 

    response = HTTParty.get(uri)        
    status = JSON.parse(response.body)  

    machine_status["running"] = status['running']
    machine_status["running_node"] = status['running_node']

    case_run_status["machine_statuses"][i] = machine_status

  end
end
threads.each{|thr| thr.join }

From what I understand this should be thread safe. Is this correct? However, the problem that I am having is that, apparently randomly, machine_status["running"] and machine_status["running_node"] get mixed up and the value status['running'] ends up in machine_status["running_node"].

If I remove the Threads and execute the code serially everything works as expected.

Question: Is this the right way to safely populate an array with Threads?

I haven't dug into your code to know if this is even relevant, but you should consider using something like github.com/ruby-concurrency/concurrent-ruby for arrays that you want to be thread-safe. — anothermh
– anothermh, Commented Dec 9, 2019 at 8:03
HTTParty is not thread safe: github.com/jnunemaker/httparty/issues/511 — Casper
– Casper, Commented Dec 9, 2019 at 8:57

cavin kwon · Accepted Answer · 2019-12-09 13:38:11Z

1

I recommend you concurrent-ruby.

install gem install concurrent-ruby
sample code

require 'concurrent'

def api_call(url)
  sleep 1
  # call api
  puts url
  url
end

def async_call(urls)
  jobs = urls.map do |url| 
    Concurrent::Promises.future { api_call(url) }
  end

  before = Time.now
  p Concurrent::Promises.zip(*jobs).value 
  puts Time.now - before
end

In the following code, the url call runs randomly asynchronously. The result is then sorted in the same order as the array.

urls = %w(a b c d e)
async_call(urls)

c 
d 
b 
e 
a 
["a", "b", "c", "d", "e"]
1.0021356

answered Dec 9, 2019 at 13:38

cavin kwon

5013 silver badges5 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Rojj Over a year ago

This looks very interesting. Unfortunately I am running Ruby from within a desktop Application (SketchUp) and there are often issues with gems. A native solution would be preferred. I was thinking to wrap the call to HTTParty within a new class and then instantiate a new object for each thread. There are are just a few threads so resources should not be an issue. I'll test later on

Shivansh Gaur · Accepted Answer · 2019-12-09 21:09:35Z

0

None of the core data structures (except for Queue) in Ruby are thread-safe. The structures are mutable, and when shared between threads, there are no guarantees the threads won’t overwrite each others’ changes.

answered Dec 9, 2019 at 21:09

Shivansh Gaur

9389 silver badges11 bronze badges

Collectives™ on Stack Overflow

Ruby arrays and Threads

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related