Clojure computing array in parallel

Question

I need some help. I have on array(sequence) in size of 4000 elements. I need to compute each part(1000 elements is one part) by four different functions. On function computing one part at least 10 sec. Another functions computing their parts in 1 sec. I need to do it at list 10 times. So, of course I want to do it in parallel. I'm trying to do it using clojure future, but it steel take computing time like a sequence program. At first I think, that's because when i'm using one array and send data to each function each future thread can get access sequental. How can I do this computing in parallel?

here s a simple example of code:

(defn funct [t n]
 (let [ a (future (fun1 (fun_take_i_part_array n 1)))
        b (future (fun2 (fun_take_i_part_array n 2)))
        c (future (fun3 (fun_take_i_part_array n 3)))
        d (future (fun4 (fun_take_i_part_array n 4)))
      ]
 (concatenate @a @b @c @d)
 )

)

So I this future-kind program take longest than sequence for 1 sec.

Each future should run in a separate thread. If fun1 takes 10 sec and the other 3 take 1 sec, it will still take 10 sec for the result to be computed. So, you are not saving much (real) time by using parallelism. — Alan Thompson
– Alan Thompson, Commented Jun 10, 2019 at 13:06

Terminus · Accepted Answer · 2019-06-11 01:21:16Z

The array access is never sequential unless these are not arrays but lazy sequences. It does not need to be because arrays are immutable - that is the distinguishing feature of Clojure. Your slowdown does not come from that.

Your idea of parallelisation is correct, look at this:

(defn fibonacci [n] (case n 1 1 2 1 (+ (fibonacci (- n 2)) (fibonacci (- n 1)))))
(defn parallel-fib [] 
        (let [s1 (future (fibonacci 34)) 
              s2 (future (fibonacci 35)) 
              s3 (future (fibonacci 36)) 
              s4 (future (fibonacci 33))] 
 [@s1 @s2 @s3 @s4]))

 user=> (time (fibonacci 35))
 "Elapsed time: 1514.663902 msecs"
 9227465

 user=> (time (fibonacci 36))
 "Elapsed time: 2403.761528 msecs"
 14930352

 user=> (time (parallel-fib))
 "Elapsed time: 2552.572043 msecs"
 [5702887 9227465 14930352 3524578]

It is clear that the parallel executed futures, running on a 4-core machine, yield time close to the most expensive computation and not to the sum of computation times.

Therefore, I see no obvious reasons why your code experiences times closer to sequential execution. The reasons that come to my mind might be:

fun1 (or any other of those funs) could be a parallel function by itself (e.g. using pmap) consuming all of your cores. Then, the CPU would get crammed and no speedup would be observed.
Somehow 'concatenate' is eating a lot of CPU to put the results together. This should not be a major hassle as if you have used standard Clojure 'concat', it produces a lazy sequence which should incur its access cost only after it's being accessed. Not to mention the access cost should be very low.
If the computations inside futures are very simple, no good scaling could be archived either. We know it is not true, though, because you have already mentioned it takes fun1 10 seconds to complete.

It could help if you could send me the whole code. My email: contact [at] spinney.io.

Collectives™ on Stack Overflow

Clojure computing array in parallel

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related