1

I want to fill up an array via python multiprocessing.

For example, I want to use "for x in range(2)" to fill up the array via two processors.

The first processor is i=1

The second processor is i=2

However, it shows "TypeError: 'int' object is not iterable"

The code might look like:

import numpy as np
import multiprocessing
from multiprocessing import Process, Queue, Array
import ctypes

def Calculation(i):
    for j in range(0,2):
        Ans[i][j] = Para_1[i][j]*Para_2[i][j]/Para_3[i][j]

if __name__ == "__main__":

    Ans = np.zeros((2,2))

    Para_1 = [[5,4],[1,2]]
    Para_2 = [[1,2],[3,4]]
    Para_3 = [[4,4],[2,5]]

    processes = [Process(target=Calculation, args=(x)) for x in range(2)]

    for p in processes:
        p.start()

    for p in processes:
        p.join()

    print(Ans)
2
  • It would help if you could edit in the complete stacktrace, especially with which line exactly throws that error. Commented May 23, 2019 at 14:21
  • Your first processor is i=0, not 1 Commented May 23, 2019 at 14:27

1 Answer 1

1

The "args" keyword argument to Process needs to be an iterable, which is hinted at by the TypeError exception.

in your code you are providing x in brackets (the same as just giving x), which here is an integer provided by the range function

(x) == x

what you are trying to do, it seems, is provide a tuple with one value. This requires you to add a comma to indicate it is a tuple with one value, rather than single value inside brackets. As per Charles' answer:

processes = [Process(target=Calculation, args=(x,)) for x in range(2)]

you could equally provide a list, which is also an iterable. Then you wouldn't need the comma, since lists use square brackets and so a list with one item won't be interpreted as an item in brackets.

processes = [Process(target=Calculation, args=[x]) for x in range(2)]

Next onto the actual answer to your question...

multiprocessing splits your function into seperate processes, which each have their own memory, so you can't edit an array in your main process with the subprocesses.

You could do this using threading, in which case your threads use the same memory and so this will work. If you don't want to change much, you can use the multiprocessing.dummy library, which uses threading on the backend instead of subprocesses. Just change your import line

from multiprocessing.dummy import Process, Queue, Array

in general this doesn't provide any benefit however. Due to the GIL, threading in python doesn't provide speedup unless you spend a lot of time waiting for something like slow IO. If you want to use multiprocessing, you should refactor your code to have the parallel function return the value it calculates.

Some other notes:

  • you don't need to import multiprocessing then import particular functions from it, just the second import line is enough.

  • you don't use ctypes here, so unless this is just a code snippet you don't need to import that either.

  • Having a function reference variables in an outer scope is bad practice. You should pass Ans, Para_1, Para_2 and Para_3 as arguments

import numpy as np
from multiprocessing.dummy import Process

def Calculation(i):
    for j in range(0,2):
        Ans[i][j] = Para_1[i][j]*Para_2[i][j]/Para_3[i][j]

if __name__ == "__main__":

    Ans = np.zeros((2,2))

    Para_1 = [[5,4],[1,2]]
    Para_2 = [[1,2],[3,4]]
    Para_3 = [[4,4],[2,5]]

    processes = [Process(target=Calculation, args=(x,)) for x in range(2)]

    for p in processes:
        p.start()

    for p in processes:
        p.join()

    print(Ans)

returns

[[1.25 2.  ]
 [1.5  1.6 ]]
Sign up to request clarification or add additional context in comments.

2 Comments

I use Spyder and it returns [[0. 0.] [0. 0.]] without any error message. However, the answer should be [[1.25, 2. ], [1.5 , 1.6 ]].
I use spyder too. I have edited the answer to include my code that output the answer correctly.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.