How to share an Array of strings in multiprocessing in Python 3.8?

Question

I am experimenting with Python's (v. 3.8) multiprocessing library, for developing a bigger program, and trying to share an multiprocessing.Array of strings between multiple processes such that this Array can be updated and read by each process with the same data. I tried c_char_p and it says to use byte strings. However, for this code:

from multiprocessing import Process, Array
from ctypes import c_char_p

def show(a):
    print("This ran")
    print("a: ", a[:])

if __name__ == "__main__":
    array = Array(c_char_p, 1)
    array[0] = b'Hello World'
    print(array[:])
    p = Process(target=show, args=(array,))
    p.start()
    p.join()

the output varies from this:

[b'Hello World']
This ran
a:  [b'c']

to this:

[b'Hello World']
This ran

but I expect:

['Hello World']
This ran
['Hello World']

I guess an obvious solution would be to share a common file between each process but I expect to use multiple arrays and that can get a bit tedious. I was wondering what be the best current solution to this in Python 3.8.

For this code:

from multiprocessing import Process, Array
from ctypes import c_char_p

def show(a):
    print("This ran")

    # Decode the encoded values
    arr = [s.decode("utf-8") for s in a]
    print(arr)

if __name__ == "__main__":
    array = Array(c_char_p, 1)
    message = b"Hello, world"
    array[0] = message

    p = Process(target=show, args=(array,))
    p.start()
    p.join()

    # Decode the encoded values
    arr = [s.decode("utf-8") for s in array]
    print(arr)

it never ends (i.e. never gets to the print statement after the process starts), the output is:

This ran

I am not sure why.

Roland Smith · Accepted Answer · 2020-03-22 20:53:59Z

1

I don't think it is supposed to work in general.

Basically, c_char_p is a pointer, a number that references a memory location in a process. And you cannot expect pointers to be portable between processes.

Since Python Process-es inherit from their parent processes it might work for strings that were created before the child process was started.

But you cannot expect to create a new string in the child process, add a pointer to the Array and expect it to be valid in the parent process. (Nor the other way around.)

Using a Pipe or a Queue is probably a better way to exchange string data.

Edit 1:

Use a Pipe to send mutations of a list of strings, e.g. as a tuple (3, "new value"). This would mean that the item 3 in the list now becomes "new value". Using a duplex Pipe means that both ends of the pipe can read and apply every message that comes in.

Edit 2:

Another option would be to use a multiprocessing.Manager. This will allow you to share lists and dicts.

edited Mar 22, 2020 at 20:53

answered Mar 22, 2020 at 20:08

Roland Smith

43.8k3 gold badges69 silver badges98 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

user12055579 Over a year ago

Okay, thanks. When I use a Queue, every time I use the get method to get the list in the Queue, the other processes now see an empty Queue when all I’m trying to do is get what’s in the Queue, update the list, then put it back in the Queue. How am I supposed to solve that problem?

Roland Smith Over a year ago

You would at least need a pipe between the parent process and each child. The parent would have to send each message to all the children. The children only have to send it to the parent.

user12055579 Over a year ago

I never really worked with Pipes, do you mind showing an example of how to do that with at least two processes with an array of strings?

Roland Smith Over a year ago

@user12055579 I don't have a good example on hand. I tend to use multiprocessing.Pool more than individual processes.

user12055579 Over a year ago

Okay. The thing about Pool is that you can't have shared memory between processes.

|

Shubham Sharma · Accepted Answer · 2020-03-22 07:05:40Z

1

The values are stored in array as the encoded strings(byte values), so when you want the string values you have to decode them.

Try this:

from multiprocessing import Process, Array
from ctypes import c_char_p

def show(a):
    print("This ran")

    # Decode the encoded values
    arr = [s.decode("utf-8") for s in a]
    print(arr)

if __name__ == "__main__":
    array = Array(c_char_p, 1)
    array[0] = b"Hello, world"

    # Decode the encoded values
    arr = [s.decode("utf-8") for s in array]
    print(arr)

    p = Process(target=show, args=(array,))
    p.start()
    p.join()

Output:

['Hello, world']
This ran
['Hello, world']

answered Mar 22, 2020 at 7:05

Shubham Sharma

71.8k6 gold badges26 silver badges58 bronze badges

1 Comment

user12055579 Over a year ago

Thanks for your response. My output is not exactly like yours, I get: ['Hello, world'] This ran (with a blank line at the end)

Collectives™ on Stack Overflow

How to share an Array of strings in multiprocessing in Python 3.8?

2 Answers 2

6 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related