Python shared string memory for multiprocessing

Question

I'm trying to use a shared string variable between my Python processes, but it seems that I'm doing something wrong since I'm getting coredumps and invalid memory values.

I use multiprocessing.Value to create a ctypes.c_char_p value and use the value attribute to access it. In my understanding of the Python docs the value attribute should be synchronized, as long it is an instance of Value (contrary of an instance of RawValue). Is that correct so far?

I've created a short example to demonstrate my use of Value and to show the inconsistency while executing:

from multiprocessing import Process, Value
from ctypes import c_char_p

def process(v):
    while True:
        val = v.value
        print val
        while val == v.value:
            pass

v = Value(c_char_p, None)
p = Process(target=process, args=(v,))
p.start()

for i in range(1,999):
    v.value = str(i)

p.terminate()

Jan Spurny · Accepted Answer · 2014-05-22 20:36:36Z

7

I think the problem may have been caused by using Value(c_char_p) to hold a string value. If you want a string, you should probably just use multiprocessing.Array(c_char).

answered May 22, 2014 at 20:36

Jan Spurny

5,5971 gold badge36 silver badges52 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Manuel Faux Over a year ago

That did the trick. I just found a small, but important section in the docs: see note in gray box. I didn't keep in mind that c_char_p is actually a pointer.

Reishin Over a year ago

here important is that on multiprocessing.Array(c_char, <init string>), should be passed string with max length which is planned to keep in this variable (as array have a fixed length)

Ryan Glenn Over a year ago

This did not work

w5e · Accepted Answer · 2014-05-22 20:44:02Z

2

From the Python-reference: https://docs.python.org/2/library/multiprocessing.html

your_string = Array('B', range(LENGHT))

You can take the identifier for the datatype from the table from the array module reference: https://docs.python.org/2/library/array.html

answered May 22, 2014 at 20:44

w5e

2992 silver badges12 bronze badges

Comments

Jim Scarborough · Accepted Answer · 2015-01-08 21:07:29Z

2

This is very similar in function to your example, though subtly different. Notice that the child process terminates on its own when it gets the None sentinel. The polling loop could consume less CPU if it were to use a timeout.

from multiprocessing import Process, Pipe

def show_numbers(consumer):
    while True:
        if consumer.poll():
           val = consumer.recv()
           if val==None:
              break
           print(val)

(consumer,producer) = Pipe(False)
proc = Process(target=show_numbers, args=(consumer,))
proc.start()

for i in range(1,999):
    producer.send(str(i))

producer.send(None)

proc.join()

answered Jan 8, 2015 at 21:07

Jim Scarborough

1237 bronze badges

Comments

Community · Accepted Answer · 2017-05-23 12:20:54Z

0

I ran into a similar problem when I attempted to set up multiple processes to access a shared I/O resource. It seems that Windows doesn't share a global variable space between processes, and items passed as arguments are squashed and passed by value.

This may not directly relate to your problem, but reading the discussion my help point you in the right direction:

Multiprocess with Serial Object as Parameter

edited May 23, 2017 at 12:20

CommunityBot

11 silver badge

answered May 22, 2014 at 20:48

codingCat

2,4064 gold badges22 silver badges28 bronze badges

1 Comment

Manuel Faux Over a year ago

Neither Windows nor Linux has a global variable space between processes. This is where IPC comes into place and we need shared memory.

Collectives™ on Stack Overflow

Python shared string memory for multiprocessing

4 Answers 4

3 Comments

Comments

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

3 Comments

Comments

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related