Segmentation fault when creating multiprocessing array

Question

I'm trying to fill a numpy array using multiprocessing, following this post. What I have works fine on my Mac, but when I port it to Ubuntu I get segmentation faults a lot of the time.

I've reduced the code to following minimal example:

import numpy as np
from multiprocessing import sharedctypes

a = np.ctypeslib.as_ctypes(np.zeros((224,224,3)))
print("Made a, now making b")
b = sharedctypes.RawArray(a._type_, a)
print("Finished.")

On Ubuntu 16.04, with Python 3.6.5 and numpy 1.15.4 (same versions as on my Mac), I get the output

Made a, now making b
Segmentation fault (core dumped)

Now, I can change the array dimensions somewhat and in some cases it'll work (e.g., change the first 224 to 100 and it works). But mostly it seg faults.

Can anyone offer any insight?

I see one post on a related topic from 2016 that no one responded to, and another one involving pointers which I'm not using.

PS- It doesn't seem to make any difference whether I specify a as a multidimensional array or as a flattened array (e.g. np.zeros(224*224*3)). It also doesn't seem to make a difference if I change the data type (e.g. float to int); it fails the same.

One further update: Even setting "size=224" in the code from the original post causes seg faults on two different Ubuntu machines with different versions of numpy, but works fine on Mac.

Does creating your zeros with np.zeros((224,224), dtype=np.float32) make any difference? It seems like a float32 vs float64 (the default size for standard CPython float) mismatch is happening somewhere, so that may be enough to "fix" it. — tel
– tel, Commented Dec 13, 2018 at 10:49
@tel That works. The following array works on float64 (1, 36861) and segments at (1, 36862) whilst unsurprisingly the following works at float32 (1, 73722) and fails at (1, 73723). — Dobz
– Dobz, Commented Dec 13, 2018 at 10:54
@tel That did not work. float32 or float64 makes no difference. — sh37211
– sh37211, Commented Dec 15, 2018 at 1:19

tel · Accepted Answer · 2018-12-14 01:32:00Z

8

This is more of a guess than an answer, but you may be running into an issue owing to garbage collection of the underlying data buffer. This may explain why there seems to be a dependence on the overall size of the array you're trying to create.

If that's the case, then the fix would be to assign the Numpy array of zeros that you create to it's own variable. This would ensure that the buffer "lives" through the creation of the RawArray. The code would then be:

zs = np.zeros((224,224,3))
a = np.ctypeslib.as_ctypes(zs)
print("Made a, now making b")
b = sharedctypes.RawArray(a._type_, a)
print("Finished.")

I only have a mac right now, so I can't test this out myself.

answered Dec 14, 2018 at 1:32

tel

14k2 gold badges48 silver badges67 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

kabanus Over a year ago

Tested on ubuntu WSL and works! Nice guess. For bigger numbers as well.

kabanus Over a year ago

And Debian as well. You should definitely submit a bug report.

sh37211 Over a year ago

@tel, that did it! Wow, just using an intermediate variable name ("zs") instead of the same variable ("a") is all it took.

anthony sottile · Accepted Answer · 2018-12-16 18:58:27Z

Additional analysis and root-cause fix.

As pointed out above, this is the result of a garbage collection bug, this gave me a hint as to how to fix it.

By keeping the reference around to the original np.zeros object, the bug was avoided. This meant (to me) that the collection of the original object corrupted the resulting array.

Looking at the implementation of as_ctypes (taken from c52543e4a)

def as_ctypes(obj):
    """Create and return a ctypes object from a numpy array.  Actually
    anything that exposes the __array_interface__ is accepted."""
    ai = obj.__array_interface__
    if ai["strides"]:
        raise TypeError("strided arrays not supported")
    if ai["version"] != 3:
        raise TypeError("only __array_interface__ version 3 supported")
    addr, readonly = ai["data"]
    if readonly:
        raise TypeError("readonly arrays unsupported")
    tp = _ctype_ndarray(_typecodes[ai["typestr"]], ai["shape"])
    result = tp.from_address(addr)
    result.__keep = ai
    return result

it's evident that the original author thought of this (assigning .__keep to maintain a reference to the original object). However, it seems they need to keep a reference to the original object.

I've written a patch which does this:

-        result.__keep = ai
+        result.__keep = obj

kabanus · Accepted Answer · 2018-12-14 07:36:54Z

Final note

Leaving my tests for posterity, but tel has the answer.

Note

The below test results are on Debian. Testing on Ubuntu (WSL) is indeed much worse. On Ubuntu n=193 for any shape crashes (also if I replace the 3d n with 1), and any n above. Looks like (see bla.py below):

py bla.py n 1 allocates 3204 on A, 29323 ob B for al 0<n<193
For n>=193 a segmentation fault occurs on B, and 3208 are allocated on A. Apparently there is some hard memory limit somewhere in ubuntu.

The old tests on Debian

After some testing it looks to my like a memory issue, with a weird scaling of memory allocations with dimension.

The edit with only 2 dimensions does not crash for me, but 3 do - I will answer assuming this.

For me:

b = sharedctypes.RawArray(a._type_, a)

will not crash if:

a = np.ctypeslib.as_ctypes(np.zeros((224**3))) #Though generating b takes a while
a = np.ctypeslib.as_ctypes(np.zeros((100,100,100)))

So it seems less demand for memory removes the problem, but oddly the same amount of needed cells in a one dimensional array works fine - so something deeper in the memory seems to be going on.

Of course you are using pointers. Let's try some things (bla.py):

import tracemalloc
import numpy as np
from sys import argv
from multiprocessing import sharedctypes

n,shape = (int (x) for x in argv[1:])
if shape == 1: shape = n
if shape == 2: shape = (n**2,n)
if shape == 3: shape = (n,n,n)

tracemalloc.start()
a = np.ctypeslib.as_ctypes(np.zeros(shape))
x=tracemalloc.take_snapshot().statistics('lineno')
print(len(x),sum((a.size for a in x)))
b = sharedctypes.RawArray(a._type_, a)
x=tracemalloc.take_snapshot().statistics('lineno')
print(len(x),sum((a.size for a in x)))

Resulting in:

           n   shape    (a mallocs sum) (b mallocs sum)
>py bla.py 100 1     => 5 3478 76 30147
>py bla.py 100 2     => 5 5916 76 948313
>py bla.py 100 3     => 5 8200 76 43033
>py bla.py 150 1     => 5 3478 76 30195
>py bla.py 150 2     => 5 5916 76 2790461
>py bla.py 150 3     => 5 8200 76 45583
>py bla.py 159 1     => 5 3478 76 30195
>py bla.py 159 2     => 5 5916 76 2937854
>py bla.py 159 3     => 5 8200 76 46042
>py bla.py 160 1     => 5 3478 76 30195
>py bla.py 160 2     => 5 5916 72 2953989
>py bla.py 160 3     => 5 8200 Segmentation fault
>py bla.py 161 1     => 5 3478 76 30195
>py bla.py 161 2     => 5 5916 75 2971746
>py bla.py 161 3     => 5 8200 75 46116

>py bla.py 221 1     => 5 3478 76 30195
>py bla.py 221 2     => 5 5916 76 5759398
>py bla.py 221 3     => 5 8200 76 55348
>py bla.py 222 1     => 5 3478 76 30195
>py bla.py 222 2     => 5 5916 76 5782877
>py bla.py 222 3     => 5 8200 76 55399
>py bla.py 223 1     => 5 3478 76 30195
>py bla.py 223 2     => 5 5916 76 5806462
>py bla.py 223 3     => 5 8200 76 55450
>py bla.py 224 1     => 5 3478 76 30195
>py bla.py 224 2     => 5 5916 72 5829381
>py bla.py 224 3     => 5 8200 Segmentation fault
>py bla.py 225 1     => 5 3478 76 30195
>py bla.py 225 2     => 5 5916 76 5853950
>py bla.py 225 3     => 5 8200 76 55552

Weird stuff (n**2,n) has a giant amount of memort allocated for it in the shared type, while not n**3 or (n,n,n). But that is besides the point.

a mallocs are consistent and only slightly depend on dimension and not at all on n (for the numbers tested).
b mallocs besides being high on shape 2, increase with n slightly as well, but with shape they vary wildly.
The segmentation faults occurs in cycles! Memory allocation for shape (n,n,n) on my machine approaches some n dependent number before sefault - but for n+1 we are OK again. Seems to be ~46k around 160 and ~56k around 224.

No good explanation from me, but the dependence on n makes me think the allocations need to fit into some bit structure nicely, and sometimes this breaks.

I am guessing using 225 for your dimensions will work - as a workaround.

This has got to be an actual bug in numpy (not super likely since the code in np.ctypeslib is identical on mac and linux) or multiprocessing.
@kebanus Thanks for doing all this checking. I just upgraded my Ubuntu from 16.04 to 18.04 but the same seg faults persist (and are obviated with the workaround by tel). So, guess I'll create a Python bug report.

Collectives™ on Stack Overflow

Segmentation fault when creating multiprocessing array

3 Answers 3

3 Comments

Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related