Cython numpy array with Openmp (No GIL)

Question

I am not sure if this has been addressed previously, i tried searching but didnt find exactly what i was looking for. I would like to get the following code working (name of file being some_CD.pyx)

import numpy as np
cimport numpy as np
cimport cython
from cython.parallel import *

ctypedef np.float64_t DTYPE

cdef DTYPE evaluate_objective(np.ndarray[DTYPE, ndim=2] A,np.ndarray[DTYPE, ndim=1] b,np.ndarray[DTYPE, ndim=1] x):
    return x.dot(A.dot(x) - b.dot(x))

cpdef DTYPE coordinate_descent(np.ndarray[DTYPE, ndim=2] A,np.ndarray[DTYPE, ndim=1] b,int nDim, int nIter ):
    cdef int i, iter
    cdef np.ndarray[DTYPE,
                    ndim=1] x = \
                    np.zeros(nDim, )
    cdef DTYPE temp
    for iter in prange(nIter,nogil=True):
        i = (iter%nDim)
        temp = (b[i]-( A[:,i].dot(x)*x[i] ) + (A[i,i]*x[i]*x[i]))/A[i,i]
        x[i] = np.max([0,np.min([temp,1])])
    return evaluate_objective(A,b,x)

My setup.py looks as following

from distutils.core import setup, Extension
from Cython.Build import cythonize
import numpy

ext_modules = [
    Extension(
        "some_CD",
        ["some_CD.pyx"],
        extra_compile_args=['-fopenmp'],
        extra_link_args=['-fopenmp'],
    )
]

setup(
    name='some_parallel',
    ext_modules=cythonize(ext_modules),
    include_dirs=[numpy.get_include()]
)

There are many things i am not sure in this code. First of all whether am i using numpy arrays the right way? Is using float64, np.int typed variables allowed within prange?

Unfortunately, you can't use numpy arrays in a nogil context because they are inherently python objects. This question bugged me for awhile too, and while I haven't written something that solves it myself I was browsing the sklearn sourcode and it seems they memcpy the data their ndarrays before going into a nogil context. That should put you into a pure C context where you can use nogil with prange. — Erotemic
– Erotemic, Commented Nov 4, 2016 at 22:43
Isn't it time consuming to copy the whole array especially in large dimensions? Is there any efficient way? Could you modify the code maybe write as an answer if possible? It will take quite a time to get to know the right syntax and all! — user52705
– user52705, Commented Nov 4, 2016 at 23:08

user6758673 · Accepted Answer · 2016-11-04 23:49:12Z

1

This loop can't be parallelized because it's an iterative algorithm. In other words, later iterations depend on the result of the earlier iterations.

Ignoring this problem, here's how you would change the code..

Without the GIL you can only use basic indexing (no slicing) and a very small number of Python functions special cased by Cython (mainly those listed here). So you just have to make the body of the prange more plain and explicit.

for iter in prange(nIter, nogil=True):
    i = iter % nDim

    Ai_dot_x = 0
    for j in range(x.shape[0]):
        Ai_dot_x += A[j,i] * x[j]

    temp = (b[i] - Ai_dot_x*x[i] + A[i,i]*x[i]*x[i]) / A[i,i]
    x[i] = max(0, min(temp, 1))

Ai_dot_x is an additional temporary variable that you have to cdef.

answered Nov 4, 2016 at 23:49

user6758673

5892 silver badges4 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Zulan Over a year ago

Not sure about the Cython details, but wouldn't splitting prange(nIter): into range(nIter/nDim): and prange(nDim, nogil=True): solve the data dependency? I do fail to see the point why those were merged in the first place.

user52705 Over a year ago

@user6758673 I added the traceback in the question, after making the changes. Please check above.

user52705 Over a year ago

@Zulan What you mentioned might be a different algorithm maybe its called parallel coordinate descent (partially asynchronous) . What i am trying to do is a truly asynchronously update where each coordinate might have been updated more than once at times.

Zulan Over a year ago

@user52705 Updating a value truly asynchronously more than once at times sounds like a race condition to me. But actually, I overlooked the dot product, which introduces the dependency to the entire array for each and every iteration.

user52705 Over a year ago

@Zulan Yeah you are right regarding the race condition, but that's what i want actually i.e running the updates in a truly asynchronous fashion.

Collectives™ on Stack Overflow

Cython numpy array with Openmp (No GIL)

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related