C code embedded python callback function

Question

C code embedded python callback function，and put data to python queue through callback, when i get data from queue, it's very slow.

Example:

c code like this

static int wrap_func(const int a, const unsigned char *b)
{
  long ret;
  PyObject *arglist;
  PyObject * result = NULL;

  arglist = Py_BuildValue("(s#)", b, a);
  result = PyEval_CallObject(my_callback, arglist);

  /* evaluate result or handle exception */
  ret = PyInt_AsLong(result);
  if (result == NULL)
    return -1;
  Py_DECREF(result);

  return ret;
}

void produce_data()
{
    while(1){
        //produce data to buffer, len is buffer length
        //call callback func 
        wrap_func(buffer, len);
    }
}

compile this c code to so like mywrap.so, and import this so in python python code like this:

import multiprocessing
import mywarp   # mywrap.so

class WorkerThread_a(threading.Thread):
    def __init__(self, workQueue):
        threading.Thread.__init__(self)
        self.workQueue = workQueue
        self.setDaemon(True)
    def run(self):
        while 1:
            try:
                recvdata = self.workQueue.get(block=False)
            except Queue.Empty:
                continue
            #do sth use recvdata

workQueue = multiprocessing.Queue()

def callback_func(a):
    if a:
        workQueue.put(a)
    return 0

def main():
    tmp = WorkerThread_a(workQueue)
    tmp.start()
    mywarp.set_callback(callback_func)
    mywarp.decode_audio()

main()

In python thread, i get data from queue, but i get data very slowly, but in c so, produce data and put to queue through python callback func quickly.

how can i get data from queue quickly like in pure python code.

DavidW · Accepted Answer · 2015-11-27 15:55:13Z

1

I think what's happening is that your C code is never releasing the global interpreter lock (GIL), so your Python code never has a chance to run. When you're running multiple threads in Python code, they automatically swap the GIL between them and so share time equally, but this doesn't happen without your intervention in C code.

It should probably work a lot better if you acquire and release the GIL once per loop in your C-code (even though you don't do anything that doesn't need it). All I've really done is added the macros Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS at the start of the function.

static int wrap_func(const int a, const unsigned char *b)
{
  Py_BEGIN_ALLOW_THREADS /* ADDED! */
  /* This should give the chance for the main Python thread
  to run, even though no code goes here */
  Py_END_ALLOW_THREADS /* ADDED */

  long ret;
  PyObject *arglist;
  PyObject * result = NULL;

  arglist = Py_BuildValue("(s#)", b, a);
  result = PyEval_CallObject(my_callback, arglist);

  /* evaluate result or handle exception */
  ret = PyInt_AsLong(result);

  /* This is chang */
  if (result == NULL)
    return -1;
  Py_DECREF(result);


  return ret;
}

(I should say - this is an untested guess that I'm 90% sure is right, but I've been wrong before!)

edited Nov 27, 2015 at 15:55

answered Nov 27, 2015 at 10:09

DavidW

31.2k7 gold badges64 silver badges99 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

DavidW Over a year ago

(I've edited this - my original code had the two macros the wrong way round so would not have worked - sorry!)

ICYMYM Over a year ago

your suggestion is wonderful, it work! Thanks very much. i run code as your suggestion , it work, but when i put the Py_BEGIN_ALLOW_THREADS at the start of the func, and put the Py_END_ALLOW_THREADS before return. i run it , it's segment fault，why is this?

DavidW Over a year ago

That's because I made a mistake! (fixed now). You aren't allowed any Python api calls between Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS. So that means any of the functions starting with Py won't work. Unfortunately none of this code can actually run in parallel - I've just added a place where it can swap between your C module and your Python code.

Collectives™ on Stack Overflow

C code embedded python callback function

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related