python c-api: create bytes using existing buffer without copying

Question

It seems to me the buffer protocol is more for exposing Python buffer to C.

I couldn't find a way to create a bytes object using existing buffer without copying in C.

Basically what I want is to implement something similar to PyBytes_FromStringAndSize() but without copying, and with a callback to free the buffer when the object is released. I don't know how big the buffer is before I receive the buffer returned from a C API. So creating bytes object in Python first and later fill it in is not an option.

I also looked into memoryview, PyMemoryView_FromMemory() doesn't copy but there is no way to pass a callback to free my buffer. And I'm not suse Python lib (e.g. Psycopg) can use memoryview object or not.

Do I have to create my own object to achieve these 2 requirements? Any other shortcut?

If I have to, how can I make sure this object works same as bytes so I can pass it to Python lib safely?

Thanks.

user3761759 · Accepted Answer · 2014-09-24 18:54:10Z

1

The only way to do this is to create a new object with PyBufferProcs* PyTypeObject.tp_as_buffer. I checked cpython source code thoroughly, as of 3.4.1, there is no out-of-box (so to speak) solution.

answered Sep 24, 2014 at 18:54

user3761759

9451 gold badge6 silver badges8 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Alan Over a year ago

Could you please explain how one would do this, preferably with a code sample?

Collectives™ on Stack Overflow

python c-api: create bytes using existing buffer without copying

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related