Exposing a C string without copying to python 3.x code

Question

I have a library written with cython that wraps a C library, and i'm exposing a few c strings into python code. Those strings are large, and static (can't deallocate them) so just making a python string from them (that makes a copy) is not an option - i get OOM errors.

I have the code working for python 2.x currently using the old buffer API, which looks more or less like:

def get_foo():
     return PyBuffer_FromMemory(c_foo_ptr,c_foo_len)

That just works (tm) for python 2.x, but the old buffer API is gone in 3.x and i can't figure out how do i get that with a the new one.

I see that there's PyMemoryView_FromBuffer and PyBuffer_FillInfo which combined will supposedly do the same thing, but PyBuffer_FillInfo wants an object which doesn't exist for me (it's just a module-level function), making a dummy object and passing it just gives me a segfault, so i guess that object should suport the buffer somehow... but where it's documented?

further from experimenting with memoryviews they don't look or act as strings (or bytes) at all, so i'll have to either rewrite all my python code or somehow recreate that functionality.

Am i missing something? is there a simple way to replace PyBuffer_FromMemory in py3k?

Note: I'm using cython, but this is raw c-api stuff, so you can answer without involving cython into it.

dequis · Accepted Answer · 2014-05-24 08:13:29Z

1

According to this thread the second argument to PyBuffer_FillInfo is optional. Could you pass NULL in its place? If not you could just create a PyBuffer instance yourself and fill the appropriate fields.

edited May 24, 2014 at 8:13

dequis

2,19020 silver badges26 bronze badges

answered Feb 16, 2011 at 12:27

macedoine

3471 silver badge3 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

bdew Over a year ago

Sorry somewhat forgot about this question... After discussing it on cython-users i went in a different direction - Wrapping the data in my own object that uses the new buffer protocol and emulating some of the old-style buffer objects functionality with it. I will mark this answers as accept regardless because it was the most useful one :)

totaam Over a year ago

@bdew can you share your end result? I am trying to do the same thing and I find the documentation unhelpful, the new buffer API is more heavyweight and all I've gained so far is memory leaks and crashes.. The mailing list link is dead btw.

Ignacio Vazquez-Abrams · Accepted Answer · 2011-02-16 12:06:46Z

0

It sounds to me as though you should create your own custom type and implement the methods in tp_as_sequence (and possibly tp_as_buffer) as appropriate.

answered Feb 16, 2011 at 12:06

Ignacio Vazquez-Abrams

804k160 gold badges1.4k silver badges1.4k bronze badges

1 Comment

bdew Over a year ago

Your links are to old-style buffer API docs (for python 2.7), the new API seems to use completely different slots. but thanks for the pointer, those are probably the methods i need to implement if i wrap this up in an object.

Collectives™ on Stack Overflow

Exposing a C string without copying to python 3.x code

2 Answers 2

2 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related