Calling C++ function which accepts and returns std::string from Python

Question

I'm trying to call C++ function using ctypes. The function declaration looks like

std::string some_function( const std::string &param )

As Python can not interact with C++ directly, I have created a C-wrapper for this function as follows

const char *my_function( const char *param )
{
  std::string result = some_function( std::string( param ) );
  char *ret = new char[result.length() + 1];
  strcpy(ret, result.c_str());
  return ret;
}

And Python wrapper

def my_function(param):
    func = lib.my_function
    func.argtypes = [ctypes.c_char_p]
    func.restype = ctypes.c_char_p
    result = func(ctypes.c_char_p(param))
    return result

But when I try to call Python wrapper with some bytes passed as param, for example

my_function(b'GP\x00\x01\xe6\x10\x00\x00\x01\x01\x00\x00\x00\x1ex\xcb\xa16l\xf1\xbfp\xe6\xaa\xc89\x81\xdd?')

It fails with the following error

terminate called after throwing an instance of 'std::length_error'
  what():  basic_string::_M_create

I'm not very good in C and C++. Can you help me to figure out how to create correct wrapper? Thanks.

the parameter you pass has no terminating \0 (unless python adds it, not sure about that) — 463035818_is_not_an_ai
– 463035818_is_not_an_ai, Commented Jun 2, 2022 at 14:57
@463035818_is_not_a_number Nah, it does: in fact, there’s a null terminator at index 2. (Of course this surely isn’t what OP wants.) — Konrad Rudolph
– Konrad Rudolph, Commented Jun 2, 2022 at 14:58
You should be able to print out the contents of the parameter in my_function before trying to create a string from it to see what it looks like. Either in a debugger or just std::cout/printf. I'm not really familiar with ctypes but nothing you're doing really stands out to me as being wrong. — Kevin
– Kevin, Commented Jun 2, 2022 at 15:30
"call C++ function using ctypes" Well ctypes is not called c++types, and for a reason. Perhaps you want to use something more suitable for C++, such as pybind11. — n. m. could be an AI
– n. m. could be an AI, Commented Jun 2, 2022 at 17:32

Mark Tolonen · Accepted Answer · 2022-06-02 17:17:08Z

Since your data contains embedded nulls, c_char_p won't work as it assumes the returned char* is null-terminated and converts the data up to the first null found to a bytes object. std::string as used also makes that assumption when pass a char* only, so it needs the data length as well.

To manage a data buffer with null content, the size of the input data must be passed in and the size of the output data must be returned. You'll also have to manage freeing the data allocated in the C++ code.

The below code demonstrates everything:

test.cpp - compiled with cl /EHsc /W4 /LD test.cpp with Microsoft Visual Studio.

#include <string>
#include <string.h>

// Windows DLLs need functions to be explicitly exported
#ifdef _WIN32
#   define API __declspec(dllexport)
#else
#   define API
#endif

std::string some_function(const std::string& param) {
    return param + std::string("some\x00thing", 10);
}

// pLength is used as an input/output parameter.
// Pass in input length and get output length back.
extern "C" API
const char* my_function(const char* param, size_t* pLength) {
    auto result = some_function(std::string(param, *pLength));
    auto reslen = result.length();
    auto res = new char[reslen];
    // Use memcpy to handle embedded nulls
    memcpy_s(res, reslen, result.data(), reslen);
    *pLength = reslen;
    return res;
}

extern "C" API
void my_free(const char* param) {
    delete [] param;
}

test.py

import ctypes as ct

dll = ct.CDLL('./test')

# Note: A restype of ct.c_char_p is converted to a bytes object
#       and access to the returned C pointer is lost.  Use
#       POINTER(c_char) instead to retain access and allow it
#       to be freed later.
dll.my_function.argtypes = ct.c_char_p, ct.POINTER(ct.c_size_t)
dll.my_function.restype = ct.POINTER(ct.c_char)
dll.my_free.argtypes = ct.c_char_p,
dll.my_free.restype = None

def my_function(param):
    # wrap size in a ctypes object so it can be passed by reference
    size = ct.c_size_t(len(param))
    # save the returned pointer
    p = dll.my_function(param, ct.byref(size))
    # slice pointer to convert to an explicitly-sized bytes object
    result = p[:size.value]
    # now the pointer can be freed...
    dll.my_free(p)
    # and the bytes object returned
    return result

result = my_function(b'data\x00more')
print(result)

Output:

b'data\x00moresome\x00thing'

Thanks a lot @Mark! Your clear and detailed explanation with sample are invaluable. This is exactly what I was looking for. Thanks again

Collectives™ on Stack Overflow

Calling C++ function which accepts and returns std::string from Python

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related