Append C++ created object to python list and make it managed by python

Question

Well, I've been checking this for a while, couldn't find an answer to it.

I wanted to append an object which is exposed to python, say Foo:

struct Foo {
  Foo(){ std::cout << "Creating a Foo object" << std::endl;}
  virtual ~Foo(){ std::cout << "Destroying a Foo object" << std::endl;}
};

I work with the Foo inherited objects, and at some point I want to append them to a python list. For this, I created a FooWrapper, which inherits from Foo and use the copy constructor to

struct FooWrapper : public Foo {
  FooWrapper(const Foo& foo):Foo(foo){ std::cout << "Creating a copy from foo using FooWrapper" << std::endl;}  
  virtual ~FooWrapper(){ std::cout << "Destroying a FooWrapper object" << std::endl;}
};

This is exposed to python:

BOOST_PYTHON_MODULE(foo_module)
{
    using namespace py = boost::python;
    py::class_<FooWrapper>("FooWrapper", py::no_init)…
}

I have a method which appends the final Foo objects as FooWrapper to a python list, say:

void appendFoosToList(py::list &list)
{
  for ( const Foo* foo : m_foos )
  {
    list.append( FooWrapper( *foo )  );
  }
}

How could I make so that instead of creating a temporary object and then copying to the list, that I append to the list this object, without having to copy the temporary?

I've read many documentations (boost_faq, boost_python_wiki), many times I got this runtime error:

TypeError: No to_python (by-value) converter found for C++ type:

BPL was unable to get C++ value from Python object.

For example, when calling extract(.attr("len")()) to get object length you omitted "()".

And didn't manage to find a solution.

I couldn't find a clear documentation about this, so I come here as the final resort.

I have good experience using boost::shared_ptr, that makes it possible to transparently move objects between c++ and python without worrying about ownership and lifetime (just careful with circular pointer structures). It might not work everywhere, but is useful for some cases. — eudoxos
– eudoxos, Commented Sep 14, 2015 at 15:10

Tanner Sansbury · Accepted Answer · 2016-06-03 19:07:10Z

In short, allocate the wrapper on the free store and use the manage_new_object result convert to transfer ownership to a Python object. This will cause Boost.Python to copy the pointer when constructing the Python object, rather than copying the pointee.

A C++ object is embedded into the Python object. This is the HeldType provided when exposing the class via class_, which defaults to the C++ type being exposed. Often, when exposing a function, one can augment the C++ type that gets returned and embedded into the Python object with CallPolicy instances. In particular, using an instance of the return_value_policy CallPolicy with a manage_new_object ResultConverterGenerator allows for the embedded type to be a pointer, and the Python object will manage ownership.

The following function can be used to transfer ownership of an object to Python without making a copy of the pointee:

/// @brief Transfer ownership to a Python object.  If the transfer fails,
///        then object will be destroyed and an exception is thrown.
template <typename T>
boost::python::object transfer_to_python(T* t)
{
  // Transfer ownership to a smart pointer, allowing for proper cleanup
  // incase Boost.Python throws.
  std::unique_ptr<T> ptr(t);

  // Use the manage_new_object generator to transfer ownership to Python.
  namespace python = boost::python;
  typename python::manage_new_object::apply<T*>::type converter;

  // Transfer ownership to the Python handler and release ownership
  // from C++.
  python::handle<> handle(converter(*ptr));
  ptr.release();

  return python::object(handle);
}

Example usage:

void appendFoosToList(boost::python::list& list)
{
  for (const Foo* foo : m_foos)
  {
    list.append(transfer_to_python(new FooWrapper(*foo)));
  }
}

Here is a complete example demonstrating this approach:

#include <iostream>
#include <boost/python.hpp>

// Mocks...
class spam
{
public:
  spam() { std::cout << "spam(): " << this << std::endl; }
  spam(const spam&)
  {
    std::cout << "spam(const spam&): " << this << std::endl;
  }
  ~spam() { std::cout << "~spam(): " << this << std::endl; }
};

/// @brief Transfer ownership to a Python object.  If the transfer fails,
///        then object will be destroyed and an exception is thrown.
template <typename T>
boost::python::object transfer_to_python(T* t)
{
  // Transfer ownership to a smart pointer, allowing for proper cleanup
  // incase Boost.Python throws.
  std::unique_ptr<T> ptr(t);

  // Use the manage_new_object generator to transfer ownership to Python.
  namespace python = boost::python;
  typename python::manage_new_object::apply<T*>::type converter;

  // Transfer ownership to the Python handler and release ownership
  // from C++.
  python::handle<> handle(converter(*ptr));
  ptr.release();

  return python::object(handle);
}

void append_to_list(boost::python::list& list)
{
  list.append(transfer_to_python(new spam()));
}

BOOST_PYTHON_MODULE(example)
{
  namespace python = boost::python;
  python::class_<spam>("Spam", python::no_init);
  python::def("append_to_list", &append_to_list);
}

Interactive usage:

>>> import example
>>> spams = []
>>> example.append_to_list(spams)
spam(): 0x25cbd90
>>> assert(type(spams[0]) is example.Spam)
>>> del spams
~spam(): 0x25cbd90

Hi Tanner, thanks for your reply. I'll check this in the weekend. Everything working as expected I'll open a bounty for giving you more points for a so detailed answer. Thank you a lot!

Collectives™ on Stack Overflow

Append C++ created object to python list and make it managed by python

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related