In short, Boost.Python maintains Python argument passing semantics with its TypeWrappers. Thus, when passing a list in Python to an exposed C++ function, a reference can be created and maintained by accepting the Python list as a boost::python::list object.
The detailed answer actually has a bit more depth to it. Before delving into it, let me expand upon some semantics to avoid confusion. With Python's garbage collection and pass-by-object semantics, the general rule of thumb is to treat the Boost.Python TypeWrappers as smart pointers.
- If the function accepts the list as a
boost::python::list object, then C++ now has a reference to the Python object. The Python list's lifetime will be extended to be at least as long as the booot::python::list object.
- If the Python list is converted to a different type, such as
std::vector, then C++ has constructed a copy to the Python list. This copy has no association with the original list.
Here is a simple example of a C++ module that can be passed a Python list, maintain a handle to it, and print its contents:
#include <iostream> // std::cout
#include <utility> // std::make_pair
#include <boost/foreach.hpp>
#include <boost/python.hpp>
#include <boost/python/stl_iterator.hpp>
boost::python::list list;
/// @brief Store handle to the list.
///
/// @param pylist Python list for which a handle will be maintained.
void set(const boost::python::list& pylist)
{
// As the boost::python::list object is smart-pointer like, this
// creates a reference to the python list, rather than creating a
// copy of the python list.
list = pylist;
}
// Iterate over the current list, printing all ints.
void display()
{
std::cout << "in display" << std::endl;
typedef boost::python::stl_input_iterator<int> iterator_type;
BOOST_FOREACH(const iterator_type::value_type& data,
std::make_pair(iterator_type(list), // begin
iterator_type())) // end
{
std::cout << data << std::endl;
}
}
BOOST_PYTHON_MODULE(example) {
namespace python = boost::python;
python::def("set", &set);
python::def("display", &display);
}
And its usage:
>>> import example
>>>
>>> x = range(2)
>>> x
[0, 1]
>>> example.set(x)
>>> example.display()
in display
0
1
>>> x[:] = range(7, 10)
>>> example.display()
in display
7
8
9
One complexity introduced in the question is the desire to read the Python list in C++ at any time. In its most complicated case, any time can occur at any point in time, from within a C++ thread.
Lets start with the basics: Python's Global Interpreter Lock (GIL). In short, the GIL is a mutex around the interpreter. If a thread is doing anything that affects reference counting of python managed object, then it needs to have acquired the GIL. Sometimes the reference counting is not very transparent, consider:
typedef boost::python::stl_input_iterator<int> iterator_type;
iterator_type iterator(list);
Although boost::python::stl_input_iterator accepts list as a constant reference, it creates a reference to the Python list from within the constructor.
In the previous example, as there were no C++ threads, all actions occurred while the GIL had been acquired. However, if display() could be invoked at any time from within C++, then some setup needs to occur.
First, the module needs to have Python initialize the GIL for threading.
BOOST_PYTHON_MODULE(example) {
PyEval_InitThreads(); // Initialize GIL to support non-python threads.
...
}
For convenience, lets create a simple class to help manage the GIL:
/// @brief RAII class used to lock and unlock the GIL.
class gil_lock
{
public:
gil_lock() { state_ = PyGILState_Ensure(); }
~gil_lock() { PyGILState_Release(state_); }
private:
PyGILState_STATE state_;
};
To show interactions with a C++ thread, lets add functionality to the module that will allow the application to schedule a delay for when the list's contents to be displayed.
/// @brief Entry point for delayed display thread.
///
/// @param Delay in seconds.
void display_in_main(unsigned int seconds)
{
boost::this_thread::sleep_for(boost::chrono::seconds(seconds));
gil_lock lock; // Acquire GIL.
display(); // Can safely modify python objects.
// GIL released when lock goes out of scope.
}
/// @brief Schedule the list to be displayed.
///
/// @param Delay in seconds.
void display_in(unsigned int seconds)
{
// Start detached thread.
boost::thread(&display_in_main, seconds).detach();
}
Here is the complete example:
#include <iostream> // std::cout
#include <utility> // std::make_pair
#include <boost/foreach.hpp>
#include <boost/python.hpp>
#include <boost/python/stl_iterator.hpp>
#include <boost/thread.hpp>
boost::python::list list;
/// @brief Store handle to the list.
///
/// @param pylist Python list for which a handle will be maintained.
void set(const boost::python::list& pylist)
{
list = pylist;
}
// Iterate over the current list, printing all ints.
void display()
{
std::cout << "in display" << std::endl;
typedef boost::python::stl_input_iterator<int> iterator_type;
BOOST_FOREACH(const iterator_type::value_type& data,
std::make_pair(iterator_type(list), // begin
iterator_type())) // end
{
std::cout << data << std::endl;
}
}
/// @brief RAII class used to lock and unlock the GIL.
class gil_lock
{
public:
gil_lock() { state_ = PyGILState_Ensure(); }
~gil_lock() { PyGILState_Release(state_); }
private:
PyGILState_STATE state_;
};
/// @brief Entry point for delayed display thread.
///
/// @param Delay in seconds.
void display_in_main(unsigned int seconds)
{
boost::this_thread::sleep_for(boost::chrono::seconds(seconds));
gil_lock lock; // Acquire GIL.
display(); // Can safely modify python objects.
// GIL released when lock goes out of scope.
}
/// @brief Schedule the list to be displayed.
///
/// @param Delay in seconds.
void display_in(unsigned int seconds)
{
// Start detached thread.
boost::thread(&display_in_main, seconds).detach();
}
BOOST_PYTHON_MODULE(example) {
PyEval_InitThreads(); // Initialize GIL to support non-python threads.
namespace python = boost::python;
python::def("set", &set);
python::def("display", &display);
python::def("display_in", &display_in);
}
And its usage:
>>> import example
>>> from time import sleep
>>>
>>> x = range(2)
>>> example.set(x)
>>> example.display()
in display
0
1
>>> example.display_in(3)
>>> x[:] = range(7, 10)
>>> print "sleeping"
sleeping
>>> sleep(6)
in display
7
8
9
While the Python console blocked for 6 seconds in the sleep(6) call, the C++ thread acquired the GIL, displayed the contents of list x, and released the GIL.
