C++ Python module crashes in Blender but not in Python console

Question

Problem

I am trying to call my C++ code from Python 3.7 with Blender 2.82a (also happens in 2.83). The code should optimize a camera path. It can be used without Blender, however, I use Blender to setup a scene with camera path and query depth values in a scene.

I tried calling the optimization function in C++ and in the Python console. Both worked without any problems. The problem is, when I call it in Blender, Blender crashes.

This is the crash report:

# Blender 2.83.0, Commit date: 2020-06-03 14:38, Hash 211b6c29f771

# backtrace
./blender(BLI_system_backtrace+0x1d) [0x6989e9d]
./blender() [0xc1548f]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7fa5fb3dc3c0]
/lib/x86_64-linux-gnu/libpthread.so.0(raise+0xcb) [0x7fa5fb3dc24b]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7fa5fb3dc3c0]
./blender(_ZN5Eigen8IOFormatD1Ev+0xa3) [0x179bc43]
/home/name/Programs/blender-2.83.0-linux64/2.83/python/lib/python3.7/optFlowCam.cpython-37m-x86_64-linux-gnu.so(_Z2_zRK6CameraRKN5Eigen6MatrixIdLi9ELi1ELi0ELi9ELi1EEEiiRKSt8functionIFdRK3RayEE+0x2e2) [0x7fa5d1538e72]
/home/name/Programs/blender-2.83.0-linux64/2.83/python/lib/python3.7/optFlowCam.cpython-37m-x86_64-linux-gnu.so(_ZN11OpticalFlow13GradPathErrorERKSt6vectorI6CameraSaIS1_EEiiRKSt8functionIFdRK3RayEEd+0x5a7) [0x7fa5d1539c77]
/home/name/Programs/blender-2.83.0-linux64/2.83/python/lib/python3.7/optFlowCam.cpython-37m-x86_64-linux-gnu.so(_Z16_gradientDescentRKSt6vectorI6CameraSaIS0_EEiiRKSt8functionIFdRK3RayEEd+0x54b) [0x7fa5d153b5fb]
/home/name/Programs/blender-2.83.0-linux64/2.83/python/lib/python3.7/optFlowCam.cpython-37m-x86_64-linux-gnu.so(_ZN11OpticalFlow12OptimizePathERKSt6vectorI6CameraSaIS1_EEiiRKSt8functionIFdRK3RayEEdNS_18OptimizationMethodE+0x22) [0x7fa5d153bcb2]
/home/name/Programs/blender-2.83.0-linux64/2.83/python/lib/python3.7/optFlowCam.cpython-37m-x86_64-linux-gnu.so(+0x3d910) [0x7fa5d1533910]
/home/name/Programs/blender-2.83.0-linux64/2.83/python/lib/python3.7/optFlowCam.cpython-37m-x86_64-linux-gnu.so(+0x317ed) [0x7fa5d15277ed]
./blender(_PyMethodDef_RawFastCallKeywords+0x2f3) [0x570f373]
./blender(_PyCFunction_FastCallKeywords+0x25) [0x570f3f5]
./blender(_PyEval_EvalFrameDefault+0x7468) [0xc0fb48]
./blender(_PyEval_EvalCodeWithName+0xadc) [0x57c0d8c]
./blender(PyEval_EvalCodeEx+0x3e) [0x57c0ebe]
./blender(PyEval_EvalCode+0x1b) [0x57c0eeb]
./blender() [0x11f35ac]
./blender() [0x1600cde]
./blender() [0xec6a93]
./blender() [0xec6d07]
./blender(WM_operator_name_call_ptr+0x1a) [0xec720a]
./blender() [0x14f2082]
./blender() [0x15020d5]
./blender() [0xeca877]
./blender() [0xecaecc]
./blender(wm_event_do_handlers+0x310) [0xecb5e0]
./blender(WM_main+0x20) [0xec2230]
./blender(main+0x321) [0xb4bfd1]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fa5facb50b3]
./blender() [0xc11c0c]

I am using Eigen for linear algebra calculations and pybind11 to compile it into a python module. The Eigen types are all fixed size as I don't need them to be dynamic (possible reason for the issue). I compile with gcc (Ubuntu 9.3.0-10ubuntu2) 9.3.0 on Ubuntu 20.04. I currently use c++11 standard but it's not a requirement.

Current findings

With faulthandler.enabled() it gives me

Fatal Python error: Segmentation fault

Current thread 0x00007fa5fab18040 (most recent call first):
  File "/Text", line 16 in <module>

I already found that it crashes on the same line in the program, which is when the result of a matrix-vector multiplication is supposed to be returned and inserted into a std::vector. I printed the vector and the matrix prior, to make sure they do not contain garbage and that worked fine.

I also tried to store it in an intermediate variable and print it and then it crashes on printing. The multiplication itself seems not to cause the segfault.

I figured, I try to call the function, where it occurs, directly from Blender, but then it works and returns a result without segfault.

I suspect it to be some kind of memory alignment issue and tried everything suggested here and on the Eigen documentary. Namely, I use Eigen::aligned_allocator in every std::vector, only pass the Eigen objects as const & and have EIGEN_MAKE_ALIGNED_OPERATOR_NEW in the camera and ray class which have Eigen type members.

Using #define EIGEN_DONT_VECTORIZE and #define EIGEN_DISABLE_UNALIGNED_ARRAY_ASSERT only gave me partial success. It doesn't crash at the same line as before anymore. Weirdly enough, if I also add a cout before the return, the function finishes and returns.

Parts where crashes occur:

The project is not public and the C++ code is quite lengthy, so I only include parts of it. Let me know if you need more. The rest looks very similar, so if there is something conceptually wrong, it is probably wrong in here too. It is not a minimal example (and it contains some debug prints), as I don't know why it happens and the error is not always on the same part.

// in header

// this helped somehow
#define EIGEN_DONT_VECTORIZE
#define EIGEN_DISABLE_UNALIGNED_ARRAY_ASSERT
// ***********************************

#include <iostream>
#include <numeric>
#include <array>
#include <vector>
#include <Eigen/Dense>
#include <Eigen/StdVector>
#include "matrix_types.h"
#include "camera.h"

// *******************************************************************************
// Vector9d is a typedef

// in cpp
Vector9d _z(const Camera& cam, const Vector9d& derivX, int x_dir, int y_dir, const std::function<double(const Ray&)>& depthTest){

    Eigen::IOFormat HeavyFmt(Eigen::FullPrecision, 0, ", ", ",\n", "[", "]", "[", "]");

    Matrix9d M0 = OpticalFlow::M(cam, x_dir, y_dir, depthTest);
    std::cout << "M_\n" << M0.format(HeavyFmt) << "\n" <<std::endl;

    Vector9d z = M0 * derivX;

    return z;
}

std::vector<Vector9d, Eigen::aligned_allocator<Vector9d>> OpticalFlow::GradPathError(const std::vector<Camera>& pathPositions, int x_dir, int y_dir, const std::function<double(const Ray&)>& depthTest, double h){

    int n = pathPositions.size()-1;
    Eigen::IOFormat HeavyFmt(Eigen::FullPrecision);
    Eigen::IOFormat HeavyMtxFmt(Eigen::FullPrecision, 0, ", ", ",\n", "[", "]", "[", "]");

    std::vector<Vector9d, Eigen::aligned_allocator<Vector9d>> gradPE;
    gradPE.reserve(n+1);

    // save values that will be used more often in calculations
    std::vector<Vector9d, Eigen::aligned_allocator<Vector9d>> derivXs;
    std::vector<Vector9d, Eigen::aligned_allocator<Vector9d>> zs;
    std::vector<std::array<Matrix9d, 9>> gradMs;
    derivXs.reserve(n+1);
    zs.reserve(n+1);
    gradMs.reserve(n+1);

    for(int i = 0; i<n+1; ++i){

        derivXs.push_back(_derivCamPath(pathPositions, i));

        Camera cam = pathPositions[i];
        Vector9d derivX = _derivCamPath(pathPositions, i);

        zs.push_back(_z(cam, derivX, x_dir, y_dir, depthTest)); // <--- crashed here, if vectorization not turned off
        gradMs.push_back(GradM(cam, x_dir, y_dir, depthTest, h));
    }

    for(int i = 0; i<n+1; ++i){
        Vector9d derivZ = _derivZ(zs, i);
        std::cout << "Zt_" << i << "\n" << derivZ.format(HeavyFmt) << "\n" << std::endl;
        gradPE.push_back(1.0/(n+1) * _w(derivZ, derivXs[i], gradMs[i]));
    }

    // if this is included and vectorization turned off, it doesn't crash
    // std::cout << "end" << std::endl;

    return gradPE; // <-- crash here if vectorization is off
}

I hope someone can help me find the cause or what else I can try to track it down further. I am not very experienced with C++, so the code might have obvious issues.

look for "align" related flags in gcc.gnu.org/onlinedocs/gcc/Option-Summary.html i had similar issues with a module using SSE it solved out of the box when i switched to a different compiler, but before that i managed to fix faults forcing alignment of structs with compiler flags. I do not know how to pass these to pybind. — Szabolcs Dombi
– Szabolcs Dombi, Commented Jun 20, 2020 at 9:29
Thanks, I will take a look. I think I found a compiler option related to Eigen and alignment issues somewhere but will have to look for it again. — LivelyLiz
– LivelyLiz, Commented Jun 20, 2020 at 12:12

LivelyLiz · Accepted Answer · 2020-06-20 14:44:56Z

1

I think I found the cause.

This line ./blender(_ZN5Eigen8IOFormatD1Ev+0xd3) [0x2041673] actually named the culprit in a not so readable format. I used gdb to debug the python call from Blender and in the backtrace, the same line is #0 0x0000000002041673 in Eigen::IOFormat::~IOFormat() ()

The problem was I used Eigen::IOFormat in my program to debug print the matrices and vectors for easier copying into another program where I just wanted to check if the values are correct. You can see it in the excerpt code I posted in the question. I only used it in those two functions and the segfault only occurred in those two functions. There might be something else wrong but for now, it seems to work.

answered Jun 20, 2020 at 14:44

LivelyLiz

435 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

C++ Python module crashes in Blender but not in Python console

Problem

Current findings

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

Problem

Current findings

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related