Multi-threading with QT + OpenCV

Question

I'm trying to code a simple program that reads three video files (actually, 3 cameras that are in the same room), using 3 different threads. The code I'm using is the following:

mainwindow.cpp

void MainWindow::init()
{
    numCams = 3;

    // Resize the video for displaying to the size of the widget
    int WidgetHeight = ui->CVWidget1->height();
    int WidgetWidth  = ui->CVWidget1->width();

    for (int i = 0; i < numCams; i++){
        // Create threads
        threads[i] = new QThread;

        // Create workers
        string Path = "/Users/alex/Desktop/PruebasHilos/Videos/" + to_string(i+1) + ".m2v";
        workers[i] = new Worker(QString::fromStdString(Path), i, WidgetHeight, WidgetWidth);

        workers[i]->moveToThread(threads[i]);

        connectSignals2Slots(threads[i], workers[i]);

        threads[i]->start();
        qDebug() << "Thread from camera " << (i+1) << " started";
    }
}

void MainWindow::connectSignals2Slots(QThread *thread, Worker *worker)
{
    connect(thread, SIGNAL(started()), worker, SLOT(readVideo()));
    connect(thread, SIGNAL(finished()), thread, SLOT(deleteLater()));
    connect(worker, SIGNAL(frameFinished(Mat, int)), this, SLOT(displayFrame(Mat,int)));
    connect(worker, SIGNAL(finished(int)), thread, SLOT(quit()));
    connect(worker, SIGNAL(finished(int)), worker, SLOT(deleteLater()));
}

void MainWindow::displayFrame(Mat frame, int index)
{
    if (index == 0) {
        // Camera 1
        ui->CVWidget1->showImage(frame);
    }
    else if (index == 1) {
        // Camera 2
        ui->CVWidget2->showImage(frame);
    }
    else if (index == 2) {
        // Camera 3
        ui->CVWidget3->showImage(frame);
    }
}

worker.cpp

Worker::Worker(QString path, int id, int WidgetHeight, int WidgetWidth) : filepath(path), index(id), WidgetHeight(WidgetHeight), WidgetWidth(WidgetWidth) {
}

Worker::~Worker(){
}

void Worker::readVideo()
{
    VideoCapture cap(filepath.toStdString());

    if (! cap.isOpened()) {
        qDebug() << "Can't open video file " << filepath;
        emit finished(index);
        return;
    }

    Mat ActualFrame;
    while (true) {
        cap >> ActualFrame;

        if (ActualFrame.empty()) {
            // Empty frame to display when the video has finished
            ActualFrame = Mat(Size(720, 576), CV_8UC3, Scalar(192, 0, 0));
            emit frameFinished(ActualFrame, index);

            qDebug() << "Video finished";
            break;
        }

        // Background Subtraction
        BackgroundSubtraction(ActualFrame, BackgroundMask);

        emit frameFinished(ActualFrame.clone(), index);
        QThread::msleep(35);
    }
    emit finished(index);
}

void Worker::BackgroundSubtraction(Mat ActualFrame, Mat &BackgroundMask)
{
    pMOG2->apply(ActualFrame, BackgroundMask);
}

Just reading the frames from VideoCapture and displaying them into the UI by another different class that uses QWidgets works well.
However, when I include the BackgroundSubstraction method, the UI does not display the same frame number for the three cameras, maybe Camera1 is computing frame 100 and Camera2 and Camera3 are in frame 110.
This is because some frames are calculated faster than other and this leads to syntonization problems.
I'm quite new using threads in QT so i would like to make some synconization between threads so I know when the three different frames have been process in order to call the displayFrame method, and so, that the three same frames are displayed at the exact same time.

EDIT:
I assume that the easiest way to do this is using Barriers.
http://www.boost.org/doc/libs/1_55_0/doc/html/thread/synchronization.html#thread.synchronization.barriers . But I have no clue how to do this.

EDIT 2: I have implemented this Syncronizacion using barriers and now the code looks like this:

barrier.h

#ifndef BARRIER_H
#define BARRIER_H

#include <QMutex>
#include <QWaitCondition>
#include <QSharedPointer>

// Data "pimpl" class (not to be used directly)
class BarrierData
{
public:
    BarrierData(int count) : count(count) {}

    void wait() {
        mutex.lock();
        --count;
        if (count > 0)
            condition.wait(&mutex);
        else
            condition.wakeAll();
        mutex.unlock();
    }
private:
    Q_DISABLE_COPY(BarrierData)
    int count;
    QMutex mutex;
    QWaitCondition condition;
};

class Barrier {
public:
    // Create a barrier that will wait for count threads
    Barrier(int count) : d(new BarrierData(count)) {}
    void wait() {
        d->wait();
    }

private:
    QSharedPointer<BarrierData> d;
};

#endif // BARRIER_H

updated worker.cpp

void Worker::readVideo()
{
    VideoCapture cap(filepath.toStdString());

    int framenumber = 0;
    if (! cap.isOpened()) {
        qDebug() << "Can't open video file " << filepath;
        emit finished(index);
        return;
    }

    Mat ActualFrame;
    while (true) {
        cap >> ActualFrame;

        if (ActualFrame.empty()) {
            // Empty frame to display when the video has finished
            ActualFrame = Mat(Size(720, 576), CV_8UC3, Scalar(192, 0, 0));
            emit frameFinished(ActualFrame, index);

            qDebug() << "Video finished";
            break;
        }

        // Background Subtraction
        BackgroundSubtraction(ActualFrame, BackgroundMask);

        QThread::msleep(5);
        barrier.wait();
        qDebug() << "Thread " << index << " processing frame " << framenumber ;
        emit frameFinished(ActualFrame.clone(), index);
        framenumber++;
    }
    emit finished(index);
}

void Worker::BackgroundSubtraction(Mat ActualFrame, Mat &BackgroundMask)
{
    pMOG2->apply(ActualFrame, BackgroundMask);
}

It seems to work perfectly, however the output of the program is the following:

Thread  1  processing frame  0
Thread  0  processing frame  0
Thread  2  processing frame  0
Thread  2  processing frame  1
Thread  1  processing frame  1
Thread  0  processing frame  1
Thread  2  processing frame  2
Thread  1  processing frame  2
Thread  0  processing frame  2
Thread  2  processing frame  3
Thread  1  processing frame  3
Thread  0  processing frame  3
Thread  2  processing frame  4
Thread  1  processing frame  4
Thread  0  processing frame  4
Thread  2  processing frame  5
Thread  0  processing frame  5
Thread  1  processing frame  5
Thread  2  processing frame  6
Thread  1  processing frame  6
Thread  2  processing frame  7
Thread  0  processing frame  6
Thread  1  processing frame  7
Thread  2  processing frame  8
Thread  0  processing frame  7
Thread  1  processing frame  8
Thread  2  processing frame  9
Thread  0  processing frame  8
Thread  1  processing frame  9
Thread  1  processing frame  10
Thread  2  processing frame  10
Thread  0  processing frame  9
Thread  1  processing frame  11
Thread  2  processing frame  11
Thread  0  processing frame  10
Thread  1  processing frame  12

At the beginning the syncronization is perfectly working, but then it seems that the barrier is not working and threads are not waiting to each other...

EDIT 3: SOLVED It seems that changing the value of

QThread::msleep(5);

to

QThread::msleep(35);

solves the problem of the syncronization although I do not really understand the reason.

even without the background subtraction you'd need some synchronization to be sure that the same frame number is processed by each thread. In Qt the easiest (and imho the right) way to do it is to remove the infinite loop and instead call a slot of each thread to compute the next image, after all the threads emitted their signal frameFinished. You could further use some buffering to precompute images in your threads and just load them from that buffer. — Micka
– Micka, Commented Apr 23, 2017 at 13:23

Micka · Accepted Answer · 2017-04-23 13:59:23Z

2

even without the background subtraction you'd need some synchronization to be sure that the same frame number is processed by each thread.

In Qt the easiest (and imho the right) way to do it is to remove the infinite loop and instead call a slot of each thread to compute the next image, after all the threads emitted their signal frameFinished.

You could further use some buffering to precompute images in your threads and just load them from that buffer. In that scenario you could do the following:

each of your threads fills his buffer in an endless loop as long as there is free buffer space available. If the buffer is full, the thread waits until buffer space got freed.
when your gui has displayed and has waited some time, it sends a signal that is connected to each thread's slot like sendMeANewImage.
each thread sends the next available image from its buffer, or waits (infinite loop or conditional wait) for an image, if the buffer is empty. Then emits a frameFinished signal and frees the used buffer-space.
when each thread has emitted the signal, display all the images, wait some time and emit sendMeANewImage again.

This isn't threadsafe yet, you'll have the critical sections in reading and writing from the buffer. For each buffer, create a QMutex and call mutex.lock() whenever reading or writing or asking size etc. from that buffer. Call mutex.unlock() immediately afterwards.

When a mutex is locked and another thread (or even the same thread) tries to lock it again, the thread will wait there, until the other thread has unlocked the mutex. That way, only a single thread can enter a critical section.

edited Apr 23, 2017 at 13:59

answered Apr 23, 2017 at 13:37

Micka

20.2k6 gold badges58 silver badges76 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Alex Over a year ago

I found this solution link that uses a Barrier to create a point in the code where the threads should wait. This means that when the last thread reaches the barrier all the threads will continue. I have implemented so the call to wait is before emitting the frameFinished(). Do you think is correct?

Micka Over a year ago

just try it. Will probably work. Didnt read it, but sounds similar to my approach.

Collectives™ on Stack Overflow

Multi-threading with QT + OpenCV

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related