Python3 multiprocessing a 'for' loop

Question

I really want to ask this, because even if I read the documentation and have seen some examples cannot really get the point of how to use this.

I have a Raspberry pi 3 model B, and I cross compiled QT 5.6 and the lastest SIP and PyQt5 versions to develop Python GUIs and use the linux frame buffer, everything was a success, until I ran this part of my code

def refresh_data(self):
        if self.setTarget == 1:
            for x in range(0, self.targetnum):
                self.target.append(getShadowInfo(x))
                if float(self.target[x]) != self.datalist[x]:
                    if float(self.target[x]) > self.datalist[x]:
                        self.step.append(float(self.target[x]) - self.datalist[x])
                        self.negative.append(0)
                    else:
                        self.step.append(self.datalist[x] - float(self.target[x]))
                        self.negative.append(1)
                else:
                    self.step.append(0)
                    self.negative.append(0)
                self.step[x] *= 0.1
            self.setTarget = 0
            self.setTodaysDate(self.year, self.month, self.day, self.hour, self.min)
        self.stopv += 10
        for x in range(0, self.targetnum):
            if self.step[x] != 0:
                if self.negative[x] == 0:
                    self.datalist[x] += self.step[x]
                else:
                    self.datalist[x] -= self.step[x]
                self.setCustomParameter(x)
        if all(i == 0 for i in self.step):
            self.timer.stop()
        if self.stopv >= 100:
            self.timer.stop()

Which reads data from an external file and set it as a target, then it increase or decrease the actual value to update it on the python GUI, that way it looks smooth, but meanwhile that happens, the performance gets poor and it even execute the code slowly than it should (is ran by an 50ms Qtimer). With htop, I notice that when running this part of the code, my RPi only use one core of its four, can somebody help me to to multiprocess the two for loops?, or maybe better, the refresh_data function?.

EDIT!

setCustomParameter function

def setCustomParameter(self, intparameter):
        if intparameter == 1:
            #RPMMeter
            self.hygrometer.setProperty("gaugeValue", round(self.datalist[1], 2))
            self.label_5.setText(QCoreApplication.translate("MainWindow", "Engine Speed: " +
            str(round(self.datalist[1], 2)) + " RPM"))
        if intparameter == 2:
            #Pressure
            self.label.setText(QCoreApplication.translate("MainWindow",
            str(round(self.datalist[2], 2)) + " KPa"))
            self.progressBar.setProperty("value", self.datalist[2])
        if intparameter == 3:
            self.thermometer.setProperty("thermoValue", round(self.datalist[3], 2))
        if intparameter == 4:
            self.KW_Meter.setProperty("gaugeValue", round(self.datalist[4], 2))
        if intparameter == 5:
            self.Battery_bank_label.setText(QCoreApplication.translate("MainWindow",
            "Battery Bank Voltage: " + str(round(self.datalist[5], 2)) + "V (MEDIUM)"))

Could you please reduce your question to the points which are essential to the problem? — Klaus D.
– Klaus D., Commented Jul 18, 2018 at 3:40
Two for loops, the first one, read info from another file and compare it with the actual infomation, then, store the target, the second one, set the target increasing or decreasing it by 10%, my raspberry turns slow doing this, how to multiprocess those two for loops of maybe if possible, how to multiprocess the function "refresh_data" — Ronald Petit
– Ronald Petit, Commented Jul 18, 2018 at 3:43
I was actually talking about the question itself. You can edit it. Many people here are pretty busy specialist who are investing they private time and it is a good idea to reduce the amount of reading they have to do to a minimum. This is BTW one of the hint you can find in the How to ask section. — Klaus D.
– Klaus D., Commented Jul 18, 2018 at 3:51
Got it, already edited, I deleted all the useless background explaination — Ronald Petit
– Ronald Petit, Commented Jul 18, 2018 at 3:56

jmunsch · Accepted Answer · 2018-07-18 06:14:20Z

1

it looks like you might be able to 3x the speed by reducing (two for loops, 1 all which is also a for loop) to 1 for loop with some clever placement of the if statements.

by zipping the target, datalist. assuming that self.target, self.negative, self.datalist, and self.step are all the same length.

for i, (t, d) in enumerate(zip(self.target, self.datalist)):
    t = float(t)
    if (t != d) and (t > d) and (self.setTarget == 1):
        self.target.append(getShadowInfo(x))
        self.datalist = d + (t - d) * .1
    elif (t != d) and (t <= d) and (self.setTarget == 1):
        self.target.append(getShadowInfo(x))
        self.datalist = d - (d - t) * .1
    else:
        self.setCustomParameter(i)

if self.setTarget == 1
    self.setTarget = 0
    self.setTodaysDate(self.year, self.month, self.day, self.hour, self.min)

if you were able to figure out how to reduce to 1 for loop, you could then multiprocess it by replacing the for loop entirely by returning the index and data from a multiprocessing.pool.Pool:

# replacing ... self.datalist = d - (d - t) * .1
# with ... return i, d - (d - t) * .1
results = mp.Pool(4).starmap(return_data, zip(self.target, self.datalist))
for i, d in results:
    self.datalist[i] = d

note: the trick is going to be figuring out how to process what needs updating without mutating self.target in the for loop

edited Jul 18, 2018 at 6:14

answered Jul 18, 2018 at 5:57

jmunsch

24.3k12 gold badges102 silver badges120 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Ronald Petit Over a year ago

Sorry, I think I'm too beginner to understand your answer (don't know the role of t and d), the script works, I can see the GUI, but when I press Refresh it does nothing, first, because the function setCustomParameter is on else, if no conditions are met, I put it inside but still it doesn't update the parameters, I'm going to edit the question to add the function. Also, I think you should not zip target and datalist, target, initially, is an empty list, it gets filled by appending with the variable "targetnum" (which is the datalist length), I still find your answer very useful.

Ronald Petit Over a year ago

So far, I got this to work: pastebin.com/rYwx8VwA However, it only update everything once, and then, doesn't keep it up, don't know why yet, I'm pretty sure the Qtimer is correctly set

Ronald Petit Over a year ago

Just to anybody who like to know, thanks to this answer I improved a lot my python script and learned a bit, however, the python script wans't even too big to slow down the RPi, the problem was not the code but the screen resolution, apparently it matters A LOT, I was using an HD TV (1080p), and the FPS were around 20 - 25 with drops to 2-7 FPS when interacting with the GUI, I set the screen resolution on raspi-config to 480p, and I got 60 FPS all the time, with only drops to 57 or 55 when interacting with the UI

Collectives™ on Stack Overflow

Python3 multiprocessing a 'for' loop

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related