What's wrong with my python multiprocessing code?

Question

I am an almost new programmer learning python for a few months. For the last 2 weeks, I had been coding to make a script to search permutations of numbers that make magic squares.

Finally I succeeded in searching the whole 880 4x4 magic square numbers sets within 30 seconds. After that I made some different Perimeter Magic Square program. It finds out more than 10,000,000 permutations so that I want to store them part by part to files. The problem is that my program doesn't use all my processes that while it is working to store some partial data to a file, it stops searching new number sets. I hope I could make one process of my CPU keep searching on and the others store the searched data to files.

The following is of the similar structure to my magic square program.

while True:
    print('How many digits do you want? (more than 20): ', end='')
    ansr = input()
    if ansr.isdigit() and int(ansr) > 20:
        ansr = int(ansr)
        break
    else:
        continue

fileNum = 0
itemCount = 0

def fileMaker():
    global fileNum, itemCount
    tempStr = ''
    for i in permutationList:
        itemCount += 1
        tempStr += str(sum(i[:3])) + ' : ' + str(i) + ' : ' + str(itemCount) + '\n'
    fileNum += 1
    file = open('{0} Permutations {1:03}.txt'.format(ansr, fileNum), 'w')
    file.write(tempStr)
    file.close()

numList = [i for i in range(1, ansr+1)]

permutationList = []
itemCount = 0

def makePermutList(numList, ansr):
    global permutationList
    for i in numList:
        numList1 = numList[:]
        numList1.remove(i)
        for ii in numList1:
            numList2 = numList1[:]
            numList2.remove(ii)
            for iii in numList2:
                numList3 = numList2[:]
                numList3.remove(iii)
                for iiii in numList3:
                    numList4 = numList3[:]
                    numList4.remove(iiii)
                    for v in numList4:
                        permutationList.append([i, ii, iii, iiii, v])
                        if len(permutationList) == 200000:
                            print(permutationList[-1])
                            fileMaker()
                            permutationList = []
    fileMaker()

makePermutList(numList, ansr)

I added from multiprocessing import Pool at the top. And I replaced two 'fileMaker()' parts at the end with the following.

if __name__ == '__main__':
    workers = Pool(processes=2)
    workers.map(fileMaker, ())

The result? Oh no. It just works awkwardly. For now, multiprocessing looks too difficult for me.

Anybody, please, teach me something. How should my code be modified?

If there is a specific problem, please put it here. If the code words and you just want to improve it, go to Code Review — Inbar Rose
– Inbar Rose, Commented Jul 15, 2013 at 12:29
I suggest using some numpy or so, because that many stacked for-loops is disaster when it comes to computing times — usethedeathstar
– usethedeathstar, Commented Jul 15, 2013 at 12:57
My real program code is more complicated than that so many stacked for-loops are essential. Thanks. — user2579484
– user2579484, Commented Jul 16, 2013 at 4:10

roippi · Accepted Answer · 2013-07-15 13:08:15Z

1

Well, addressing some things that are bugging me before getting to your asked question.

numList = [i for i in range(1, ansr+1)]

I know list comprehensions are cool, but please just do list(range(1, ansr+1)) if you need the iterable to be a list (which you probably don't need, but I digress).

def makePermutList(numList, ansr):
...

This is quite the hack. Is there a reason you can't use itertools.permutations(numList,n)? It's certainly going to be faster, and friendlier on memory.

Lastly, answering your question: if you are looking to improve i/o performance, the last thing you should do is make it multithreaded. I don't mean you shouldn't do it, I mean that it should literally be the last thing you do. Refactor/improve other things first.

You need to take all of that top-level code that uses globals, apply the backspace key to it, and rewrite functions that pass data around properly. Then you can think about using threads. I would personally use from threading import Thread and manually spawn Threads to do each unit of I/O rather than using multiprocessing.

answered Jul 15, 2013 at 13:08

roippi

26k4 gold badges52 silver badges75 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user2579484 Over a year ago

Thank you very much. Still I can't understand why using multi processes is worse than single thing. But I'll study threading.

Collectives™ on Stack Overflow

What's wrong with my python multiprocessing code?

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related