Use all cpu core in a python script for one process

Question

i wanna use all cpu in a python script i find some code same :

def do_sum():
    min = 0
    max = 100000000
    while min < max:
        min += 1
        file = open('mytext.txt','a')
        file.write(str(min))
def main():
    q = Queue()
    p1 = Process(target=do_sum)
    p2 = Process(target=do_sum)
    p1.start()
    p2.start()
    r1 = q.get()
    r2 = q.get()
    print r1+r2

if __name__=='__main__':
    main()

but it's not match cpu together p1 start write from 1,2,3,4,5 .... and p2 not continue p2 also start from begin 1,2,3,4 so result is : 1122334455

how i can match 2 core of cpu together ? i want write file with fastest my PC can do it , i use i7 cpu ,how can i use all

You're bottleneck here is hardly cpu, most likely you're bound by file write speed. And note that file writing is not thread safe, you'll get mangled data without some sort of locks. — alko
– alko, Commented Dec 20, 2013 at 13:38
possible realated to Python: Why different threads get their own series of values from one generator? — alko
– alko, Commented Dec 20, 2013 at 13:39
i think this thread work same that code and have different values — user2244393
– user2244393, Commented Dec 20, 2013 at 14:01

lucasg · Accepted Answer · 2013-12-20 14:23:25Z

2

You need a lock mechanism : http://en.wikipedia.org/wiki/Lock_%28computer_science%29 and references for (min, max), not local copies. The multiprocessing lib has already a Lock() object to avoid overwriting and a Value() object to share a mutual state between several process.

from multiprocessing import Queue, Process, Lock,Value

def do_sum(id, counter, lock):
    MAX = 50
    while counter.value < MAX:    

        lock.acquire()
        counter.value += 1

        file = open('mytext.txt','a')
        file.write(str(counter.value))
        file.write("\n")
        file.close()

        lock.release()


def main():

    counter = Value('d', 0.0)
    lock = Lock()

    #f = open('mytext.txt','w')
    #f.close()
    print 'atat'
    q = Queue()
    p1 = Process(target=do_sum, args=(0, counter, lock,) )
    p2 = Process(target=do_sum, args=(1,counter, lock,) )
    p1.start()
    p2.start()
    r1 = q.get()
    r2 = q.get()
    print r1+r2

if __name__=='__main__':
    main()

Anyway, you can harness the power of your cpu all you want, the perfs' bottleneck of your algorithm is located in the I/O operations (which are inherently sequentials).

edited Dec 20, 2013 at 14:23

answered Dec 20, 2013 at 13:40

lucasg

11k4 gold badges38 silver badges58 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

user2244393 Over a year ago

can get me a example in python script :D ? about this lock science i'm new python , i'm not pro , thanks

user2244393 Over a year ago

that's it dude :D now i test for use my cpu 5core >:) thanks again

Collectives™ on Stack Overflow

Use all cpu core in a python script for one process

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related