Multithreaded Python Script but just 20% CPU? How can I speed up my Python Script?

Question

I programmed a simple SHA256 blockchain mining algorithm in python using hashlib. What I am trying to do here is changing a random number in my data and then calculating the SHA256 hash. Whenever the hash has the predefined amount of zeroes in the beginning, it prints the hash and its corresponding nonce to the console. The goal is to have 11 zeroes in the beginning.

import hashlib
import threading

#bn=block number d=input data, ph=previous hash
bn = 1

d = "mydata"

ph = "0000000000000000000000000000000000000000000000000000000000000000"


class MyThread(threading.Thread):
    def __init__(self, threadID, begin, end):
        threading.Thread.__init__(self)
        self.threadID = threadID
        self.end = end
        self.begin = begin


    def run(self):
        for val in range(self.begin, self.end):
            mine(self.threadID, val)
        print("done " + str(self.threadID))

#hashing function

def mine(id, x):
    hash = hashlib.sha256((str(bn) + str(x) + str(d) + str(ph)).encode('utf-8')).hexdigest()
    if hash.find("0000000", 0, 7) > -1:
        # print(id)
        print(x)
        print(hash)

#now break it up and distribute it to the threads

#Possible SHA256 Combinations 8388608000000

pc = 8388608000000

#Number of desired Threads (more threads help more?)

nt = 704

#calculating the steps

step = round((pc)/(nt))

#starting the threads with each thread calculating a different range of nonces.
for x in range (1, nt):
    thread = MyThread((x), (x-1)*(step), (x)*(step))
    thread.start()

How can I speed this thing up?

Currently, I run this on an Intel 8250u quadcore and I can get hashes with 7 zeroes in the beginning comfortably. However, 8 zeroes already take 5 hours. Funnily, only around 20-30% of my CPU is used. How can I make it run on all cores with threads running in parallel?

Do you have another Idea how I can speed this thing up? I have better hardware (Threadripper 1920) available but I'm afraid that alone will not give me a 300x - 4000x (best case) speed increase...

My first thought was outsourcing this to the GPU since I know that Bitcoin Mining evolved from CPU to GPU. When looking into Numba I saw that it does not support Hashlib. I also saw that compiling this using Pypy might speed it up? What approach would you guys favour? I'm confused...

Thanks for being patient with me

OS threads have a significant overhead in both creation and communication. Furthermore in Python in particular you have a global interpreter lock (GIL) meaning that threads in python are a poor choice for parallel calculations. Also unless you have hundreds of cores having hundreds of threads is going to slow you down, way, way down. Python does have the multiprocessing module for true parallel processing. — Jared Smith
– Jared Smith, Commented Jul 21, 2020 at 13:22
I see. So I should not use multithreading at all but just multiprocessing? What do you think about ProcessPoolExecutors from this tutorial? youtube.com/watch?v=J7w_G6ZKzz4&feature=youtu.be — Pirmin Oe
– Pirmin Oe, Commented Jul 21, 2020 at 13:51
Use multiprocessing for parallel calculations. If you just need concurrency for e.g. non-blocking I/O, then it makes sense to use threading or asyncio. I use threading all the time because my work involves coordinating multiple data streams from TCP sockets, HTTP, and the file system but if I need true parallelism it's multiprocessing all the way. — Jared Smith
– Jared Smith, Commented Jul 21, 2020 at 14:24
I did use multiprocessing module, its a little bit faster now. Do you think it is possible to compile with cython or nuitka or it is not possible because of the libraries? — Pirmin Oe
– Pirmin Oe, Commented Jul 21, 2020 at 22:06

Marcel · Accepted Answer · 2020-07-21 13:34:01Z

1

Python is a bit weird in this regard. What you're doing here is multithreading and not multiprocessing. Because of the Global Interpreter Lock, these threads aren't actually running at the same time. Check out the multiprocessing module for Python if you want to actually run calculations in parallel.

EDIT: As Jared mentioned, Bitcoin mining on your personal computer is no longer profitable due to most mining now being done with specialized hardware. As a project this is cool, but I wouldn't expect it to make you money.

edited Jul 21, 2020 at 13:34

answered Jul 21, 2020 at 13:24

Marcel

1,0581 gold badge10 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Jared Smith Over a year ago

You might also mention that mining bitcoin on general purpose computers hasn't been profitable for a long time now, even using GPU's.

Pirmin Oe Over a year ago

Yes, I know, this is just for learning purposses. It is an assignment that I am required to do.

Collectives™ on Stack Overflow

Multithreaded Python Script but just 20% CPU? How can I speed up my Python Script?

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related