Simple python script, multithreaded?

Question

I have a simple python script that reads from two .txt files and saves an output on a new text file.

This is what my code looks like:

import os

with open("users.txt", encoding="utf-8") as f:
    users = f.readlines()
users = [x.strip() for x in users] 

with open("users1.txt", encoding="utf-8") as f:
    passes = f.readlines()
passes = [x.strip() for x in passes]

results = []

for user in users:
    h = user[user.find(':') + 1:]
    for p in passes:
        if p[:p.find(':')] == h:
            results.append((user[:user.find(':')], p[p.find(':') + 1:]))
            passes.remove(p)
            break

with open('results.txt', 'w+', encoding="utf-8") as f:
    for item in results:
        f.write(f"{item[0]}:{item[1]}\n")

The user.txt has about 3-10M lines.. Which of course will take ages to process, im talking 1-3 hours. I thought the only reason of this is because the script is running in a single thread. I did some digging around and im out of ideas. Is there any way to make this simple script run on more than 1 thread? Will it speed up the process?

Thanks!

Frank Yellin · Accepted Answer · 2020-10-23 06:39:37Z

2

It would save you a lot of time if you first converted passes into a dictionary of some sort. I don't completely understand your code, but it seems that each user and each pass has some sort of id or key embedded in it, and you're looking to find matching ids.

Convert passes into a dictionary

my_dictionary = { pass[:pass.find(':')] : pass for pass in passes }

Your outer loop is then:

for user in users:
    id = user[user.find(':') + 1:]
    if id in my_dictionary:
        pass = my_dictionary[id]
        .... whatever you do with user and pass ...
        del my_dictionary[id]

A single dictionary lookup instead of a nested loop.

answered Oct 23, 2020 at 6:39

Frank Yellin

11.6k1 gold badge15 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Frank Yellin Over a year ago

Overnight, I realized that the above code assumes that each id only occurs in one pass. If there is supposed to be duplication, you'll need to make a couple of changes. my_dictionary becomes a dictionary from the id to a list of pass records that it matches.

Collectives™ on Stack Overflow

Simple python script, multithreaded?

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related